Logging facilities in Linux
Logging in Linux got another component to take into account with the introduction of systemd. In this post, I’ll give an overview of the components involved in handling logs on an average modern Linux system.
The syslog daemon
Historically an application would log its events by writing to
It is a Unix domain socket and everything that went into it was
received by the syslog daemon.
This is such a special socket it’s even hardcoded into glibc
git clone --depth 1 git://sourceware.org/git/glibc.git cd glibc grep -r "/dev/log" --include=*.h # bits/syslog-path.h:#define _PATH_LOG "/dev/log"
Let’s write some logs directly into the socket:
MESSAGE=log-me-$RANDOM echo "<1>$MESSAGE" | socat -t 0 - /dev/log grep $MESSAGE /var/log/syslog # Mar 22 22:40:50 user-pc log-me-9211
The log message ended in
A proper way to do this from the shell is with
It will take care of adding the required metadata so we can concentrate only on the actual message.
MESSAGE=log-me-$RANDOM logger $MESSAGE grep $MESSAGE /var/log/syslog # Mar 22 22:42:26 user-pc user: log-me-15856
Listening to that socket was a syslog daemon.
It’s duty is accepting raw log messages coming from
/dev/log and process them according to the config file. In the above example, we searched in
/var/log/syslog, a popular location where the syslog daemon stores logs.
Often distributions come with rsyslog as the default syslog daemon. I will use rsyslog and system daemon interchangeably and they will mean the same thing.
That was just a high-level summary of the functions a syslog daemon can do. Concrete examples of how to configure rsyslog is a topic on its own.
systemd introduces a new system daemon, journald. It takes over the task of reading messages from
/dev/log from the syslog daemon. This results in rsyslog becoming a systemd service with its own unit files. It doesn’t read
/dev/log but is being fed logs by journald using some systemd related features. The more thorough technical description on how this happens can be found here.
The summary of the essential changes with systemd introduction:
- the syslog daemon isn’t handling
- an underlying systemd service - journald is the primary aggregation point for logs
- the system daemon gets the logs from journald
- nothing conceptually changes for the system daemon except how it gets the log
journald is a service of systemd and on systems with systemd, it is the default local logging daemon. I say “core local logging daemon” because it has certain differences compared to the above-mentioned syslog daemon. In a general sense, they are both services that gather logs and provide some means of accessing them.
They do differ in how you use them and internal implementation:
- journald was built with the belief that log handling should be a built-in operating system capability. An example of what that belief results in is showing logs when running
systemctl status <service>(i.e. the command
systemctl status dbus)
- it stores logs in an indexed structured unstandardized binary format. This means accessing logs is fast and filterable but needs to happen through a dedicated CLI tool
journalctl(as opposed to going through a text file in rsyslog)
- rsyslog is still the battle-tested tool for forwarding logs to remote servers. There are systemd solutions (systemd-journal-remote, systemd-journal-upload) but the amount of talk about them in internet searches doesn’t give an impression they are catching up on popularity with the rest of systemd
Example on accessing logs with
MESSAGE=log-me-$RANDOM logger $MESSAGE journalctl -n 10 | grep $MESSAGE # Mar 22 22:44:17 user-pc user: log-me-31625
What about kernel messages
The kernel keeps a data structure in memory that stores kernel logs. That structure is called the kernel ring buffer and it’s readable and even writable from userspace.
The most popular way of accessing these logs at system runtime by the
It will read the content of the ring buffer and provide other helpful ring buffer related operation.
The kernel exposes two special files to userspace:
/dev/kmsgThis is a device file from which you can read the kernel buffer directly and even write to it from userspace
/proc/kmsgReading this file is a direct invocation of the
syslogsystem call. It’s not clear to me why use it over
/dev/kmsg, am assuming it’s still exposed not to break code depending on it. The mainstream logging tools (journald, rsyslog, dmesg) read
MESSAGE=log-me-$RANDOM sudo su echo $MESSAGE > /dev/kmsg dmesg | grep $MESSAGE # [ 645.489050] log-me-6860
So what is then responsible for reading the kernel logs and collecting them with all the other logs?
Most likely journald. It can be enabled for rsyslog by enabling the imkmsg module but you should probably have a good reason to want to not go for the defaults. journald will pass it to rsyslog anyways.