Logging in Linux got another component to take into account with the introduction of systemd. In this post, I’ll give an overview of the components involved in handling logs on an average modern Linux system.

The syslog daemon

Historically an application would log its events by writing to /dev/log. It is a Unix domain socket and everything that went into it was received by the syslog daemon.

This is such a special socket it’s even hardcoded into glibc

git clone --depth 1 git://sourceware.org/git/glibc.git
cd glibc
grep -r "/dev/log" --include=*.h
#    bits/syslog-path.h:#define      _PATH_LOG       "/dev/log"

Let’s write some logs directly into the socket:

MESSAGE=log-me-$RANDOM
echo "<1>$MESSAGE" | socat -t 0 - /dev/log
grep $MESSAGE /var/log/syslog
#    Mar 22 22:40:50 user-pc log-me-9211

The log message ended in /var/log/syslog. A proper way to do this from the shell is with logger. It will take care of adding the required metadata so we can concentrate only on the actual message.

MESSAGE=log-me-$RANDOM
logger $MESSAGE
grep $MESSAGE /var/log/syslog 
#    Mar 22 22:42:26 user-pc user: log-me-15856

Listening to that socket was a syslog daemon. It’s duty is accepting raw log messages coming from /dev/log and process them according to the config file. In the above example, we searched in /var/log/syslog, a popular location where the syslog daemon stores logs. Often distributions come with rsyslog as the default syslog daemon. I will use rsyslog and system daemon interchangeably and they will mean the same thing.

That was just a high-level summary of the functions a syslog daemon can do. Concrete examples of how to configure rsyslog is a topic on its own.

systemd introduces a new system daemon, journald. It takes over the task of reading messages from /dev/log from the syslog daemon. This results in rsyslog becoming a systemd service with its own unit files. It doesn’t read /dev/log but is being fed logs by journald using some systemd related features. The more thorough technical description on how this happens can be found here.

The summary of the essential changes with systemd introduction:

• the syslog daemon isn’t handling /dev/log directly anymore
• an underlying systemd service - journald is the primary aggregation point for logs
• the system daemon gets the logs from journald
• nothing conceptually changes for the system daemon except how it gets the log

journald

journald is a service of systemd and on systems with systemd, it is the default local logging daemon. I say “core local logging daemon” because it has certain differences compared to the above-mentioned syslog daemon. In a general sense, they are both services that gather logs and provide some means of accessing them.

They do differ in how you use them and internal implementation:

• journald was built with the belief that log handling should be a built-in operating system capability. An example of what that belief results in is showing logs when running systemctl status <service> (i.e. the command systemctl status dbus)
• it stores logs in an indexed structured unstandardized binary format. This means accessing logs is fast and filterable but needs to happen through a dedicated CLI tool journalctl (as opposed to going through a text file in rsyslog)
• rsyslog is still the battle-tested tool for forwarding logs to remote servers. There are systemd solutions (systemd-journal-remote, systemd-journal-upload) but the amount of talk about them in internet searches doesn’t give an impression they are catching up on popularity with the rest of systemd

More information.

Example on accessing logs with journalctl:

MESSAGE=log-me-$RANDOM
logger $MESSAGE
journalctl -n 10 | grep $MESSAGE
#    Mar 22 22:44:17 user-pc user[1396]: log-me-31625

What about kernel messages

The kernel keeps a data structure in memory that stores kernel logs. That structure is called the kernel ring buffer and it’s readable and even writable from userspace.

The most popular way of accessing these logs at system runtime by the dmesg command. It will read the content of the ring buffer and provide other helpful ring buffer related operation.

The kernel exposes two special files to userspace:

• /dev/kmsg This is a device file from which you can read the kernel buffer directly and even write to it from userspace
• /proc/kmsg Reading this file is a direct invocation of the syslog system call. It’s not clear to me why use it over /dev/kmsg, am assuming it’s still exposed not to break code depending on it. The mainstream logging tools (journald, rsyslog, dmesg) read /dev/kmsg by default.

MESSAGE=log-me-$RANDOM
sudo su
echo $MESSAGE > /dev/kmsg
dmesg | grep $MESSAGE
#    [  645.489050] log-me-6860

So what is then responsible for reading the kernel logs and collecting them with all the other logs?

Most likely journald. It can be enabled for rsyslog by enabling the imkmsg module but you should probably have a good reason to want to not go for the defaults. journald will pass it to rsyslog anyways.