Get started Bring yourself up to speed with our introductory content.

Journald primer for Linux server admins

The latest Linux distributions all use journald, which means a lot of changes to system logging and management.

Journald is the new system logging method for Linux servers and it spells the end of text log files. Now that log information is written to a binary file read with journalctl, Linux administrators will need practice obtaining the information they want.

Red Hat Enterprise Linux 7, SUSE Linux Enterprise Server 12 -- these next-generation Linux distributions are managing services with systemd. The journal, a component of systemd, is handled by journald. It captures syslog messages, kernel log messages, messages coming from the initial RAM disk, early boot messages and everything that is written to the STDOUT and STDERR streams from all services. Journald radically changes how servers handle log messages and how administrators access them.

Goodbye log files

There are no log files in the systemd and journald world. The journald log is written to a binary file; on a Red Hat system, it resides in /run/log/journal. You shouldn't -- and cannot -- open this file with a pager. Instead, use journalctl to see its contents. This command shows you everything that has ever been logged to the server (see Listing 1).

Listing 1. This journalctl output list is a small example of the default format.

Apr 04 09:48:59 localhost.localdomain chronyd[768]: Can't synchronise: no majority

Apr 04 09:50:01 localhost.localdomain systemd[1]: Starting Session 3 of user root.

Apr 04 09:50:01 localhost.localdomain systemd[1]: Started Session 3 of user root.

Apr 04 09:50:01 localhost.localdomain CROND[3699]: (root) CMD (/usr/lib64/sa/sa1 1 1)

Apr 04 09:50:03 localhost.localdomain chronyd[768]: Selected source

Apr 04 09:50:03 localhost.localdomain chronyd[768]: System clock wrong by -2.417074 seconds, adjustment started

Apr 04 09:50:36 localhost.localdomain pulseaudio[3163]: [alsa-sink] alsa-sink.c: ALSA woke us up to write new data to the device, but there

Apr 04 09:50:36 localhost.localdomain pulseaudio[3163]: [alsa-sink] alsa-sink.c: Most likely this is a bug in the ALSA driver 'snd_ens1371'.

Apr 04 09:50:36 localhost.localdomain pulseaudio[3163]: [alsa-sink] alsa-sink.c: We were woken up with POLLOUT set -- however a subsequent s

Apr 04 09:51:07 localhost.localdomain chronyd[768]: Selected source

Apr 04 09:52:12 localhost.localdomain chronyd[768]: System clock wrong by 0.669116 seconds, adjustment started

Apr 04 09:53:17 localhost.lo

But don't worry -- journalctl has many filtering options. Journalctl -b filters for messages generated while booting only. Journalctl --since=yesterday shows only messages that have been logged since yesterday. Administrators can search for messages from a specific range of days: for example, journalctl --since=2014-03-15 --until="2014-03-17 23:59:59". Use journalctl -u httpd since=00:00 --until=8:00 to see what the httpd process logged last night. Once administrators are comfortable with the advanced filtering options from journald, analyzing log files gets much easier.

Become a RHEL 7, SLES 12 expert

New Linux boot options

Red Hat onboards Docker

In-depth: firewalld, XFS and systemd

On some occasions, the default log information that journalctl shows is not detailed enough. For more information, set the output format to verbose, using journalctl -o verbose -n.

Listing 2. By displaying verbose log information, Linux admins will get more information from log files.

Fri 2014-04-04 10:12:32.072521 CEST [s=a52ddd97575747a18c6378d388b2b9ff;i=955;b=bc03fb52eddb41
b0bb4829ae19c1c286;m=8f1dd 5f2;t=4f633145a58d9;














 _CMDLINE=/sbin/dhclient -H localhost -1-q-lf/var/lib/dhclient/ -pf /var/run/





 MESSAGE=bound to -- renewal in 892 seconds.


Logrotate and remote logging

Not everything works the way you're used to. The system, logrotate, which closes and archives log files that grow too large, is one example. On journald, there is no need to rotate log files; it was built to monitor the amount of free space on its storage volume. It shrinks itself by deleting oldest entries if the volume is filling up. To set a maximum size for the journald log, modify the SystemMaxUse parameter in the /etc/systemd/journal.conf file.

Remote logging is a different story. If your data center has a remote log server, you probably want to keep it; journald doesn't offer a full replacement for centralized log servers like rsyslog or syslog-ng do. Journald doesn't have options to open it for reception of log files coming in from other servers or devices. It also has no options to specify to which log server these log events should be forwarded. If you want journald to store its log messages elsewhere, the best approach is to forward messages to [r]syslog[{d-ng}] and handle centralized logging there.

About the author:
Sander van Vugt is an independent trainer and consultant based in the Netherlands. He is an expert in Linux high availability, virtualization and performance. He has authored many books on Linux topics, including 
Beginning the Linux Command Line, Beginning Ubuntu LTS Server Administration and Pro Ubuntu Server Administration.

Dig Deeper on Linux servers

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.