Journald primer for Linux server admins

The latest Linux distributions all use journald, which means a lot of changes to system logging and management.

Journald is the new system logging method for Linux servers and it spells the end of text log files. Now that log information is written to a binary file read with journalctl, Linux administrators will need practice obtaining the information they want.

Red Hat Enterprise Linux 7, SUSE Linux Enterprise Server 12 -- these next-generation Linux distributions are managing services with systemd. The journal, a component of systemd, is handled by journald. It captures syslog messages, kernel log messages, messages coming from the initial RAM disk, early boot messages and everything that is written to the STDOUT and STDERR streams from all services. Journald radically changes how servers handle log messages and how administrators access them.

Goodbye log files

There are no log files in the systemd and journald world. The journald log is written to a binary file; on a Red Hat system, it resides in /run/log/journal. You shouldn't -- and cannot -- open this file with a pager. Instead, use journalctl to see its contents. This command shows you everything that has ever been logged to the server (see Listing 1).

Listing 1. This journalctl output list is a small example of the default format.

Apr 04 09:48:59 localhost.localdomain chronyd[768]: Can't synchronise: no majority

Apr 04 09:50:01 localhost.localdomain systemd[1]: Starting Session 3 of user root.

Apr 04 09:50:01 localhost.localdomain systemd[1]: Started Session 3 of user root.

Apr 04 09:50:01 localhost.localdomain CROND[3699]: (root) CMD (/usr/lib64/sa/sa1 1 1)

Apr 04 09:50:03 localhost.localdomain chronyd[768]: Selected source 46.249.47.127

Apr 04 09:50:03 localhost.localdomain chronyd[768]: System clock wrong by -2.417074 seconds, adjustment started

Apr 04 09:50:36 localhost.localdomain pulseaudio[3163]: [alsa-sink] alsa-sink.c: ALSA woke us up to write new data to the device, but there

Apr 04 09:50:36 localhost.localdomain pulseaudio[3163]: [alsa-sink] alsa-sink.c: Most likely this is a bug in the ALSA driver 'snd_ens1371'.

Apr 04 09:50:36 localhost.localdomain pulseaudio[3163]: [alsa-sink] alsa-sink.c: We were woken up with POLLOUT set -- however a subsequent s

Apr 04 09:51:07 localhost.localdomain chronyd[768]: Selected source 81.171.44.131

Apr 04 09:52:12 localhost.localdomain chronyd[768]: System clock wrong by 0.669116 seconds, adjustment started

Apr 04 09:53:17 localhost.lo

But don't worry -- journalctl has many filtering options. Journalctl -b filters for messages generated while booting only. Journalctl --since=yesterday shows only messages that have been logged since yesterday. Administrators can search for messages from a specific range of days: for example, journalctl --since=2014-03-15 --until="2014-03-17 23:59:59". Use journalctl -u httpd since=00:00 --until=8:00 to see what the httpd process logged last night. Once administrators are comfortable with the advanced filtering options from journald, analyzing log files gets much easier.

Become a RHEL 7, SLES 12 expert

New Linux boot options

Red Hat onboards Docker

In-depth: firewalld, XFS and systemd

On some occasions, the default log information that journalctl shows is not detailed enough. For more information, set the output format to verbose, using journalctl -o verbose -n.

Listing 2. By displaying verbose log information, Linux admins will get more information from log files.

Fri 2014-04-04 10:12:32.072521 CEST [s=a52ddd97575747a18c6378d388b2b9ff;i=955;b=bc03fb52eddb41
b0bb4829ae19c1c286;m=8f1dd 5f2;t=4f633145a58d9;

 PRIORITY=6

 _UID=0

 _GID=0

 _BOOT_ID=bc03fb52eddb41b0bb4829ae19c1c286

 _MACHINE_ID=1fbfd90ac4fc49919fe1b63d6bcf9097

 _HOSTNAME=localhost.localdomain

 SYSLOG_FACILITY=3

 _TRANSPORT=syslog

 _SYSTEMD_CGROUP=/system.slice/network.service

 _SYSTEMD_UNIT=network.service

 SYSLOG_IDENTIFIER=dhclient

 _COMM=dhclient

 _EXE=/usr/sbin/dhclient

 _CMDLINE=/sbin/dhclient -H localhost -1-q-lf/var/lib/dhclient/dhclient-0b5faf33-6df0-4f11-bbb9-659b5cd940e9-ens33.lease -pf /var/run/

 _CAP_EFFECTIVE=0000000000203402

 _SELINUX_CONTEXT=system_u:system_r:dhcpc_t:s0

 SYSLOG_PID=1760

 _PID=1760

 MESSAGE=bound to 192.168.4.232 -- renewal in 892 seconds.

 _SOURCE_REALTIME_TIMESTAMP=1396599152072521

Logrotate and remote logging

Not everything works the way you're used to. The system, logrotate, which closes and archives log files that grow too large, is one example. On journald, there is no need to rotate log files; it was built to monitor the amount of free space on its storage volume. It shrinks itself by deleting oldest entries if the volume is filling up. To set a maximum size for the journald log, modify the SystemMaxUse parameter in the /etc/systemd/journal.conf file.

Remote logging is a different story. If your data center has a remote log server, you probably want to keep it; journald doesn't offer a full replacement for centralized log servers like rsyslog or syslog-ng do. Journald doesn't have options to open it for reception of log files coming in from other servers or devices. It also has no options to specify to which log server these log events should be forwarded. If you want journald to store its log messages elsewhere, the best approach is to forward messages to [r]syslog[{d-ng}] and handle centralized logging there.

About the author:
Sander van Vugt is an independent trainer and consultant based in the Netherlands. He is an expert in Linux high availability, virtualization and performance. He has authored many books on Linux topics, including 
Beginning the Linux Command Line, Beginning Ubuntu LTS Server Administration and Pro Ubuntu Server Administration.

This was first published in May 2014
This Content Component encountered an error

Pro+

Features

Enjoy the benefits of Pro+ membership, learn more and join.

Related Discussions

ITKE asks:

Are systemd and journald a welcome change or a hassle to get used to?

2  Responses So Far

Join the Discussion

0 comments

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

-ADS BY GOOGLE

SearchWindowsServer

SearchEnterpriseLinux

SearchServerVirtualization

SearchCloudComputing

Close