Most hosts and devices are configured to log important events. The default solution however, is to write log entries locally. There are several reasons why that approach isn't the best. If the device is compromised, the first thing the hacker will wipe is the log file that contains his traces. Another reason why using centralized logging is better is that after a fatal device failure, you'll still have access to the logging information. This may help you analyze why things went wrong. In this tip, you will learn how to set up a centralized Linux log server.
The benefit of using Linux as a central logging server is that many devices allow you to set up logging via syslog, and
Before talking about how to set up a Linux log host using syslog, you should be aware of the different syslog solutions currently existing. First, there is the syslogd implementation. This is the classical syslog solution and is supported by all devices that can send their log messages to syslog. Because this solution has some limitations, there are two newer versions of syslog: syslog-ng and rsyslog. One of the major advantages of syslog-ng is that administrators can filter out very specific information. Rsyslog is a new addition to the world of syslog, and it can do everything that syslog-ng and syslog can do. It is currently the best choice and also the default choice on important Linux distributions, such as Red Hat and Ubuntu.
Setting up a server for centralized logging is not that difficult. To set up the server, you first need a large and preferably dedicated file system for the /var/log directory. Log files on individual hosts can grow quickly, so if you set up a log host that logs files for many hosts, this file system can become full extremely fast. If you start allocating 5 GB of disk space for each machine that is going to write log files to your server, you have a good starting point and won't run out of disk space quickly.
Another requirement is logrotate. By default, most Linux distributions have a decent setup that closes log files weekly and opens a new copy, with a maximum of four old versions of the file that are allowed to exist. In listing 1, you can see the part of text from /etc/logrotate.conf that takes care of this by default. However, you should carefully read the entire logrotate.conf configuration file and all logrotate files in /etc/logrotate.d to make sure there are no nasty exceptions that may fill up the log file system quickly.
Listing 1: By default, the logrotate configuration should prevent your log file system from filling up too quickly.
[root@localhost etc]# cat logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
# keep 4 weeks worth of backlogs
# create new (empty) log files after rotating old ones
After checking the logrotate configuration, there's nothing else you need to do. Logrotate runs as a cron job, which means that your server will execute the logrotate jobs automatically once a day.
Opening the log server
Once you've confirmed the working of logrotate, you can set up your rsyslog server. This is rather easy to do -- just open either UDP or TCP port 514. In listing 2, you can see the lines that are taking care of this by default:
Listing 2: The default lines to enable UDP or TCP log reception
# Provides UDP syslog reception
# Provides TCP syslog reception
To make your server a log server, just remove the hash mark and restart the rsyslogd server using service rsyslogd restart. As the final step, you need to enable your devices to send their log messages to your rsyslogd server. How to do this depends on the device that you are using. You'll often find a Web interface that makes it easy to write messages to the log server. If you want another Linux server to write messages to a syslog server, you need to enter the IP address of the syslog server as the log destination. Also, you need to specify if you want to log over UDP or over TCP. To use UDP, include a single @ sign. To enable TCP logging, you need two @ signs, as shown in the example below:
Every Linux server has what it takes to configure it as a centralized log server. In this article, you've read how to set up rsyslogd as a log server, which helps ease management of log entries in a data center environment.
About the expert
Sander van Vugt is an independent trainer and consultant living in the Netherlands. Van Vugt is an expert in Linux high availability, virtualization and performance and has completed several projects that implement all three. He is also the writer of various Linux-related books, such as Beginning the Linux Command Line, Beginning Ubuntu Server Administration and Pro Ubuntu Server Administration.
This was first published in March 2011