Tip

Turn aggregated syslog messages into OpenNMS events

This tip is the fourth in a series focused on the practice of enterprise-grade network and systems management using free software, including OpenNMS and syslog-ng.

Requires Free Membership to View

In the first two installments of this series, we configured OpenNMS for service assurance and performance data collection. The third tip switched to aggregation of syslog event messages using the functionality of syslog-ng. Now, it's time to turn the most important of those syslog messages into OpenNMS events so that they can become first-class participants in your network management processes.

We'll pick up where we last left off in the configuration of the syslog aggregation server. You'll recall that we already configured a message source named s_net for reading messages from the network, as well as a log destination named d_hier for writing those received messages to a hierarchical layout on the syslog server's filesystem. Before we can start sending messages back over the network to an OpenNMS server, we first need to configure another destination. We'll also configure a filter, which will act as a rough sieve to avoid filling the network management system with low-severity noise.

Configure the OpenNMS destination after the d_hier definition in /etc/syslog-ng/syslog-ng.conf, using the IP address of your OpenNMS server and the default OpenNMS Syslogd listening port of 10514. For example, if your OpenNMS server is at 192.168.200.100:

destination d_opennms {
    udp(ip(192.168.200.100) port(10514));
};

For our custom filter, we'll start with a simple example that lets messages pass only if their severity level is WARNING or higher. This will exclude messages that are intended as debug- or informational-level logs, as well as those considered normal but significant. These settings will not be appropriate for every environment, but the syslog-ng filter syntax is powerful enough to allow for considerable tweaking. Here is our sample filter definition, which we'll place just below the existing f_cron filter in syslog-ng.conf:

filter f_opennms {
         level ( warning..emerg );
};

Finally, we bring together the network message source, the OpenNMS log destination and the custom filter in a log action definition at the bottom of syslog-ng.conf:

log {
   source(s_net);
   filter(f_opennms);
   destination(d_opennms);
};

Moving now to the OpenNMS server, we'll need to enable the syslog daemon since it's turned off by default. Edit /opt/opennms/etc/service-configuration.xml and uncomment the <service> XML tag for Syslogd. The default version of this file has several daemons on either side of Syslogd, all surrounded by a single set of XML comment tags, which look like <!-- for a start-comment tag and -->> for an end-comment tag. By adding a close-comment tag above the <service> tag for Syslogd and an open-comment tag below the corresponding </service> tag, you'll have uncommented the correct set of lines. When you’re done, it should look something like the following text, with your additions in boldface:

...
    <invoke at="status" pass="0" method="status"/>
    <invoke at="stop" pass="0" method="stop"/>
  </service>
  -->
  <service>
    <name>OpenNMS:Name=Syslogd</name>
    <class-name>org.opennms.netmgt.syslogd.jmx.Syslogd</class-name>
    <invoke at="start" pass="0" method="init"/>
    <invoke at="start" pass="1" method="start"/>
    <invoke at="status" pass="0" method="status"/>
    <invoke at="stop" pass="0" method="stop"/>
  </service>
  <!--
  <service>
    <name>OpenNMS:Name=Xmlrpcd</name>

One more OpenNMS configuration file edit is needed to make the syslog receiver work with the message format that syslog-ng uses. Open /opt/opennms/etc/syslogd-configuration.xml and change the forwarding-regexp, matching-group-host, and matching-group-message attributes of the top-level <configuration> element from the default:

forwarding-regexp="^.*\s(19|20)\d\d([-/.])(0[1-9]|1[012])\2(0[1-9]|[12][0-9]|3[01])(\s+)(\S+)(\s)(\S.+)"

matching-group-host="6"

matching-group-message="8"

To use a much more straightforward regular expression:

forwarding-regexp="^((.+?) (.*))\r?\n?$"

matching-group-host="2"

matching-group-message="3"

Save the file and restart OpenNMS to bring up Syslogd:

service opennms restart

As syslog messages begin arriving at the OpenNMS server, you'll begin to see events with uniform event identifiers (UEIs) starting with uei.opennms.org/syslogd/ followed by the facility name (e.g. “kernel,” “mail,” etc.) and severity. The original syslog message is visible in the event details.


Figure 1: An authentication error event in OpenNMS created from a remote syslog message (click to enlarge)

Just like any other OpenNMS event, it's easy to configure notifications to be sent via email, XMPP and other methods when particular syslog events arrive. Because all events created from syslog messages with the same facility and severity have the same UEI, the parameter-matching facility of the notification wizard comes in very handy. Every syslog-derived event in OpenNMS has an event parameter called syslogmessage that carries the original log message. If a tilde (~) is the first character of the parameter-match value, OpenNMS will evaluate the match as a regular expression rather than a literal string match.


Figure 2: Configuring notifications to be sent for syslog-derived OpenNMS events (click to enlarge)

Future releases of OpenNMS will come with a growing number of specific syslog event definitions that match commonly encountered log messages and have their own UEIs. Many of these events also carry alarm data that enables OpenNMS to reduce, or de-duplicate, many events to a single alarm. If you're comfortable crafting your own regular expressions, you can make your own syslog event definitions as well. In addition to giving your events distinct UEIs, you can extract subexpressions into individual event parameters. This facility affords power and flexibility on par with similar features in proprietary management platforms, such as the syslogd probe in IBM's Tivoli Netcool OMNIbus.

This tip touched on the use of syslog-ng along with OpenNMS' syslog message receiver to turn largely unstructured log messages into well-structured events, notifications and alarms. The next two tips in this series will introduce and look more deeply at another piece of free software, the Net-SNMP management agent, which makes a powerful companion to OpenNMS.

This was first published in March 2011

There are Comments. Add yours.

 
TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.