Using filters to separate log messages in rsyslog

Using filters to separate out syslog messages into separate log files is a useful way to organize system logs. You can define different log retention policies for each of those log message files. I will share the absolute essentials of how to do this.

Background

syslog messaging is the age old way of writing log messages in UNIX systems. Now syslog messaging is supported in almost all UNIX-like systems (Linux/FreeBSD/OpenBSD/etc.). Basic unit of logging is called a record. Each record contains a few properties:

  1. Timestamp
  2. Hostname
  3. Process name that emits the log message
  4. Syslog facility/level (think of it like log level)
  5. Message

syslog messages can be emitted to a syslog daemon running in local host. This daemon can collect and write logs to the disk or it can in turn forward the messages to another syslog daemon running in another server. The transport used to send the log messages can be unix socket (if daemon is running locally) or UDP (if daemon is running remote). You can also configure to send syslog messages over TLS while sending across machines. TLS will provide added security by encrypting the messages.

Sending messages

To send syslog meesages from command line, you can use logger.

Depending on the programming language you use, you may already have support for sending the syslog messages. For insance, C language standard library has syslog functions. If you are using SLF4j, you can use SyslogAppender.

syslog daemon

syslog is a specification of the format of log messages and how to send them. There was one implmentation initilly available that was called syslogd. Later there were multiple implementations available due to the performance and flexibility restrictions in the original syslogd. Two of those implementations are quite popular: rsyslog and syslog-ng. You can read about the difference between those two here. Rest of this article assumes you are using rsyslogd.

Filtering configuration

All these configurations go into your /etc/rsyslog.conf or in one of the included files (typically from /etc/rsyslog.d/*.conf).

There are two ways you can specify filter expressions in rsyslogd conf file.

  • Property based filters - These filters start with a ":" followed by a property.
  • Expression based filters - These filters start with "if" at the beginning.

Generally I would recommend expression based filters as they are a bit easier to read. Please see the examples below.

Properties

In both property and expressions based filters, you will refer to properties in the syslog records. Some of the frequently used properties are given below. For the full list refer here.

msg
hostname
source
fromhost
fromhost-ip
syslogtag
programname
pri
pri-text

Compare operations

Following are the operations that are allowed in the filters:

contains
contains_i # Ignores case
isequal
startswith
startswith_i # Ignores case
regex
ereregex

For the full list, see here.

Action

This is the last field in the filter expressions. What is the action to be taken when the expression matches. Typical actions include:

  • You can specify an absolute path to a file name. The log message will be written there.
  • You can specify destination host using @ (for UDP) or @@ (for TCP). Messages will be forwarded there.
  • You can specify the discard operation using "~" or the word "stop". No further processing of the message will be done.

Gotchas

Please note that the filter expressions are applied one by one until discard action is encountered. This is very important to keep in mind. Unless you want your matches to be sent to multiple destinations, you must always specify discard action.

There is a default rule in /etc/rsyslog.d/50-rsyslog.conf file that matches all messages and log them to /var/log/syslog. So you must have your filter config file named such that it is loaded before /etc/rsyslog.d/50-rsyslog.conf. Typically a name like /etc/rsyslog.d/40-custom.conf will achieve this effect.

Examples

# When the message contains string "mmdb", write it in a separate log file.
# The second line with "~" is important. It says, discard the message 
# after writing to/var/log/mmdb.log. Otherwise the same message will appear 
# in /var/log/syslog also.
:msg, contains, "mmdb" /var/log/mmdb.log
:msg, contains, "mmdb" ~

# A concise way of writing the same expression is to make use of "&" in the second line.
# It means "after matching the expression in previous line, do this as well."
:msg, contains, "mmdb" /var/log/mmdb.log
& ~

# Equivalent of the above filter in expression form:
if $msg contains "mmdb" then /var/log/mmdb.log
if $msg contains "mmdb" then ~

# Or we can use blocks to write both actions.
if $msg contains "mmdb" then {
    /var/log/mmdb.log
    stop
}

# Or you can separate out messages by name of the program that sends it.
if $programname isequal "httpd" then {
    /var/log/httpd.log
    stop
}

References

links

social