Rsyslog log forwarding: Best practices and implementation with New Relic

Centralizing and managing your log data can be extremely challenging, especially when you are collecting logs from many sources. Traditional logging solutions are often hard to implement, requiring multiple technologies and dedicated infrastructure specialists. Open source tools like rsyslog and syslog-ng can make it much easier to forward your logs to centralized locations like New Relic. In this blog, you’ll get hands-on and learn about the following:

• The basics of syslog, syslog-ng, and rsyslog
• How to use rsyslog to forward logs to New Relic
• How to configure tailing specific log files
• How to use rsyslog to collect logs from networking and security devices
• How to use disk-assisted queuing for reliability

What is the difference between syslog, syslog-ng, and rsyslog?

Understanding Syslog

System Logging Protocol (or syslog) is a standard protocol used to send system log or event messages to a specific server, called a syslog server. It’s primarily used to collect various device logs from several machines in a central location for monitoring and review. Engineering teams have been using the syslog protocol for decades to transport messages. Due to its longevity and popularity, most major operating systems, including macOS, Linux, and Unix, support the syslog protocol.

What is Syslog-ng?

Syslog-ng is a free and open source implementation of the syslog protocol for Unix and Unix-like systems. It extends the original syslog model with content-based filtering, rich filtering capabilities, flexible configuration options, and adds important features to syslog, such as using TCP for transport. Syslog-ng is a very popular tool with many contributors. It’s widely used because of its flexibility and its simplicity to set up and configure.

What is Rsyslog?

Rsyslog (for “rocket-fast” syslog) is an open source, Unix-based utility for collecting, transforming, filtering, and routing log messages in an IP network—similar to open source log forwarders like Fluent Bit. Available since 2004, rsyslog’s present-day popularity can be attributed, in part, to its ubiquity; it is the default syslog utility in both Ubuntu and Debian, and is available out-of-the-box from several Linux distributions.

As such, rsyslog is often the default when your team is looking to get their log management program up and running quickly. It is equally popular with security-minded teams who wish to avoid deploying third-party software to sensitive systems.

Rsyslog monitoring best practices

There are a lot of things to consider when monitoring rsyslogs. To start off a bit more high level, here are some basic best practices

• Centralize and aggregate log data from various sources into a dedicated log management system to streamline analysis and troubleshooting. 
• Ensure you configure rsyslog to rotate and compress log files regularly to manage storage efficiently. 
• Employ log forwarding mechanisms to reliably transport logs to the centralized system, ensuring data integrity and availability. 
• Implement structured log formats, such as JSON, to make parsing and analysis more straightforward. 
• Lastly, set up alerting and monitoring for log anomalies and critical events to proactively detect and address issues. 

Adhering to these practices can help maintain efficient, reliable, and actionable log monitoring with rsyslogs. These are a bit more high level, but let us walk you through some more detailed instructions for specific use cases within New Relic.

How to get started with rsyslog

This example covers forwarding logs to New Relic from a new Ubuntu machine on AWS running rsyslog.

• Install the following packages to allow rsyslog to send logs over an encrypted connection:

sudo apt-get install rsyslog-gnutls ca-certificates

• Next, create a text file in /etc/rsyslog.d/ called newrelic.conf. Add the following to your newly created text file, making sure to replace YOUR_NR_INSERT_KEY with your New Relic Event API key.

•  Restart rsyslog to ensure your changes are applied:

sudo service rsyslog restart

Finally, you can write messages directly to syslog to validate your configuration:

logger "hello world"

Configuring specific logs

The steps above provide a simple configuration that will forward any logs that rsyslog currently collects to New Relic. Let’s look at how to configure tailing specific log files.

• Enable the imfile module to allow rsyslog to tail files. Add the following to the Modules section of your /etc/rsyslog.conf if it isn’t already present:

$ModLoad imfile

• Define the files that rsyslog should tail by adding the following to the top of the newrelic.conf file that you created in the previous section:

Note: Depending on your version of rsyslog, wildcards are supported when defining $InputFileName.

•  Restart rsyslog and check your New Relic account for logs:

sudo service rsyslog restart

You might notice that when using the above configuration, rsyslog will send both system logs from the host it’s running on and the logs it has been configured to tail. Luckily, rsyslog makes it easy to keep only the logs you want using filter conditions and control structures.

For example, in the newrelic.conf file below, only the logs read from /var/log/apache2/access.log will be sent to New Relic:

A simpler way to monitor network and security device logs

Rsyslog is often used to collect logs from networking and security devices like firewalls, most of which have a built-in syslog module. These devices’ syslog capabilities tend to be limited; you cannot install agents on them, and data transformation is nearly impossible.

Rsyslog bridges the gap by providing the ability to receive syslog traffic from network devices, transform logs as needed, and then send them on to a custom destination, such as New Relic.

We’ll begin by enabling the imudp module, which allows rsyslog to listen for incoming UDP traffic.

• Enable imudp by adding the following to the Modules section of your /etc/rsyslog.conf

$ModLoad imudp

•  Navigate to /etc/rsyslog.d/, create a new file named newrelic-firewall.conf, and add the following, making sure to replace YOUR_INSERT_KEY with your New Relic API key:

With the above configuration, any logs you send to our host with a program name starting with %ASA will be sent to New Relic.

Using disk-assisted queuing for reliability

So far, we’ve covered how to ingest log data from a few different sources using rsyslog. The next step is to ensure that the ingested data arrives safely at its destination, which is where queues come in.

Rsyslog offers four queue modes: direct, disk, in-memory, and disk-assisted. The disk-assisted queue mode is perhaps the most popular, as it combines memory and disk queueing to ensure both speed and reliability.

To enable disk-assisted queuing, edit your /etc/rsyslog.conf file or create a file with the following content in the /etc/rsyslog.d/ directory:

As always, restart rsyslog to ensure your changes are applied.