A lightweight central log management recipe

Home / A lightweight central log management recipe

To prepare pancakes (my favorite dish) you can find a lot of recipes on the internet. Some are more complicated to prepare other more easy, some tasty others not. My favor pancake recipe is really simple to prepare, it’s composed of only few basic ingredients and it tastes all. When introducing a central log management infrastructure to a productive system it’s also a question of ingredients driven by requirements you have to meet for your environment, business etc.

Alike for pancakes I like simple approaches to start small and grow as you go. Thus the underlying recipe for a central log management setup is quite pragmatic and easy to introduce. It consists of the following elements:

  1. Capturing logs on each component in the infrastructure and forwarding them in near real-time to a central log destination server using syslog as the most popular, simple and very fast log transport protocol.
  2. Storing received logs as plain files on the central log destination server and retaining them for some weeks.
  3. Enabling access to centrally stored log files using logsniffer. It provides an end to end web interface for parsing, viewing, tailing and searching log data.
  4. Monitoring logs for events of special interest with optional alerting to channels like mail, http etc.

 

central-log-management

Architecture for the central log management setup with logsniffer

Centralizing / forwarding logs

Your infrastructure will be usually composed of several components like application servers, web servers, databases etc. deployed in the cloud or on-premise on multiple nodes. Each generates logs which have to be centralized and forwarded to the dedicated central log destination server for further processing. The most popular log transport protocol is syslog. It’s the standard logging solution on Unix-like systems. Support for Windows platforms is also available. The modern syslog implementations like Rsyslog or syslog-ng are usually installed by default in common distributions of Linux. These support collecting logs from any source and delivering them to a wide variety of destinations in near real-time.

In my example I’ll focus on the syslog-ng open source product for forwarding logs to a remote server. To get started you have to install and configure syslog-ng on each log source system and on the central log destination server. On each log source node a corresponding configuration for collecting and forwarding logs has to be placed.

Example syslog-ng configuration for log source nodes

Usually you will have production systems configured to write logs to local files with activated log rotation. To avoid changes to that setup syslog-ng supports a nice feature to collect logs line per line from a plaint text file. The below configuration defines a file source s_myapp1_logfile for the live log file located under the path “/opt/myapp1/logs/all.log“. The file is checked every second for new input to transfer to the log server. To be able to separate received logs on the destination server according to the source the program field (part of the syslog protocol) is set in this example to the value “myapp1” explicitly.

The collected logs are forwarded by the configured destination d_central_log_server via network using the UDP networking protocol to the central log server. UDP is fast, but not reliable and not secure. If you like you can also use TCP or TLS. The last supports a secured communication between the log sources and the central log server which is really useful for cloud and non private network scenarios.

source s_myapp1_logfile {
    file(
       "/opt/myapp1/logs/all.log"
       follow-freq(1)
       program_override("myapp1")
       flags(no-parse)
    );
};
destination d_central_log_server {
     network(
        "192.168.0.100"
        transport("udp")
     );
};
log {
     source(s_myapp1_logfile);
     destination(d_central_log_server);
};

Adapt the above configuration to the logs you want to forward to the central log server and apply them to the syslog-ng daemon on the log source systems. You can do it usually by placing the config file to “/etc/syslog-ng/conf.d” and restarting the daemon by “/etc/init.d/syslog-ng restart“.

Note that logs on the source systems are only collected and forwarded to the central log server without parsing regarding the underlying logging format etc. It saves CPU time on production systems and reduces configuration efforts.

syslog-ng configuration for the central log server

The syslog-ng configuration on the central log server is quite simple. The forwarded logs are received via UDP and stored in the form they were captured from the original log line per line to files as defined in the d_file destination. In the underlying config the incoming logs are split into separate files according to the source host and the program they come from. By adding the current year, month and day macro to the path for the target log file we can achieve a simple and very useful rotation for the transferred logs. If you remember in the syslog-ng configuration on the source host the program field was explicitly set to “myapp1”. Thus the logs from host “host1” are stored on 2015-10-08 to “/var/log/central/host1/myapp1-2015-10-08.log”.

source s_net {
     network(
          ip("0.0.0.0")
          transport("udp")
     );
};
destination d_file {
     file (
          "/var/log/central/$HOST/$PROGRAM-$YEAR-$MONTH-$DAY.log" perm(0775) dir_perm(0775) create_dirs(yes)
          template("${MSG}\n")
          template-escape(no)
     );
};
log {
     source(s_net);
     destination(d_file);
};

Last but not least you have to care about the retaining strategy for the transferred logs to don’t run out of space on the central log server. The fact of daily rotated log files makes it easy to achieve it by deleting logs older then XX days using a simple cron job.

Configuring logsniffer

Once the syslog-ng configuration is applied and logs have been transferred and stored on the central log server to files it’s time to provide a web access to them as well as further features like tailing, searching, monitoring, aggregating and etc. And here logsniffer comes into play. It provides a smart web interface to serve the stored log files to the user.

logsniffer can be installed directly on the central log server with file access to the transferred logs. It’s lightweight, has a small footprint and install-able in a minute. Just follow the setup instructions on the download page.

Registering and parsing logs

In logsniffer the stored log files have to be registered as so called “log sources”. A log source accesses the log data from it’s physical medium like files, database etc. and parses it as log entries, which will be displayed in the viewer, monitored and analysed. A log source is defined and configured flexibly on logsniffer’s web interface. The configuration addresses typically the following aspects:

  • Localization of the physical log data e.g. file(s), database etc. and how this can be accessed. In underlying case we deal with daily rotated files.
  • A reader configuration for parsing the raw log format into log entries composed of fields.
  • Several filters to be able to post-process the parsed fields. Example: Type conversion from string to date format for fields with date information.

For the current scenario we are going to create for each application log sequence a log source in logsniffer. The log source should be of type “Rolling log file with dynamic live file name”. This supports composition of rotated files into a single continuous log. Just configure in the creation wizard the path pattern referencing the stored log files and continue with the reader configuration responsible for parsing logs according the specific log format. To get started quickly you can use the Regex pattern reader with the simplest pattern “(?<myField>.+)” which matches every log line and assigns it to the field “myField”. The wizard settings could look as follows:

Log source wizard

Main log source wizard

Reader wizard for parsing

Reader wizard for parsing the log content

Tailing, scrolling and searching through logs

After setting up the log sources the log files can be accessed in real-time using a browser. logsniffer operates and navigates directly on the stored files for use cases like viewing, searching, tailing and monitoring. This fact enables to process logs at incredible speeds. Take a look at the below video to get an impression of logsniffer’s log viewer capabilities.

Monitoring logs for events

Monitoring of logs is another important aspect in the central log management area. In logsniffer so-called sniffers can be created and configured flexibly to scan the transferred logs continuously for entries of the specific interest. Detected events are stored in an Elasticsearch index for further retrieval and published optionally to configured channels like e-mail, http targets etc.

In the following example I like to show a setup for monitoring the logs of “myapp1” for the restart message. The monitoring goals are:

  • To receive an e-mail alert each time the application get restarted
  • To get a report about frequency of restarts over time

When creating a sniffer you have first to define the log source to be scanned and the interval the sniffer should be scheduled. Furthermore you have to define how much of the log should be scanned consecutively in each iteration:

Main monitoring settings

Main monitoring settings

On the “scanner” tab the search criteria is configured. In this case I use the RegEx scanner to filter for log entries matching the regular expression “Started @(?<duration>\d+)ms$” for texts like “Started @8956ms” in the message field. For the server startup duration I use a named group in the pattern to let extract the duration value to a field named “duration”. It will be used later in the mail template.

Scanner settings to search for start message

Scanner settings to search for the start message

Last but not least a mail notification publisher is configured for alerting via e-mail each time a restart event get identified. The mail content configured via a Velocity template is quite simple containing only the link to the event, the timestamp of the restart and the needed startup duration.

Mail publisher config to get notified about restarts

Mail publisher config to get notified about restarts

An example e-mail I got this way looks like:

Example alert e-mail

Example alert e-mail

All detected restart events are stored in an embedded or a remote Elasticsearch index and available for tracing and further analysis on the events page.

All monitored restart events in a histogram

All monitored restart events in a histogram

Summary

The distinguishing features of the presented central log management setup can be summarized as follows:

  • No significant loss of performance or availability for the production systems generating the logs due to
    • using the lightweight and at the same time highly performant syslog-ng log forwarding via UDP,
    • no need for log parsing on the source system.
  • No change to log appenders on the source systems – the current log files are grabbed and streamed out in real-time
  • Simple retaining strategy for over several months of log data by just providing the appropriated disk space
  • No lock-in regarding the log persistence on the central log server due to using plain files
  • Harmonization of log data by parsing the specific log format
  • Use case specific parsing and extraction of values in log messages
  • Real-time tailing, searching and monitoring of logs
  • Only one extra server with basic hardware requirements is needed to use as central log server

As you can see, the combination of syslog-ng and logsniffer can provide you with a simple and highly efficient architecture for centralizing and processing logs to be used by DevOps teams enabling them to build and operate great products.

By the way, if somebody is interested in my favor pen cake recipe just drop me a line at @mboook.

About logsniffer

Inspired by shell tools like less, tail and grep we built logsniffer with the vision to make logs accessible in a same way from a central, smart web interface. logsniffer empowers DevOps teams to build and operate great products. It's free, open source and runs in a minute on premise.