File monitoring via syslog-ng

• Syslog-ng is an open source implementation of the syslog protocol.
• It has several additional functionalities when compared to syslog including filters, flexible configuration, TPC support, SSL, and more. 
• It usually consists of a configuration file (usually /etc/syslog-ng/syslog-ng.conf) and a directory (usually /etc/syslog-ng/conf.d/) to store the filters and templates for processing syslog-ng rules in a structured form and separated by files.

This article describes how to manually configure syslog-ng to monitor and process log files from applications installed in the system and send them to Devo via syslog-ng.

Configuration 

This is an extract of /etc/syslog-ng/syslog-ng.conf, and an example of the syslog-ng template to process a log file. Note that this configuration is valid for syslog-ng version 2.0 and later.

source s_myfile {
    file("/path/to/file.log" follow_freq(1) flags(no-parse));
};
 
 
destination d_devo_myfile {tcp("DEVO-RELAY" port(PORT)
                            template("<$PRI>$DATE $HOST my.devo.tag: $MESSAGE\n"));};
 
 
log { source(s_myfile); destination(d_devo_myfile); };

You need to set some variables in the template to suit your environment, then save the file.

• Replace DEVO-PORT RELAY and PORT for the server and port of the Devo relay. Go to Administration → Relays in Devo to see a list of available relays.

• Replace my.devo.tag with the tag used to classify the log type in Devo (see List of supported technologies).

 The following configuration file is monitoring several log files of an Apache Server:

# Apache access log
source s_apache_access {
    file("/var/log/apache2/access.log" follow_freq(1) flags(no-parse));
};
destination d_devo_apache_access {tcp("DEVO-RELAY" port(PORT)
                            template("<$PRI>$DATE $HOST web.apache.access-combined.pro.webFoobar.www1: $MESSAGE\n"));};
log { source(s_apache_access); destination(d_devo_apache_access); };
 
# Apache SSL access log
source s_apache_ssl_access {
    file("/var/log/apache2/ssl_access.log" follow_freq(1) flags(no-parse));
};
destination d_devo_apache_ssl_access {tcp("DEVO-RELAY" port(PORT)
                            template("<$PRI>$DATE $HOST web.apache.access-combined.pro.webFoobar-ssl.www1: $MESSAGE\n"));};
log { source(s_apache_ssl_access); destination(d_devo_apache_ssl_access); };
 
# Apache error log
source s_apache_error {
    file("/var/log/apache2/error.log" follow_freq(1) flags(no-parse));
};
destination d_devo_apache_error {tcp("DEVO-RELAY" port(PORT)
                            template("<$PRI>$DATE $HOST web.apache.error.pro.webFoobar.www1: $MESSAGE\n"));};
log { source(s_apache_error); destination(d_devo_apache_error); };
 
# Apache SSL error log
source s_apache_ssl_error {
    file("/var/log/apache2/ssl_error.log" follow_freq(1) flags(no-parse));
};
destination d_devo_apache_ssl_error {tcp("DEVO-RELAY" port(PORT)
                            template("<$PRI>$DATE $HOST web.apache.error.pro.webFoobar-ssl.www1: $MESSAGE\n"));};
log { source(s_apache_ssl_error); destination(d_devo_apache_ssl_error); };

Ensure that both the file and the directory where it resides can be read by the user running syslog-ng (that is, syslog). If not, you need to grant ownership of the directory and the file to the syslog user.  

chow :syslog /var/log/apache2 /var/log/apache2/*.log

If the log files are part of a logrotate policy and logrotate create mode is being used, ensure that the syslog-ng user will still have permissions over the new file. For example, in Apache: 

/var/log/apache2/*.log {
        ...
        create 640 root syslog
        ...

Now restart syslog-ng:

/etc/init.d/syslog-ng restart

Once restarted, open Devo and go to Data Search to locate the table associated with the tag you used (for example, web.apache.access-combined).

• If you want to configure syslog-ng to use an encrypted and authenticated channel, see Secure sending via syslog-ng.
• If the system has SELinux enabled in enforcing mode (run the getenforce command to check the status) it may be necessary to add exceptions to the SELinux policy. See syslog & SELinux configuration for more information.

Log rotation

Here is an example of truncated logrotate configuration file.

/var/log/file.log
{
        rotate 12
        weekly
        copytruncate
        missingok
        notifempty
        compress
}

This is an example of a logrotate command that applies to several log files.

/var/logs/file.out
/var/logs/file.log
/var/logs/localhost.log
/var/logs/localhost_access_log.txt
{
        rotate 10
        daily
        copytruncate
        missingok
        notifempty
        compress
        lastaction
                service syslog-ng reload
        endscript
}

The last action directive reloads syslog-ng once after the rotation of all log files.