File monitoring via rsyslog

  • Rsyslog is the default syslog package that is commonly used by Linux distributions.
  • It usually consists of a configuration file (usually /etc/rsyslog.conf) and a directory (usually /etc/rsyslog.d/) to store the filters and templates for processing rsyslog rules in a structured form and separated by files.  

This article describes how to manually configure rsyslog to monitor and process log files and send them to Devo via rsyslog.

For the primary Linux distributions, this configuration can be done automatically.

Configuration

This is an example of an rsyslog template used for processing log files: 

/etc/rsyslog.d/45-filemonitor.conf file monitoring example

$template myFileMonitorTemplate,"<%PRI%>%timegenerated% %HOSTNAME% %syslogtag% %msg%"

# File access
$InputFileName /path/to/file.log
$InputFileTag my.devo.tag:
$InputFileStateFile stat-file1-myFileMonitor
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor

if $syslogtag contains 'my.devo.tag' and $syslogfacility-text == 'local7' then @@DEVO-RELAY:PORT;myFileMonitorTemplate
:syslogtag, contains, "my.devo.tag" ~

The 00-logtrust.conf file described in Sending via rsyslog must be preloaded (since it depends on the imfile module and the variable $WorkDirectory, it must be defined).

Variables in the templatedevo

You need to set the variables in the template to suit your environment, then save the file.

$InputFileName The path and filename where the log file is located.
$InputFileTag The tag used in Devo to classify the log type. 
$InputFileStateFile The unique state file name of the file that contains a copy of the information that has been processed. In case of a rsyslog crash or reboot, the data in this file will not be processed in order to avoid duplication.
$InputFileSeverity / $InputFileFacility Together, these define the syslog priority level you assign to this file.
$InputFilePollInterval The interval in seconds between checking the file for new data.
$InputFilePersistStateInterval The number of events after which the processing status should be updated.
If $syslogtag contains 'my.devo.tag' … This line instructs rsyslog to send all incoming events with the ‘my.devo.tag’ tag via TCP to ‘DEVO-RELAY:PORT’ using the log format defined in ‘myFileMonitorTemplate in the configuration file header.  
:syslogtag, contains, 'my.devo.tag.' ~ This line is used to remove the message of rsyslog processing chain so that it cannot be processed by other configuration files.

The configuration file below is an example of how to monitor multiple log files of an Apache server. Please note that you can process up to a limit of 100 log files on a rsyslog default compilation.

File /etc/rsyslog.d/45-apache.conf monitoring example

$template apache,"<%PRI%>%timegenerated% %HOSTNAME% %syslogtag% %msg%"

# File1 access
$InputFileName /var/log/apache2/access.log
$InputFileTag web.apache.access-combined.pro.webFoobar.www1:
$InputFileStateFile stat-file1-ApacheAccess
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor

# File2 access
$InputFileName /var/log/apache2/ssl_access.log
$InputFileTag web.apache.access-combined.pro.webFoobar-ssl.www1:
$InputFileStateFile stat-file2-ApacheAccess
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor

# File1 Error
$InputFileName /var/log/apache2/error.log
$InputFileTag web.apache.error.pro.webFoobar.www1:  
$InputFileStateFile stat-file1-ApacheError
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor

# File2 Error
$InputFileName /var/log/apache2/ssl_error.log
$InputFileTag web.apache.error.pro.webFoobar-ssl.www1:  
$InputFileStateFile stat-file2-ApacheError
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor

if $syslogtag contains 'web.apache.' and $syslogfacility-text == 'local7' then @@DEVO-RELAY:PORT;apache
:syslogtag, contains, "web.apache." ~

Ensure that both the file and the directory where it resides can be read by the user running rsyslog (that is, syslog). If not, you need to grant ownership of the directory and the file to the syslog user. 

chown :syslog /var/log/apache2 /var/log/apache2/*.log

If the log files are part of a logrotate policy and logrotate create mode is being used, ensure that syslog will still have permissions over the new file. For example, in Apache: 

/etc/logrotate.d/apache2 configuration file extract

/var/log/apache2/*.log {
        ...
        create 640 root syslog
        ...

Now restart syslog:

/etc/init.d/rsyslog restart

Once restarted, open Devo and go to Data Search to locate the table associated with the tag you used (for example, web.apache.access-combined).

Have we answered your question?

If not, please contact our technical support team via email by clicking the button below.

CONTACT US