- Getting started
- Architecture
- Administration
- User interface
-
System configuration
- Installation of software packages
-
Relays
- The In-house Relay
- In-house Relay rules
- In-house Relay configuration
- In-house Relay installation using a virtual machine
- In-house Relay installation using the software package
- Installing the USA relay on an Ubuntu 16 VM server
- CentOS relay installation
- Relay installation in any Linux distribution
- High-availability relay
- In-house Relay troubleshooting
- Sending the data
- Supported technologies
-
Data Search
- Running a search
- LINQ
- Viewing the data tables
- Viewing column info
- Running queries (tutorials)
- Last queries
- Query management
- Lookup management
- Favorite queries
- Sharing queries
- Table toolbar features
-
Additional tools
- Dashboard data source
-
Charts
- Affinity chord diagram
- Availability timeline
- Bipartite chord diagram
- Bubble chart
- Chart aggregation
- Custom date chart aggregation
- Flame graph
- Flat world map by coordinates
- Flat world map by country
- Google animated heat map
- Google area map
- Google heat map
- Graph diagram
- Histogram
- Pie chart
- Pie layered chart
- Punch card
- Sankey diagram
- Scatter plot
- Time heatmap
- Voronoi treemap
- Graphical correlation
- Query Info
- Custom tables
- Aliased finder
- Custom finder
- Data reinjection
- Available operations
- Best practices for data search
- Alerts management
-
Dashboards
- Setup a data source
- Create a new dashboard
-
Working with dashboard widgets
- Availability timeline widget
- Chord diagram widget
- Circle world map widget
- Color key value widget
- Color world map widget
- Column chart widget
- Comparative chart widget
- Funnel widget
- Gauge meter widget
- Google heatmap widget
- Heat calendar widget
- Line chart widget
- Monitoring widget
- Pie chart widget
- Punch card widget
- Sectored pie chart widget
- Table widget
- Time heatmap widget
- Tree diagram widget
- Voronoi tree widget
- Configuring and sharing dashboards
- API
- Use cases
System configuration / Sending the data / Sending from Unix-based operating systems / File monitoring via syslog-ng
Download as PDFFile monitoring via syslog-ng
- Syslog-ng is an open source implementation of the syslog protocol.
- It has several additional functionalities when compared to syslog including filters, flexible configuration, TPC support, SSL, and more.
- It usually consists of a configuration file (usually /etc/syslog-ng/syslog-ng.conf) and a directory (usually /etc/syslog-ng/conf.d/) to store the filters and templates for processing syslog-ng rules in a structured form and separated by files.
This article describes how to manually configure syslog-ng to monitor and process log files from applications installed in the system and send them to Devo via syslog-ng.
For the primary Linux distributions, this configuration can be done automatically.
Configuration
This is an extract of /etc/syslog-ng/syslog-ng.conf, and an example of the syslog-ng template to process a log file. Note that this configuration is valid for syslog-ng version 2.0 and later.
source s_myfile {
file("/path/to/file.log" follow_freq(1) flags(no-parse));
};
destination d_devo_myfile {tcp("DEVO-RELAY" port(PORT)
template("<$PRI>$DATE $HOST my.devo.tag: $MESSAGE\n"));};
log { source(s_myfile); destination(d_devo_myfile); };
You need to set some variables in the template to suit your environment, then save the file.
Replace DEVO-PORT RELAY and PORT for the server and port of the Devo relay. Go to Administration → Relays in Devo to see a list of available relays.
- Replace my.devo.tag with the tag used to classify the log type in Devo (see List of supported technologies).
The following configuration file is monitoring several log files of an Apache Server:
# Apache access log
source s_apache_access {
file("/var/log/apache2/access.log" follow_freq(1) flags(no-parse));
};
destination d_devo_apache_access {tcp("DEVO-RELAY" port(PORT)
template("<$PRI>$DATE $HOST web.apache.access-combined.pro.webFoobar.www1: $MESSAGE\n"));};
log { source(s_apache_access); destination(d_devo_apache_access); };
# Apache SSL access log
source s_apache_ssl_access {
file("/var/log/apache2/ssl_access.log" follow_freq(1) flags(no-parse));
};
destination d_devo_apache_ssl_access {tcp("DEVO-RELAY" port(PORT)
template("<$PRI>$DATE $HOST web.apache.access-combined.pro.webFoobar-ssl.www1: $MESSAGE\n"));};
log { source(s_apache_ssl_access); destination(d_devo_apache_ssl_access); };
# Apache error log
source s_apache_error {
file("/var/log/apache2/error.log" follow_freq(1) flags(no-parse));
};
destination d_devo_apache_error {tcp("DEVO-RELAY" port(PORT)
template("<$PRI>$DATE $HOST web.apache.error.pro.webFoobar.www1: $MESSAGE\n"));};
log { source(s_apache_error); destination(d_devo_apache_error); };
# Apache SSL error log
source s_apache_ssl_error {
file("/var/log/apache2/ssl_error.log" follow_freq(1) flags(no-parse));
};
destination d_devo_apache_ssl_error {tcp("DEVO-RELAY" port(PORT)
template("<$PRI>$DATE $HOST web.apache.error.pro.webFoobar-ssl.www1: $MESSAGE\n"));};
log { source(s_apache_ssl_error); destination(d_devo_apache_ssl_error); };
Ensure that both the file and the directory where it resides can be read by the user running syslog-ng (that is, syslog). If not, you need to grant ownership of the directory and the file to the syslog user.
chow :syslog /var/log/apache2 /var/log/apache2/*.log
If the log files are part of a logrotate policy and logrotate create mode is being used, ensure that the syslog-ng user will still have permissions over the new file. For example, in Apache:
Configuration file /etc/logrotate.d/apache2 extract
/var/log/apache2/*.log {
...
create 640 root syslog
...
Now restart syslog-ng:
/etc/init.d/syslog-ng restart
Once restarted, open Devo and go to Data Search to locate the table associated with the tag you used (for example, web.apache.access-combined).
- If you want to configure syslog-ng to use an encrypted and authenticated channel, see Secure sending via syslog-ng.
- If the system has SELinux enabled in enforcing mode (run the getenforce command to check the status) it may be necessary to add exceptions to the SELinux policy. See syslog & SELinux configuration for more information.
Log rotation
Here is an example of truncated logrotate configuration file.
/var/log/file.log
{
rotate 12
weekly
copytruncate
missingok
notifempty
compress
}
This is an example of a logrotate command that applies to several log files.
/var/logs/file.out
/var/logs/file.log
/var/logs/localhost.log
/var/logs/localhost_access_log.txt
{
rotate 10
daily
copytruncate
missingok
notifempty
compress
lastaction
service syslog-ng reload
endscript
}
The last action directive reloads syslog-ng once after the rotation of all log files.