• Services & Support
  • Devo.com
  • Contact
    • Contact Us
    • Request a Demo
    • Partner Inquiry
  • Log In
    • USA Devo
    • EU Devo
  • Getting started
    • Concepts
    • Devo video tutorials
  • Architecture
    • Deployment models
  • Administration
    • Users administration
      • Add a new user
    • User roles
      • Create custom roles
    • Credentials
  • User interface
    • Panels
    • Intro to Administration
      • Data Management
    • Intro to Applications
    • Notifications
    • Preferences
      • SAML
        • Google as an identity provider
        • Okta as an identity provider
        • OneLogin as an identity provider
      • Multi-factor authentication
    • Social Intelligence
  • System configuration
    • Installation of software packages
      • Ubuntu
      • Debian
      • CentOS V5-6
        • How to monitor files in CentOS using wildcards
      • Fedora
      • RHEL
    • Relays
      • The In-house Relay
      • In-house Relay rules
      • In-house Relay configuration
      • In-house Relay installation using a virtual machine
      • In-house Relay installation using the software package
      • Installing the USA relay on an Ubuntu 16 VM server
      • CentOS relay installation
      • Relay installation in any Linux distribution
      • High-availability relay
        • Helpful tips and commands
      • In-house Relay troubleshooting
    • Sending the data
      • Sending from Unix-based operating systems
        • Agent configuration for Unix systems
        • File monitoring via rsyslog
        • File monitoring via syslog-ng
        • Secure sending via rsyslog
        • Secure sending via syslog-ng
        • Sending via rsyslog
        • Sending via syslog-ng
        • Sending via traditional syslog
        • Syslog & SELinux configuration
      • Sending from Windows operating systems
        • Devo Agent for Windows events sending
          • Activating WMI
        • MagicLog
          • MagicLog installation
          • MagicLog configuration
        • ProxyServerContainer
          • ProxyServerContainer installation
          • ProxyServerContainer configuration
        • MonitorService
        • Windows Snare agent
        • Integration between Windows Azure monitoring and Devo
      • Sending from MacOS X
      • Sending from your Java application
        • JDK java.util.logging
        • Scoja client library
      • Data upload
      • Sending logs from AWS S3
      • Sending the data using Logstash
        • Sending directly to Devo
          • Configuration file examples
            • Querying databases
            • Read data from keyboard
            • Reading a file
            • Read data from Kafka
        • Sending from an In-house Relay
      • Sending logs using OPSEC LEA
      • HTTP sending
  • Supported technologies
    • Introduction to tags
    • List of supported technologies
      • Antivirus
        • av.mcafee
      • Box (PC/Server)
        • box.iptables
        • box.stat
        • box.unix
        • box.vmware
        • box.win
      • Firewall
        • firewall.checkpoint
        • firewall.cisco
        • firewall.fortinet
        • firewall.huawei
        • firewall.juniper
        • firewall.meraki
        • firewall.paloalto
        • firewall.pfsense
        • firewall.sonicwall
        • firewall.sophos
        • firewall.stonegate
        • firewall.windows
      • Network
        • netstat.netflow
        • switch.cisco
      • Proxy
        • proxy.bluecoat
        • proxy.squid
      • Web
        • web.apache
        • web.apache.mod-security
        • web.iis
        • web.jboss
        • web.nginx
        • web.tomcat
      • Other technologies
        • db.mysql
        • dns.bind
        • edr.cylance
        • my.app
        • social.salesforce
        • test.drop
        • test.keep
        • uba.varonis
        • unknown.unknown
  • Data Search
    • Running a search
    • LINQ
      • Performing operations using LINQ
      • LINQ query examples
    • Viewing the data tables
    • Viewing column info
    • Running queries (tutorials)
    • Last queries
    • Query management
    • Lookup management
      • Dynamic lookups
    • Favorite queries
    • Sharing queries
    • Table toolbar features
      • Time interval history
      • Edit the column layout
      • View selected events
      • Column operations
      • Toggle query editor
      • Toggle search tree
      • New alert definition
      • Aggregations
      • Group
      • Filters
      • Create column
      • Download
    • Additional tools
      • Dashboard data source
      • Charts
        • Affinity chord diagram
        • Availability timeline
        • Bipartite chord diagram
        • Bubble chart
        • Chart aggregation
        • Custom date chart aggregation
        • Flame graph
        • Flat world map by coordinates
        • Flat world map by country
        • Google animated heat map
        • Google area map
        • Google heat map
        • Graph diagram
          • Creating a graph diagram
          • Graph diagram menu
        • Histogram
        • Pie chart
        • Pie layered chart
        • Punch card
        • Sankey diagram
        • Scatter plot
        • Time heatmap
        • Voronoi treemap
      • Graphical correlation
        • Cross-Search Graph Diagram
        • Cross-Search Table Join
        • Cross-Search Sankey Diagram
        • Cross-Search Line Chart
      • Query Info
      • Custom tables
    • Aliased finder
    • Custom finder
      • Creating a custom finder
      • Assigning a custom finder to a role
      • Edition mode
    • Data reinjection
    • Available operations
      • Examples
      • Aggregation operations
      • Filtering operations
        • Order group (filter)
        • String group (filter)
        • General group (filter)
        • Name group (filter)
        • Network group (filter)
        • Logic group (filter)
        • Web group (filter)
      • Create column operations
        • Order group
        • Arithmetic group
        • String group
        • General group
        • Date group
        • Name group
        • Network group
        • Geolocation group
        • Logic group
        • Flow group
        • Web group
        • Mathematical group
        • Conversion group
        • Cryptography group
        • Packet group
          • Ethernet operations
          • IPv4 operations
          • TCP operations
          • UDP operations
    • Best practices for data search
  • Alerts management
    • How to set up a delivery method
      • Setting up an Email type delivery method
      • Setting up an HTTP-JSON type delivery method
      • Setting up a Service Desk type delivery method
      • Setting up a Jira type delivery method
      • Setting up a Pushover type delivery method
      • Setting up a PagerDuty type delivery method
    • Alert policies
    • Alerts dashboard
    • Alerts and correlation libraries
    • How to?
      • Create a new alert
      • Set up an inactivity alert
      • Modify an existing alert
      • Deactivate an alert
      • Create an alert over an alert
      • Create an alert annotation
      • Create a post-filter
      • Edit a query alert
      • Quick filter saved alerts
  • Dashboards
    • Setup a data source
    • Create a new dashboard
    • Working with dashboard widgets
      • Availability timeline widget
      • Chord diagram widget
      • Circle world map widget
      • Color key value widget
      • Color world map widget
      • Column chart widget
      • Comparative chart widget
      • Funnel widget
      • Gauge meter widget
      • Google heatmap widget
      • Heat calendar widget
      • Line chart widget
        • Customize your line chart
      • Monitoring widget
      • Pie chart widget
      • Punch card widget
      • Sectored pie chart widget
      • Table widget
      • Time heatmap widget
      • Tree diagram widget
      • Voronoi tree widget
    • Configuring and sharing dashboards
  • API
    • REST API v2
      • Authorization methods
      • How to query with API v2
        • Forwarding query responses to HDFS
        • Forwarding query responses to Kafka
        • Forwarding query responses to S3
        • Send requests with Postman
      • Job requests
    • Provisioning API
    • OData API feeds management
      • Connecting with Excel
      • Connecting with Tableau
      • Connecting with Power BI
  • Use cases
    • Uploading, analyzing and visualizing data
      • Uploading data
      • Finding data
      • Structuring data
      • Converting data types
      • Performing data analysis
      • Creating a chart
      • Creating a dashboard data source
      • Adding a widget to a dashboard
    • Installing, configuring and uploading data with a Devo relay
      • Installing a Devo relay
      • Configuring and activating the Devo relay
      • Devo relay rules overview
      • Sending data to the Devo cloud through a relay
    • Monitor intranet traffic to dangerous websites
    • Kaspersky integration
    • Monitoring Oracle databases
      • Audit log
      • Alert log
      • Listener log
      • Incident log
      • Trace log
      • Inventory events
      • Performance
PREVIOUS
File monitoring via rsyslog
NEXT
Secure sending via rsyslog

System configuration / Sending the data / Sending from Unix-based operating systems / File monitoring via syslog-ng

Download as PDF

File monitoring via syslog-ng

  • Syslog-ng is an open source implementation of the syslog protocol.
  • It has several additional functionalities when compared to syslog including filters, flexible configuration, TPC support, SSL, and more. 
  • It usually consists of a configuration file (usually /etc/syslog-ng/syslog-ng.conf) and a directory (usually /etc/syslog-ng/conf.d/) to store the filters and templates for processing syslog-ng rules in a structured form and separated by files.

This article describes how to manually configure syslog-ng to monitor and process log files from applications installed in the system and send them to Devo via syslog-ng.

For the primary Linux distributions, this configuration can be done automatically.

Configuration 

This is an extract of /etc/syslog-ng/syslog-ng.conf, and an example of the syslog-ng template to process a log file. Note that this configuration is valid for syslog-ng version 2.0 and later.

source s_myfile {
    file("/path/to/file.log" follow_freq(1) flags(no-parse));
};
 
 
destination d_devo_myfile {tcp("DEVO-RELAY" port(PORT)
                            template("<$PRI>$DATE $HOST my.devo.tag: $MESSAGE\n"));};
 
 
log { source(s_myfile); destination(d_devo_myfile); };

You need to set some variables in the template to suit your environment, then save the file.

  • Replace DEVO-PORT RELAY and PORT for the server and port of the Devo relay. Go to Administration → Relays in Devo to see a list of available relays.

  • Replace my.devo.tag with the tag used to classify the log type in Devo (see List of supported technologies).

 The following configuration file is monitoring several log files of an Apache Server:

# Apache access log
source s_apache_access {
    file("/var/log/apache2/access.log" follow_freq(1) flags(no-parse));
};
destination d_devo_apache_access {tcp("DEVO-RELAY" port(PORT)
                            template("<$PRI>$DATE $HOST web.apache.access-combined.pro.webFoobar.www1: $MESSAGE\n"));};
log { source(s_apache_access); destination(d_devo_apache_access); };
 
# Apache SSL access log
source s_apache_ssl_access {
    file("/var/log/apache2/ssl_access.log" follow_freq(1) flags(no-parse));
};
destination d_devo_apache_ssl_access {tcp("DEVO-RELAY" port(PORT)
                            template("<$PRI>$DATE $HOST web.apache.access-combined.pro.webFoobar-ssl.www1: $MESSAGE\n"));};
log { source(s_apache_ssl_access); destination(d_devo_apache_ssl_access); };
 
# Apache error log
source s_apache_error {
    file("/var/log/apache2/error.log" follow_freq(1) flags(no-parse));
};
destination d_devo_apache_error {tcp("DEVO-RELAY" port(PORT)
                            template("<$PRI>$DATE $HOST web.apache.error.pro.webFoobar.www1: $MESSAGE\n"));};
log { source(s_apache_error); destination(d_devo_apache_error); };
 
# Apache SSL error log
source s_apache_ssl_error {
    file("/var/log/apache2/ssl_error.log" follow_freq(1) flags(no-parse));
};
destination d_devo_apache_ssl_error {tcp("DEVO-RELAY" port(PORT)
                            template("<$PRI>$DATE $HOST web.apache.error.pro.webFoobar-ssl.www1: $MESSAGE\n"));};
log { source(s_apache_ssl_error); destination(d_devo_apache_ssl_error); };

Ensure that both the file and the directory where it resides can be read by the user running syslog-ng (that is, syslog). If not, you need to grant ownership of the directory and the file to the syslog user.  

chow :syslog /var/log/apache2 /var/log/apache2/*.log

If the log files are part of a logrotate policy and logrotate create mode is being used, ensure that the syslog-ng user will still have permissions over the new file. For example, in Apache: 

Configuration file /etc/logrotate.d/apache2 extract

/var/log/apache2/*.log {
        ...
        create 640 root syslog
        ...

Now restart syslog-ng:

/etc/init.d/syslog-ng restart


Once restarted, open Devo and go to Data Search to locate the table associated with the tag you used (for example, web.apache.access-combined).

  • If you want to configure syslog-ng to use an encrypted and authenticated channel, see Secure sending via syslog-ng.
  • If the system has SELinux enabled in enforcing mode (run the getenforce command to check the status) it may be necessary to add exceptions to the SELinux policy. See syslog & SELinux configuration for more information.

Log rotation

Here is an example of truncated logrotate configuration file.

/var/log/file.log
{
        rotate 12
        weekly
        copytruncate
        missingok
        notifempty
        compress
}

This is an example of a logrotate command that applies to several log files.

/var/logs/file.out
/var/logs/file.log
/var/logs/localhost.log
/var/logs/localhost_access_log.txt
{
        rotate 10
        daily
        copytruncate
        missingok
        notifempty
        compress
        lastaction
                service syslog-ng reload
        endscript
}

The last action directive reloads syslog-ng once after the rotation of all log files.

Download as PDF

Did you find what you were looking for?

If not, please let us know what you need. Your feedback will help us to improve.

PREVIOUS
File monitoring via rsyslog
NEXT
Secure sending via rsyslog

Export

See what Devo can do for you. Request a demo!
Discover what's new (Release notes)
  • Services & Support
  • Devo.com
  • Contact
    • Contact Us
    • Request a Demo
    • Partner Inquiry
  • Log In
    • USA Devo
    • EU Devo
  • +1 888 6830910 (USA)
  • +34 900 838 880 (Spain)
Copyright © 2019 Legal Terms Privacy Policy Cookies Policy

Powered by Confluence and Scroll Viewport