Relay troubleshooting tips (for versions prior to 1.4.0)
The Devo In-house Relay software is based on a Syslog engine called Scoja written in Java. Also, there are some processes written in Python that perform other tasks.
This software is prepared to be installed in any Linux system, but Devo recommends using Ubuntu.
Once running, the Scoja process listens for TCP/UDP connections on ports and, depending on the port, it modifies the Syslog events and forwards them to the Devo central collector. The main role of the relay is to tag events based on a defined rule, then forward the tagged events to Devo.
The relay comes with a few ports that are preconfigured with rules for managing specific types of events. For example, port 13000 is preconfigured to forward the events received directly to Devo without any other processing. This port should be used for events that are properly tagged in the data source. You can also create custom rules to tag and/or filter the events sent to the associated port on the relay. Each rule is stored in one of these two folders:
- /etc/logtrust/scoja/current/rules → Rules managed from the Devo web application.
- /etc/logtrust/scoja/current/unrules → Rules not managed from the Devo web application. These rules are directly managed from the relay machine itself.
About the relay logs files
First, a word about the logs generated by the Devo Relay that can be used to investigate and identify problems with the relay.
The main logs files are:
- /var/log/scoja.log → Main activity log of the Scoja engine. For example, if there is an error starting the process, it will be logged in this file.
- /var/log/lt-relay.log → Contains events related to rules and changes to the relay configuration.
In addition, daily log files are generated and stored in:
- main.log → main process log file
- source.log → logs related to the sources configured
- thread.log → threads opened for each source
- stats.log → statistics about received and sent events
- target.log → logs related to the sending process
These five daily logs are compressed at night by a cron task.
Make sure only one Scoja process (Java) is running
ps auxwww|grep java|grep scoja|grep -v grep root 884 0.2 34.6 2513192 173008 ? Sl 08:08 0:10 java -server -Xms512M -Xmx512M -XX:+UseConcMarkSweepGC -classpath :/opt/logtrust/scoja/scoja.jar:/opt/logtrust/scoja/scoja-cc.jar:/opt/logtrust/scoja/scoja-compression.jar:/opt/logtrust/scoja/scoja-rpc.jar:/opt/logtrust/scoja/scoja-beep.jar:/opt/logtrust/scoja/jython.jar -Djava.library.path=/opt/logtrust/scoja -Dscoja.home=/opt/logtrust/scoja -Xms500M -Xmx500M org.scoja.server.Scoja -r 5s -G /etc/logtrust/scoja/current/all-me.conf -j /etc/logtrust/scoja/current/all-var.conf
If there is more than one process running, you should kill all of them using the command kill -9 <PID>, then start the relay again:
If the Java process is still not appearing after restarting the relay, check the scoja.log file for errors.
Check the scoja.log file for errors
Use cat to display the contents of the scoja.log:
# cat /var/log/scoja.log May 24 16:03:54 relayInhouseSecMon syslog.scoja.configuration: (Re)Loading configuration file /etc/logtrust/scoja/version1/all-me.conf May 24 16:03:54 relayInhouseSecMon syslog.scoja.thread: Starting thread 1 for Queue "internal queue" with 1 queued events, 1 max threads, 1 threads started, 0 threads awaiting May 24 16:03:54 relayInhouseSecMon syslog.scoja.configuration: (Re)Loading configuration file /etc/logtrust/scoja/version1/parameters.conf May 24 16:03:55 relayInhouseSecMon syslog.scoja.configuration: Configuration file /etc/logtrust/scoja/version1/parameters.conf successfully parsed May 24 16:03:55 relayInhouseSecMon syslog.scoja.configuration: Configuration file /etc/logtrust/scoja/version1/parameters.conf successfully (re)loaded May 24 16:03:55 relayInhouseSecMon syslog.scoja.configuration: (Re)Loading configuration file /etc/logtrust/scoja/version1/me.conf May 24 16:03:56 relayInhouseSecMon syslog.scoja.configuration: Configuration file /etc/logtrust/scoja/version1/me.conf successfully parsed May 24 16:03:56 relayInhouseSecMon syslog.scoja.configuration: Configuration file /etc/logtrust/scoja/version1/me.conf successfully (re)loaded May 24 16:03:56 relayInhouseSecMon syslog.scoja.configuration: Configuration file /etc/logtrust/scoja/version1/all-me.conf successfully (re)loaded May 24 16:03:56 relayInhouseSecMon syslog.scoja.configuration: (Re)Loading configuration file /etc/logtrust/scoja/version1/all-var.conf May 24 16:03:57 relayInhouseSecMon syslog.scoja.configuration: Configuration file /etc/logtrust/scoja/version1/all-var.conf successfully parsed May 24 16:03:57 relayInhouseSecMon syslog.scoja.configuration: Configuration file /etc/logtrust/scoja/version1/all-var.conf successfully (re)loaded ... May 24 16:03:57 relayInhouseSecMon syslog.scoja.source.net.udp: Starting UDP source listening at 0.0.0.0:13002 May 24 16:03:57 relayInhouseSecMon syslog.scoja.source.net.tcp: Opening server socket for Selecting TCP source listening at 0.0.0.0:13001 May 24 16:03:57 relayInhouseSecMon syslog.scoja.thread: Starting thread 1 for UDP source listening at 0.0.0.0:13002 May 24 16:03:57 relayInhouseSecMon syslog.scoja.source.net.tcp: Starting Selecting TCP source listening at 0.0.0.0:13002 May 24 16:03:57 relayInhouseSecMon syslog.scoja.source.net.udp: Opening server socket for UDP source listening at 0.0.0.0:13002 May 24 16:03:57 relayInhouseSecMon syslog.scoja.thread: Starting thread 1 for Selecting TCP source listening at 0.0.0.0:13002 May 24 16:03:57 relayInhouseSecMon syslog.scoja.main: Reloader started to run every 5.0 seconds May 24 16:03:57 relayInhouseSecMon syslog.scoja.main: Scoja started
If the log file contains the error /opt/logtrust/scoja/scoja.sh: 60: java: not found, this means that it is unable to find the Java interpreter. In this case, consider creating a symbolic link or editing the JAVA_HOME environment variable.
Confirm that the default ports are listening
- Ports 13000-13002 are listening on both TCP and UDP.
- Port 12999 is listening on UDP and is prepared to receive external Netflow traffic (Netflow version < 9).
- Port 5141 is for internal Netflow traffic obtained using fprobe.
- Port 5140 is listening on TCP and UDP to handle internal relay logs.
# netstat -atunp | grep java | grep -v ESTABLISHED tcp6 0 0 :::13000 :::* LISTEN 887/java tcp6 0 0 :::13001 :::* LISTEN 887/java tcp6 0 0 :::13002 :::* LISTEN 887/java tcp6 0 0 127.0.0.1:5140 :::* LISTEN 887/java udp6 0 0 :::12999 :::* 887/java udp6 0 0 :::13000 :::* 887/java udp6 0 0 :::13001 :::* 887/java udp6 0 0 :::13002 :::* 887/java udp6 0 0 127.0.0.1:5140 :::* 887/java udp6 0 0 127.0.0.1:5141 :::* 887/java
Ensure that the client certificate is installed
The client certificate file, client.jks, should be installed in the relay's /etc/logtrust/scoja/current/keys/ folder.
# ls -al /etc/logtrust/scoja/current/keys/ total 16 drwx------ 2 root root 4096 Jan 28 10:24 . drwxr-xr-x 5 root root 4096 Jan 28 10:24 .. -rw-r--r-- 1 root root 8048 Jan 28 10:24 client.jks lrwxrwxrwx 1 root root 43 Jan 28 10:14 me.jks -> /etc/logtrust/scoja/current/keys/client.jks
If the certificate download fails, you can force the download again:
- Log into the Devo web application.
- Go to Administration → Relays and select the name of the relay in the list. The Rules screen appears.
- Select the Force Generate New Certificate check box, then click the Apply Configuration button.
Ensure that the relay configuration is updating correctly
A cron task runs every minute to check if any changes to the relay configuration have been made in Devo. The result of this check appears in the lt-relay.log file.
# cat /var/log/lt-relay.log: 2016-01-28 10:16:01.374445 [ERROR] Property 'api.key' not found in conf file 2016-01-28 10:17:01.510618 [ERROR] Property 'api.key' not found in conf file 2016-01-28 10:18:01.651220 [ERROR] Property 'api.key' not found in conf file 2016-01-28 10:19:01.791669 [ERROR] Property 'api.key' not found in conf file 2016-01-28 10:20:01.932002 [ERROR] Property 'api.key' not found in conf file 2016-01-28 10:21:02.073736 [ERROR] Property 'api.key' not found in conf file 2016-01-28 10:22:01.208316 [ERROR] Property 'api.key' not found in conf file 2016-01-28 10:23:00.488298 [SUCCESS] Relay activation is still pending, waiting for user to activate the Relay 2016-01-28 10:23:01.158783 [SUCCESS] Relay activation is still pending, waiting for user to activate the Relay 2016-01-28 10:24:01.331976 [SUCCESS] Relay is active, deploying new configuration set 2016-01-28 10:25:01.794021 [SUCCESS] No changes in the Relay Configuration 2016-01-28 10:26:01.975251 [SUCCESS] No changes in the Relay Configuration ...
- The "Property 'api.key' not found in conf file" meesage indicates that the devo-relay-configure process has not been executed yet and there is no API key defined in the relay.
- The "Relay activation is still pending" message indicates that relay is registered but has not yet been activated in the Devo web application.
- The "Relay is active" message confirms that the relay has been activated and new configuration has been downloaded.
Once activated, you will be able to define the relay rules using the web application. The relay will begin checking the configuration for updates every 1 minute.
- The "No changes in Relay Configuration" message occurs every minute to confirm that no changes were detected in the relay configuration.
During the first configuration, the certificate is also downloaded. If the certificate download fails, you must force the download again.
Send test events to confirm relay is receiving and forwarding correctly
We can use netcat to send 100 events to the relay port 13000 to confirm that it is receiving and forwarding events to Devo. Enter this command on the relay machine:
To confirm that the events were processed correctly, open Devo, go to Data Search, and locate the test.keep.free table. This table should contain 100 events.
If there are no events logged in the test.keep.free table, check the unknown.unknown table. If there was a problem applying the tags, events will be logged in this table.
On Ubuntu 18.04, you need to specify that netcat close the port when the transmission is done so you should use this command:
Check the relay's internet proxy configuration
If the Devo relay uses a proxy to access the Internet, you can review the configuration for possible errors.
The internet proxy is configured in /etc/logtrust/scoja/current/local.py. In the sample configuration below, the proxy IP address is 18.104.22.168 and the port is 8080.
... # Devo collectors nextHops = [ #hop(host,port[,proxy,proxyPort]) hop(host="eu.elb.relay.logtrust.net", port=443, proxy="22.214.171.124", proxyPort=8080), ] ...
The hop host should be the endpoint for the Devo region you are using and the port must always be 443. These are the endpoints by region.
- Europe: eu.elb.relay.logtrust.net
- USA: us.elb.relay.logtrust.net
- Spain: es.elb.relay.logtrust.net
- South America: collector-sa.devo.io: 443
Some common problems and solutions
Problem 1: Scoja doesn't start
In certain Linux distributions, edits need to be made to the /etc/hosts file to define the localhost IP address. If these edits are not made, the Scoja process will not start and the following error is written to the /var/log/scoja.log file.
# cat /var/log/scoja.log Exception in thread "main" java.lang.ExceptionInInitializerError at org.scoja.server.Scoja.<init>(Scoja.java:115) at org.scoja.server.Scoja.main(Scoja.java:65) Caused by: java.lang.RuntimeException: Fatal error: Cannot find localhost address at org.scoja.server.source.Internal.<clinit>(Internal.java:131) ... 2 more
Solution: Open the /etc/hosts for editing and set the relay_hostname to resolve to localhost IP 127.0.0.1 as indicated here.
# cat /etc/hosts 127.0.0.1 localhost 127.0.1.1 <relay_hostname> ...
Problem 2: Daily logs not getting compressed at night
Each night, a cron task should compress the daily logs stored in /var/logt/local/<year>/<month>/<day>/<relayName>/syslog/scoja/. If these are not getting compressed at night, there is a problem.
Solution: Fix the configuration of the cron task. The cron task must run as the root user. For example:
- Incorrect: cat /etc/cron.d/lt-compress 30 02 * * * devo /usr/local/bin/daily_compress.sh
- Correct: cat /etc/cron.d/lt-compress 30 02 * * * root /usr/local/bin/daily_compress.sh
Problem 3: Out of memory errors from the scoja Java process
This problem is usually related to the memory buffer.
Solution: To increase the memory allocated to the scoja process, you need to edit a couple of files, then restart the scoja process. In the steps below, we increase the memory allocation to scoja to 1GB:
Edit the following part of /opt/logtrust/scoja/scoja.sh:
Edit the SCOJA_JVM_OPTIONS parameter in /etc/profile.d/logtrust.sh:
Restart scoja using the following commands: