In-house Relay troubleshooting

Overview

The Devo In-house Relay software is based on a Syslog engine called Scoja written in Java. Also, there are some processes written in Python that perform other tasks.

This software is prepared to be installed in any Linux system, but Devo recommends using Ubuntu.

Once running, the Scoja process listens for TCP/UDP connections on ports and, depending on the port, it modifies the Syslog events and forwards them to the Devo central collector. The main role of the relay is to tag events based on a defined rule, then forward the tagged events to Devo. 

The relay comes with a few ports that are preconfigured with rules for managing specific types of events. For example, port 13000 is preconfigured to forward the events received directly to Devo without any other processing. This port should be used for events that are properly tagged in the data source. You can also create custom rules to tag and/or filter the events sent to the associated port on the relay. Each rule is stored in one of these two folders:

  • /etc/logtrust/scoja/current/rules → Rules managed from the Devo web application.
  • /etc/logtrust/scoja/current/unrules → Rules not managed from the Devo web application. These rules are directly managed from the relay machine itself.

The relay is configured using the command logtrust-relay-configure, which registers the relay in a Devo domain. Once registered, it needs to be activated in the Administration → Relays area of the web application. Once activated, all events sent to this relay will be forwarded to the Devo collector and stored in the corresponding domain.

The Devo collector address is based on the Devo platform that is being used. These are the hostnames to be used for our cloud platforms:

Cloud

Devo collector address (hostname:port)

EU

eu.elb.relay.logtrust.net:443

USA

us.elb.relay.logtrust.net:443

VDC (Spain)

es.elb.relay.logtrust.net:443

Installing or uninstalling the relay using an APT package

There is an APT package available for each Devo cloud platform. The packages are used for installing, or uninstalling, the In-house Relay on an Ubuntu or Debian system. 

Cloud APT package name
EU logtrust-relay
USA logtrust-relay-aws-usa
VDC (Spain) logtrust-relay-vdc
  • To install the In-house Relay:

    # apt-get install <package_name>
     
    # For AWS EU:
    # apt-get install logtrust-relay
  • To uninstall the In-house Relay:

    # apt-get purge <package_name>
     
    # For AWS EU:
    # apt-get purge logtrust-relay

Starting and stopping the In-house Relay

Simple commands are available to start or stop the relay. 

# /etc/init.d/<package_name> stop
# /etc/init.d/<package_name> start
 
# For AWS EU:
# /etc/init.d/logtrust-relay stop
# /etc/init.d/logtrust-relay start

Using the relay logs files

The main logs files are:

  • /var/log/scoja.log → Main activity log of the Scoja engine. For example, if there is an error starting the process, it will be logged in this file.
  • /var/log/lt-relay.log → Contains events related to rules and changes to the relay configuration.

In addition, daily log files are generated and stored in:

/var/logt/local/<year>/<month>/<day>/<relayName>/syslog/scoja/

  • main.log → main process log file
  • source.log → logs related to the sources configured
  • thread.log → threads opened for each source
  • stats.log → statistics about received and sent events
  • target.log  → logs related to the sending process

These five daily logs are compressed at night by a cron task.

Troubleshooting checklist

Make sure only one Scoja process (Java) is running 

# ps auxwww|grep java|grep scoja|grep -v grep
 
root       884  0.2 34.6 2513192 173008 ?      Sl   08:08   0:10 java -server -Xms512M -Xmx512M -XX:+UseConcMarkSweepGC -classpath :/opt/logtrust/scoja/scoja.jar:/opt/logtrust/scoja/scoja-cc.jar:/opt/logtrust/scoja/scoja-compression.jar:/opt/logtrust/scoja/scoja-rpc.jar:/opt/logtrust/scoja/scoja-beep.jar:/opt/logtrust/scoja/jython.jar -Djava.library.path=/opt/logtrust/scoja -Dscoja.home=/opt/logtrust/scoja -Xms500M -Xmx500M org.scoja.server.Scoja -r 5s -G /etc/logtrust/scoja/current/all-me.conf -j /etc/logtrust/scoja/current/all-var.conf

If there is more than one process running, you should kill all of them using the command kill -9 <PID>, then start the relay again by running /etc/init.d/logtrust-relay start.

Check the scoja.log file for errors

# cat /var/log/scoja.log
root@relayInhouseSecMon:~# cat /var/log/scoja.log
May 24 16:03:54 relayInhouseSecMon syslog.scoja.configuration: (Re)Loading configuration file /etc/logtrust/scoja/version1/all-me.conf
May 24 16:03:54 relayInhouseSecMon syslog.scoja.thread: Starting thread 1 for Queue "internal queue" with 1 queued events, 1 max threads, 1 threads started, 0 threads awaiting
May 24 16:03:54 relayInhouseSecMon syslog.scoja.configuration: (Re)Loading configuration file /etc/logtrust/scoja/version1/parameters.conf
May 24 16:03:55 relayInhouseSecMon syslog.scoja.configuration: Configuration file /etc/logtrust/scoja/version1/parameters.conf successfully parsed
May 24 16:03:55 relayInhouseSecMon syslog.scoja.configuration: Configuration file /etc/logtrust/scoja/version1/parameters.conf successfully (re)loaded
May 24 16:03:55 relayInhouseSecMon syslog.scoja.configuration: (Re)Loading configuration file /etc/logtrust/scoja/version1/me.conf
May 24 16:03:56 relayInhouseSecMon syslog.scoja.configuration: Configuration file /etc/logtrust/scoja/version1/me.conf successfully parsed
May 24 16:03:56 relayInhouseSecMon syslog.scoja.configuration: Configuration file /etc/logtrust/scoja/version1/me.conf successfully (re)loaded
May 24 16:03:56 relayInhouseSecMon syslog.scoja.configuration: Configuration file /etc/logtrust/scoja/version1/all-me.conf successfully (re)loaded
May 24 16:03:56 relayInhouseSecMon syslog.scoja.configuration: (Re)Loading configuration file /etc/logtrust/scoja/version1/all-var.conf
May 24 16:03:57 relayInhouseSecMon syslog.scoja.configuration: Configuration file /etc/logtrust/scoja/version1/all-var.conf successfully parsed
May 24 16:03:57 relayInhouseSecMon syslog.scoja.configuration: Configuration file /etc/logtrust/scoja/version1/all-var.conf successfully (re)loaded
...
May 24 16:03:57 relayInhouseSecMon syslog.scoja.source.net.udp: Starting UDP source listening at 0.0.0.0:13002
May 24 16:03:57 relayInhouseSecMon syslog.scoja.source.net.tcp: Opening server socket for Selecting TCP source listening at 0.0.0.0:13001
May 24 16:03:57 relayInhouseSecMon syslog.scoja.thread: Starting thread 1 for UDP source listening at 0.0.0.0:13002
May 24 16:03:57 relayInhouseSecMon syslog.scoja.source.net.tcp: Starting Selecting TCP source listening at 0.0.0.0:13002
May 24 16:03:57 relayInhouseSecMon syslog.scoja.source.net.udp: Opening server socket for UDP source listening at 0.0.0.0:13002
May 24 16:03:57 relayInhouseSecMon syslog.scoja.thread: Starting thread 1 for Selecting TCP source listening at 0.0.0.0:13002
May 24 16:03:57 relayInhouseSecMon syslog.scoja.main: Reloader started to run every 5.0 seconds
May 24 16:03:57 relayInhouseSecMon syslog.scoja.main: Scoja started

Confirm that the default ports are listening

  • Port 13000-13002 are listening both in TCP and UDP.
  • Port 12999 is listening in UDP and is prepared to receive external Netflow traffic (Netflow version < 9). 
  • Port 5141 is for internal Netflow traffic obtained using fprobe.
  • Port 5140 is listening in TCP and UDP and handles internal relay logs. 
# netstat -atunp | grep java | grep -v ESTABLISHED
tcp6       0      0 :::13000                :::*                    LISTEN      887/java
tcp6       0      0 :::13001                :::*                    LISTEN      887/java
tcp6       0      0 :::13002                :::*                    LISTEN      887/java
tcp6       0      0 127.0.0.1:5140          :::*                    LISTEN      887/java
udp6       0      0 :::12999                :::*                                887/java
udp6       0      0 :::13000                :::*                                887/java
udp6       0      0 :::13001                :::*                                887/java
udp6       0      0 :::13002                :::*                                887/java
udp6       0      0 127.0.0.1:5140          :::*                                887/java
udp6       0      0 127.0.0.1:5141          :::*                                887/java

Ensure that the client certificate is installed

The client certificate file, client.jks, should be installed in the relay's /etc/logtrust/scoja/current/keys/ folder. 

# ls -al /etc/logtrust/scoja/current/keys/
total 16
drwx------ 2 root root 4096 Jan 28 10:24 .
drwxr-xr-x 5 root root 4096 Jan 28 10:24 ..
-rw-r--r-- 1 root root 8048 Jan 28 10:24 client.jks
lrwxrwxrwx 1 root root 43 Jan 28 10:14 me.jks -> /etc/logtrust/scoja/current/keys/client.jks

If the certificate download fails, you must force the download again:

  1. Log into the Devo web application.
  2. Go to Administration → Relays and select the name of the relay in the list. The Rules screen appears. 
  3. Select the Force Generate New Certificate check box, then click the Apply Configuration button.

Ensure that the relay configuration is updating correctly

A cron task runs every minute to check if any changes to the relay configuration have been made in Devo. The result of this check appears in the lt-relay.log file. 

# cat /var/log/lt-relay.log:
2016-01-28 10:16:01.374445 [ERROR] Property 'api.key' not found in conf file
2016-01-28 10:17:01.510618 [ERROR] Property 'api.key' not found in conf file
2016-01-28 10:18:01.651220 [ERROR] Property 'api.key' not found in conf file
2016-01-28 10:19:01.791669 [ERROR] Property 'api.key' not found in conf file
2016-01-28 10:20:01.932002 [ERROR] Property 'api.key' not found in conf file
2016-01-28 10:21:02.073736 [ERROR] Property 'api.key' not found in conf file
2016-01-28 10:22:01.208316 [ERROR] Property 'api.key' not found in conf file
2016-01-28 10:23:00.488298 [SUCCESS] Relay activation is still pending, waiting for user to activate the Relay
2016-01-28 10:23:01.158783 [SUCCESS] Relay activation is still pending, waiting for user to activate the Relay
2016-01-28 10:24:01.331976 [SUCCESS] Relay is active, deploying new configuration set
2016-01-28 10:25:01.794021 [SUCCESS] No changes in the Relay Configuration
2016-01-28 10:26:01.975251 [SUCCESS] No changes in the Relay Configuration
...

Initial errors are because the logtrust-relay-configure process has not been executed yet and there is no API key defined in the relay.

  • The message Relay activation is still pending appears when the relay is registered but the activation has not been performed from the Devo web application.  
  • The message Relay is active appears when we activate the relay and the configuration is downloaded.

Since this moment, the relay rules can be managed from the web application. The relay will check if there is any change in the configuration every 1 minute.

The message  No changes in Relay Configuration means that the configuration read from the web application has not changed.

During the first configuration, the certificate is also downloaded. If the certificate download fails, you must force the download again:

  1. Log into the Devo web application.
  2. Go to Administration → Relays and select the name of the relay in the list. The Rules screen appears. 
  3. Select the Force Generate New Certificate check box, then click the Apply Configuration button.

Send test events to confirm relay is functioning

We can use netcat to send 100 events to the virtual IP to confirm that the relays are processing events correctly and forwarding them to Devo. 

# for i in `seq 1 100`; do (echo "<14>Jan  1 00:00:00 xxx test.keep.free: test event $i"|nc localhost 13000); done
  • Change localhost in the command above to match your virtual IP.
  • To confirm that the events were processed correctly, open Devo, go to Data Search, and locate the test.keep.free table. This table should contain 100 events. 
  • If there are no events logged in the test.keep.free table, check the unknown.unknown table. If there was a problem applying the tags, events will be logged in this table.

Check the relay's internet proxy configuration

If the In-house Relay uses a proxy to access the Internet, you can review the configuration for possible errors.

  • The internet proxy is configured in /etc/logtrust/scoja/current/local.py. In the sample configuration below, the proxy IP address is 12.12.12.12 and the port is 8080.

    ...
    # LogTrust collectors
    nextHops = [
    #hop(host,port[,proxy,proxyPort])
    hop(host="es.elb.relay.logtrust.net", port=443, proxy="12.12.12.12", proxyPort=8080),
    ]
    ...
  • The proxy is allowed to make connections to:
    • For EU users:
      • eu.elb.relay.logtrust.net: 443 
      • app.logtrust.com: 443

    • For USA users:
      • us.elb.relay.logtrust.net: 443
      • usa.logtrust.com: 443

    • For VDC (Spain) users:
      • es.elb.relay.logtrust.net: 443
      • spain.logtrust.com: 443

Known problems and solutions

Problem 1: Scoja doesn't start

In certain Linux distributions, edits need to be made to the /etc/hosts.localhost file to define the localhost IP address. If these edits are not made, the Scoja process will not start and the following error is written to the /var/log/scoja.log file.

[root@Tenant-Proxy01 ~]# cat /var/log/scoja.log
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.scoja.server.Scoja.<init>(Scoja.java:115)
at org.scoja.server.Scoja.main(Scoja.java:65)
Caused by: java.lang.RuntimeException: Fatal error: Cannot find localhost address
at org.scoja.server.source.Internal.<clinit>(Internal.java:131)
... 2 more

Solution: Open the /etc/hosts.locahost for editing and set the relay_hostname to resolve to localhost IP 127.0.0.1 as below.

root@relayInhouseVDC:/opt/logtrust/relay# cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 <relay_hostname>
...

Problem 2: Daily logs not getting compressed at night

Each night, a cron task should compress the daily logs stored in /var/logt/local/<year>/<month>/<day>/<relayName>/syslog/scoja/. If these are not getting compressed at night, there is a problem.

Solution: Fix the configuration of the cron task. The cron task must run as the root user. Therefore: 

  • Wrong configuration:
    cat /etc/cron.d/lt-compress 30 02 * * * logtrust /usr/local/bin/daily_compress.sh
  • Correct configuration:
    cat /etc/cron.d/lt-compress 30 02 * * * root /usr/local/bin/daily_compress.sh

Have we answered your question?

If not, please contact our technical support team via email by clicking the button below.

CONTACT US