Scoja client library

The Scoja project has a library for the sending via syslog from Java. This library abstract the protocol operation, but allows to choose between different sending mechanism and configure all the details of a syslog event. Before use it, it is necessary to add the file scoja-cc.jar and scoja-client.jar to the CLASSPATH.

Example 1 

This is a simple example that sends logs to a syslog collector listening on port 514 via TCP (syslog standard port). Devo In-house Relay will start listening on that port. 

import org.scoja.client.Syslogger;
import org.scoja.client.ReusingTCPSyslogger;
 
public class Test {
  static final String HOST = "relay";
  static final int PORT = 514;
 
  public static void main(String args[]) throws Exception {
    Syslogger logger = new ReusingTCPSyslogger(HOST, PORT);
    logger.log("Evento de prueba");
    logger.close();
  }
}

The ReusingTCPSyslogger class:

• Sends syslog events via TCP to the host and port you've configured on its constructor.
• Extends a basic Syslogger class that allows accessing all the details of the syslog protocol ignoring the transport details.
• Uses TCP as transport. 

The log method builds the event with the message it has as an argument and sends it. There are several syslogger sub-classes for the most common transport protocols: UDP, SSL... In case of errors, there are also sub-classes to retry automatically, to do broadcasting, etc.

It's convenient that the events shipping code uses syslogger type objects to make it easy to replace it for another class when you'll like to change the sending mechanism, or send to multiple destinations simultaneously, etc. 

All the syslogger sub-classes can be used to send as many events as you like. In fact, it is convenient to reuse them to send as many events as possible, thus, recovering the cost of it's creation (e.g. to open connections). The most common use is to create the necessary sysloggers at the beginning of the application and keep them open during the execution, using them to send events when needed. If an application only needs a syslogger at times, it can be closed with close(). 

All the sysloggers are thread-safe, so they can be used in concurrent contexts shared in multiple threads simultaneously.

Details

The syslog event contains:

• Date
• Priority, which is built joining a facility and a level
• The source hostname
• A tag
• A message with the event content 

With the log() seen in the first example, we could only control the message part. There's a method in which you can specify the value for each of these fields:

public void log(Date when, int priority, String host, String tag, String message)
throws LoggingException;

Although syslog allows to determine all the fields in each event, the most common is that all the events from an application have some of theses fields fixed (priority or tag) and others are filled by the library with fair values (when, host...). We'll see later how to set them as fixed; for now, we'll focos in 2 ways to avoid having to set them in each message and let the library fill them with appropriate values:

First is to provide this call with empty values, in which case the library will replace them with fixed values. The empty values are null, except for priory which is org.scoja.common.PriorityUtils.UNKNOWN_PRIORITY. Therefore, the following tag is correct and will use fixed values (or calculated by the library) for when, priority, host and tag.

logger.log(null, UNKNOWN_PRIORITY, null, null, message);

The second is to use other versions of the log method with less arguments, the first of them is the one we used in the first example:

public void log(String message)
throws LoggingException;
 
public void log(int priority, String message)
throws LoggingException;   
 
public void log(int priority, String host, String tag, String message)
throws LoggingException;

This variants of the log method also allow to specify empty arguments (null o UNKNOWN_PRIORITY).

The library fills a missing value in various ways. The event date is filled in the actual moment. The host is filled with the name of the host producing the event or with the value it has been given with:

public Syslogger setHost(String host);

The priority and tag are filled with values that have been given with the following methods:

public Syslogger setPriority(int priority);
public Syslogger setTag(String tag);

Priority

Syslog priority is a value composed of two parts: facility and level. Facility and level are numbers, whose meaning is fixed in syslog specification. Priority is also a number that is build by making certain numerical manipulations on its two parts. The org.scoja.common.PriorityUtils class, names the facilities and legitimate levels with constants, and has a statistic methods to construct a priority from a facility and level and to extract the facility and level from a priority.

• The levels are: EMEG, ALERT, CRIT, ERR, WARNING, NOTICE, INFO and DEBUG.
• The facilities are: KERN, USER, MAIL, DAEMON, AUTH, SYSLOG, LPR, NEWS, UUCP, CRON, AUTHPRIV, FTP, LOCAL0, LOCAL1, ..., LOCAL7...

A priority is build with buildPriority (facility,level). The facility and level are extracted from a priority with getFacility(priority) and getLevel(priority).

Example 2

In the following example, we use some of the elements that we've discussed so far. Assume that you want to send to 2 collectors: one is a logtrust relay listening to the port 514 via TCP; the other is a standard syslog daemon also listening to port 514 via UDP.

import org.scoja.common.PriorityUtils;
import org.scoja.client.Syslogger;
import org.scoja.client.UDPSyslogger;
import org.scoja.client.ReusingTCPSyslogger;
import org.scoja.client.SpreadingSyslogger;
 
public class Test {
  static final String HOST1 = "relay1";
  static final int PORT1 = 514;
  static final String HOST2 = "syslog";
  static final int PORT2 = 514;
 
  public static void main(String[] args) throws Exception {
    Syslogger logger = new SpreadingSyslogger(
      new ReusingTCPSyslogger(HOST1,PORT1),
      new UDPSyslogger(HOST2,PORT2));
    logger.setPriority(PriorityUtils.buildPriority(PriorityUtils.LOCAL0,PriorityUtils.INFO));
    logger.setTag("my.app.app1.activity");
    logger.log("Event body");
    logger.close();
  }
}

Two sysloggers are build, a ReusingTCPSyslogger to send to the relay via TCP and an UDPSyslogger p to send via UDP to the traditional syslog collector. To avoid duplicating each sending, they meet in a SpreadingSyslogger that is in charge of duplicating all the actions it receives. This way, it's easy to change the program to send to one or more sources.

Errors

The sending of an event can fail. The syslogger will produce a SysloggerException exception that must be captured and treated.A syslogger that failed to send an event probably has run into an inoperative state. You have to restart it by calling the reset() method, which will make it ready to continue sending events.If the error happens due to network problems; the errors will be repeated and you should opt for an alternative action.All this logic (trap errors, retry and choose an alternative action) is encapsulated in syslogger subclass called RetryingSyslogger. This is an example that retries 3 times and, in case of persistent failure, it shows the log through the standard output.

import java.util.Calendar;
import org.scoja.common.PriorityUtils;
import org.scoja.client.LoggingException;
import org.scoja.client.Syslogger;
import org.scoja.client.ReusingTCPSyslogger;
import org.scoja.client.RetryingSyslogger;
import org.scoja.client.LoggingErrorHandler;
 
public class Test {
  static final String HOST = "relay1";
  static final int PORT = 514;
  static final int RETRIES = 3;
 
  public static void main(String[] args) throws Exception {
    Syslogger base = new ReusingTCPSyslogger(HOST,PORT);
 
    LoggingErrorHandler onError = new LoggingErrorHandler() {
      public void log(Syslogger logger, LoggingException error,
          Calendar when, int priority, String host, String tag, String message)
      throws LoggingException {
        ilog(logger, error, when, priority, host, tag, message);
      }
      public void ilog(Syslogger logger, LoggingException error,
          Calendar when, int priority, String host, String tag, String message)
      throws LoggingException {
        System.err.println("Cannot log to " + logger + "; failed with " + error
          + " while sending " + when + " " + priority + " " + host + " " + tag + " " + message);
      }
    };
    base.setTag("my.app.app1.activity");
    Syslogger logger = new RetryingSyslogger(base, RETRIES, onError);
 
    logger.setPriority(PriorityUtils.buildPriority(PriorityUtils.LOCAL0,PriorityUtils.INFO));
    logger.log("Event body");
    logger.close();
  }
} 

The  RetryingSyslogger class is built with a delegate (base) Syslogger, a number of retries (RETRIES) and an action in case the error persists after the retries (onError). In case of error, the action can do several things with the event that caused the error: discard it, throw an exception, send it to another destination with another logger...

When the transport is UDP, there's usually no error or crashes because the problem usually involves packages in some point of the connection network; but with TCP, the link between the client and the collector is stronger: if the collector is heavily loaded or fallen, or there are network problems that makes it inaccessible, you'll see various anomalous behaviour in your application as exceptions or recurring, or, what is worse, as blocks. 

Java's TCP sockets don't have time-outs for the writing, you can only configure a time-out for the connection. The ReusingTCPSyslogger class makes that property visible with the method setConnectionTimeout(millis).

Non-blocking connections

With the Scoja library, it's possible to build a Syslogger with a more sophisticated behaviour on the transport layer. We'll focus on how to build a Syslogger that operates in a non-blocking manner, both during the connection negotiation as well as the sending; so the calls to log(...) methods will just last long enough to copy the data (either to operating systems or internal buffers). Before getting into the example, it's worth mentioning that we'll achieve the prevention of blocking by sacrificing other things. In this case, we'll have to assume that, if something misbehaves, some events will be lost.

import org.scoja.common.PriorityUtils;
import org.scoja.util.ByteFall;
import org.scoja.util.MemoryByteFall;
import org.scoja.client.Syslogger;
import org.scoja.client.ReusingTransportSyslogger;
import org.scoja.trans.Transport;
import org.scoja.trans.tcp.TCPConf;
import org.scoja.trans.nbtcp.NBTCPTransport;
 
public class Test {
  static final String HOST = "relay1";
  static final int PORT = 514;
  static final int BUFFER_SIZE = 64*1024*1024;
 
  public static void main(String[] args) throws Exception {
    Transport<TCPConf> trans = new NBTCPTransport(HOST,PORT);
    ByteFall buffer = new MemoryByteFall(BUFFER_SIZE);
    Syslogger logger = new ReusingTransportSyslogger(trans, buffer);
 
    logger.setPriority(PriorityUtils.buildPriority(PriorityUtils.LOCAL0,PriorityUtils.INFO));
    logger.setTag("my.app.app1.activity");
    for (int i = 0; i < 1000; i++) {
      logger.log("Event body");
      Thread.sleep(1);
    }
    logger.close();
  }
}

The Syslogger construction is now done in 3 steps. A ReusingTransportSyslogger is a Syslogger that doesn't understand about transportation and expects to be given a Transport type object that will take care of that problem. It does know that the sending via a Transport may fail or jam, in which case, it'll save the events in a buffer waiting for the Transport to recover.

In this example, the Transport is a NBTCPTransport (transport via TCP and non-blocking). All of this transport operations are performed in a non-blocking manner, both the connection and the sending. This is not achieved with a separate thread, but putting the socket in a non-blocking mode. It has the advantage that it coexist well with any application server that doesn't introduce new threads, but it has the disadvantage that it can't progress by itself and needs to receive a request to advance in the connection and sending process. The connection is not fully established until it has processed a few events and are visible in the syslog collector. In practice, this procedure is not visible because there are events continuously.

The buffer where data is stored must be limited, if not, a connectivity problem may cause the client to overflow. The MemoryByteFall class implements a buffer in memory, that occupies as much the size as it has been defined and discards the other data when full.

It should be noted that although this solution may lose packages, it has a better behavior than a sending based on UDP. First, it can work in a very congested network without losses, where package retransmission are necessary. In addition, by selecting a suitable size for the buffer, it'll be possible to restart the collector without losing events.

Encrypted sending

Once you've come to the capabilities of Scoja library to configure the transport layer, there's a world of possibilities. Next, we'll explore the sending with SSL.

When working with Devo, there are multiple ways to send events: In-house Relays, public relays, relays in a cloud provider's safety zone, etc. These alternatives are simple intermediaries for Devo's central collectors, they're designed to facilitate the sending; but, sometimes, it can be interesting to have a program that sends directly to the central collectors.

All the logs must get to the central collector through a SSL channel, which forces the double authentication by certificates: the server presents its certificate to the client, but the client must show the certificate to the server. This ensures that the traffic is encrypted when transmitted through internet and that it's not possible to do impersonations in either of the 2 extremes.

We'll adapt the example to send directly to logtrust central collectors via SSL:

import java.io.File;
import org.scoja.common.PriorityUtils;
import org.scoja.util.ByteFall;
import org.scoja.util.MemoryByteFall;
import org.scoja.client.Syslogger;
import org.scoja.client.ReusingTransportSyslogger;
import org.scoja.trans.Transport;
import org.scoja.trans.tcp.TCPConf;
import org.scoja.trans.nbtcp.NBTCPTransport;
import org.scoja.trans.ssl.SSLConf;
import org.scoja.trans.ssl.SSLTransport;
import static org.scoja.trans.ssl.SSLClientAuthenticationMode.REQUIRE;
import static org.scoja.trans.ssl.SSLUtils.loadTrusts;
import static org.scoja.trans.ssl.SSLUtils.loadKeys;
 
public class TestSSL {
    static final String TRUSTJKS = "logtrust.jks";
    static final String MEJKS = "me.jks";
    static final String HOST = "eu.public.relay.logtrust.net";
    static final int PORT = 443;
    static final int BUFFER_SIZE = 64*1024*1024;
 
    public static void main(String[] args) throws Exception {
        char[] password = args[0].toCharArray();
         
        Transport<TCPConf> tcp = new NBTCPTransport(HOST,PORT);
        Transport<SSLConf> ssl = SSLTransport.tls(tcp);
        SSLConf sslConf = ssl.configuration();
        sslConf.setClientAuth(REQUIRE);
        sslConf.setTrustManagers(loadTrusts(new File(TRUSTJKS)));
        sslConf.setKeyManagers(loadKeys(new File(MEJKS), password));
     
        ByteFall buffer = new MemoryByteFall(BUFFER_SIZE);
        Syslogger logger = new ReusingTransportSyslogger(ssl, buffer);
 
        logger.setPriority(PriorityUtils.buildPriority(PriorityUtils.LOCAL0,PriorityUtils.INFO));
        logger.setTag("my.app.app1.activity");
        for (int i = 0; i < 1000; i++) {
            logger.log("Event body");
            Thread.sleep(1);
        }
        logger.close();
    }
}

As you can see, we've added a new transport build with SSLTransport.tls(tcp). It's a transport element that makes a TLS negotiation over the tcp; which at the same time is a non-blocking TCP transport. The result is a non-blocking SSL transport.

As mentioned above, the communication with Devo's central collectors has to be (necessarily) via SSL with client authentication. To do so, the client must have a private key to communicate with logtrust; you can generate and download that key on the Devo application. In this code, we assume this has been done and is saved in the file me.jks.

It's also desirable to authenticate our central servers (to avoid a man-in.the.middle attack). To do so, you must have the central collectors public certificate, which you can also download from our web application. We assume this have been done and is saved in the file logtrust.jks.

A SSL Transport default configuration doesn't know which type of authentication it has to negotiate, neither the certificates and private keys to work with. All this is indicated with the corresponding sslConf methods. Specifically, the collector certificate and our private key are given with setTrustManagers and setKeyManagers, that wait for an array from javax.net.ssl.TrustManager and javax.net.ssl.KeyManager, respectively. There are different ways to build these arrays. Since we're assuming that we have information on JKS files, it's best to use the loadTrusts and loadKeys methods of the org.scoja.trans.ssl.SSLUtils class.

In the description of the non-blocking sending we've explained that the negotiation of the TCP connection is also made in a non-blocking manner, which usually means that the first elements sent through the Syslogger doesn't arrive immediately and are delayed until the connection is established. For TCP it's usually a couple of events. When SSL Transport operates on a non-blocking layer, it performs its own negotiation and subsequent data sending also in a non-blocking manner. Since the SSL negotiation consumes way more resources than the TCP, with several exchanges that require computation, the initial block of events that can't be sent immediately is usually larger (between 5 and 10 events).