web.tomcat

The tag  web.tomcat.type.environment.application.clon  is used to label the Apache Tomcat application server logs.

When using Apache Tomcat as application server, there are two distinct areas producing logs and that can be integrated with Devo:

  • Tomcat logs
  • Logs from the applications managed by Tomcat

This article describes how to configure Tomcat for integration of the first type of logs, produced and managed by the application server itself.

Tomcat generates by default the following log types: 

  • catalina.yyyy-MM-dd.log.
  • localhost.yyyy-MM-dd.log.
  • catalina.out.
  • localhost_access_log.yyyy-MM-dd.log.

Except catalina.out, the other files are rotated daily in their default configuration. All these files are stored in the CATALINA_BASE/logs directory.

Tag Structure

The elements environment, application and clon are free, but mandatory. They can have the value chosen by the user to identify the event source, but they they must always be used. The name of each of these elements reflects the intended use:

  • environment - the environment where the event occurs (development testing, production, etc.). The environments number and names are not fixed because we understand there may be a lot of variation between installations. 
  • application - the web application name.
  • clon - the name of the Tomcat instance where the event has occurred. Depending on the company it can be a machine name, the virtual name they want to give a Tomcat process, etc.

The element type is fixed and it identifies the type and format of the sent event. This element can take one of the following values:  

  • catalina
  • app
  • out
  • access-clf
  • access-combined
  • access-lt

Logging in Tomcat

Tomcat uses Apache Common Logging to generate the log events. The log messages that arrive through ServletContex log methods are also routed to this log library and associated to the following category:

org.apache.catalina.core.ContainerBase.[engine].[host].[context]

  • Usually, the engine is Catalina and the host is localhost.
  • The context name it is used as context, but there are some exceptions. For example, an application whose context path is /cava would have the category org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/cava] associated.

In Tomcat default distribution, Apache Common Logging is associated to the JUL framework (java.util.logging). Hence, all events, including the ones generated by ServletContext are managed by JUL.

Tomcat configures JUL with the file specified in the LOGGING_CONFIG environment variable. In case it is missing, the installation file CATALINA_BASE/conf/logging.properties is used.

catalina.yyyy-MM-dd.log

This log contains the messages generated by Tomcat, usually associated with the server and applications life cycle. It is similar to Devo catalina type.

The definition is found in CATALINA_BASE/conf/logging.properties. The logs coming from the container are configured to be stored in this file. This is done with the help of a handler built to write in catalina.yyy-MM-dd.log and associated with the root logger.

catalina.yyyy-MM-dd.log Configuration

1catalina.org.apache.juli.FileHandler.level = FINE
1catalina.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
1catalina.org.apache.juli.FileHandler.prefix = catalina.
 
.handlers = 1catalina.org.apache.juli.FileHandler

localhost.yyyy-MM-dd.log

This log contains the messages written by the applications with JUL (java.util.logging).It is similar to Devo app type.

The definition is found in CATALINA_BASE/conf/logging.properties. The logs coming from the applications are configured to be stored in this file. This is done with the help of a handler built to write in localhost.yyy-MM-dd.log and associated with ServletContext log.

localhost.yyyy-MM-dd.log Configuration

2localhost.org.apache.juli.FileHandler.level = FINE
2localhost.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
2localhost.org.apache.juli.FileHandler.prefix = localhost.
 
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].handlers = 2localhost.org.apache.juli.FileHandler

catalina.out

This log contains the information sent by the applications to stdout and stderr. It is generated by redirecting to this files standard and error output at operating system level. Since Tomcat is not directly responsible of its creation, it doesn't rotate. 

It is similar to Devo out type.

Access log (localhost_access_log.yyyy-MM-dd.log)

This file contains an access log similar to web servers logs. In the log, there is an event for each petition processed by the server. You can control the event content (format and fields) in detail.

The access log is configured as a Tomcat Valve. The definition is found in CATALINA_BASE/conf/server.xml. 

Tomcat access-log Configuration

<Valve className="org.apache.catalina.valves.AccessLogValve"
    directory="logs"
    prefix="localhost_access_log." suffix=".txt"
    pattern="%h %l %u %t &quot;%r&quot; %s %b" />

Devo supports 3 access events formats, two of them are standard and a third defined by Devo for users who need more details. Each of these formats is identified with a value in the type concept of the syslog's tag we're defining: 

  • access-clf for the Common Log Format (CLF). This is Tomcat and Apache default format. The directive to define this format is:

    CLF

    pattern="%h %l %u %t &quot;%r&quot; %s %b"
    pattern="common"
  • access-combined for the NCSA extended/combined Log Format format. The directive for this format is:

    Combined

    pattern="%h %l %u %t &quot;%r&quot; %s %b &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot;"
    pattern="combined"
  • access-lt for the Devo defined format that adds more detail to the standard formats listed above. The directive to define this format is:

    logtrust Access

    pattern="%t %a %l %u %S %v:%p %m &quot;%U%q&quot; %H &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot; &quot;%{cookieName1}c:%{cookieName2}c&quot; %s %D %B %I"

   ***where %{cookieName1}c:%{cookieName2}c:...:$cookieNameN}c is the name of the cookies you want to be reflected in the log. 

  •  If you don't want to save any cookie, the format would be as follows:

    logtrust Access without cookies

    pattern="%t %a %l %u %S %v:%p %m &quot;%U%q&quot; %H &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot; &quot;&quot; %s %D %B %I"

Sending to Devo via files

  • UNIX environments

    The standard method is to configure Tomcat to write the logs to file and rely on another tool (like rsyslog o syslog-ng) to send the events.

    In rsyslog case, since it cannot read dynamic files (Tomcat 3 log files are rotated daily), it is necessary to prevent Tomcat from rotating the files. This way, they will have a static name and rsyslog can process them correctly.

    To disable the rotation, define the rotatable attribute as false in the configuration of the 3 log files.

    • For localhost-yyyy-MM-dd.log and catalina-yyyy-MM-dd.log  you have to modify CATALINA_BASE/conf/logging.properties:

      Configuration of CATALINA_BASE/conf/logging.properties without rotation

      1catalina.org.apache.juli.FileHandler.level = FINE
      1catalina.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
      1catalina.org.apache.juli.FileHandler.prefix = catalina
      1catalina.org.apache.juli.FileHandler.rotatable = false
       
      2localhost.org.apache.juli.FileHandler.level = FINE
      2localhost.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
      2localhost.org.apache.juli.FileHandler.prefix = localhost
      2localhost.org.apache.juli.FileHandler.rotatable = false
    • For the access log you have to modify CATALINA_BASE/conf/server.xml:

      Configuration of the access-log without rotation

      <Valve className="org.apache.catalina.valves.AccessLogValve"
          directory="logs"
          prefix="localhost_access_log" suffix=".txt"
          pattern="%h %l %u %t &quot;%r&quot; %s %b"
          rotatable="false" />
      Please note that the final dot from the  prefix fields has been removed (use  catalina instead of catalina., localhost instead of localhost. and localhost_access_log instead of localhost_access_log.). This procedure avoids that when eliminating the rotation date from the file name, the log would end up as catalina..log, localhost..log, etc.
    • Use the following configuration file for rsyslog: 

      /etc/rsyslog.d/45-tomcat.conf File

      $template tomcat,"<%PRI%>%timegenerated% %HOSTNAME% %syslogtag% %msg%"
       
      # File catalina.log
      $InputFileName CATALINA_BASE/logs/catalina.log
      $InputFileTag web.tomcat.catalina.ENV.APP.CLON:
      $InputFileStateFile stat-Tomcat-catalina
      $InputFileSeverity info
      $InputFileFacility local7
      $InputFilePollInterval 1
      $InputFilePersistStateInterval 1
      $InputRunFileMonitor
       
      # File localhost.log
      $InputFileName CATALINA_BASE/logs/localhost.log
      $InputFileTag web.tomcat.app.ENV.APP.CLON:
      $InputFileStateFile stat-Tomcat-localhost
      $InputFileSeverity info
      $InputFileFacility local7
      $InputFilePollInterval 1
      $InputFilePersistStateInterval 1
      $InputRunFileMonitor
       
      # File catalina.out
      $InputFileName CATALINA_BASE/logs/catalina.out
      $InputFileTag web.tomcat.out.ENV.APP.CLON: 
      $InputFileStateFile stat-Tomcat-out
      $InputFileSeverity info
      $InputFileFacility local7
      $InputFilePollInterval 1
      $InputFilePersistStateInterval 1
      $InputRunFileMonitor
       
      # File access log
      $InputFileName CATALINA_BASE/logs/localhost_access_log.txt
      $InputFileTag web.tomcat.access-clf.ENV.APP.CLON:
      $InputFileStateFile stat-Tomcat-accessClf
      $InputFileSeverity info
      $InputFileFacility local7
      $InputFilePollInterval 1
      $InputFilePersistStateInterval 1
      $InputRunFileMonitor
       
      # SSL config for logtrust secure relay
      #$DefaultNetstreamDriver gtls # use gtls netstream driver
      #$DefaultNetstreamDriverCAFile /etc/rsyslog.d/ca.crt
      #$DefaultNetstreamDriverCertFile /etc/rsyslog.d/user.crt
      #$DefaultNetstreamDriverKeyFile /etc/rsyslog.d/user.key
      #$ActionSendStreamDriverMode 1 # require TLS for the connection
      #$ActionSendStreamDriverAuthMode x509/name
      #$ActionSendStreamDriverPermittedPeer collector
       
      if $syslogtag contains 'web.tomcat.' and $syslogfacility-text == 'local7' then @@LOGTRUST-RELAY:PORT;tomcat
      :syslogtag, contains, "web.tomcat." ~
    You should personalize the previous configuration as follows:
    • Replace CATALINA_BASE by the absolute path where Tomcat logs reside.
    • Replace ENV.APP.CLON by the values you have assign to the application (for more information, check the Tag Structure section).
    • In the current example, the access log uses the type access-clf, but remember that you can also use the types access-combined and access-lt depending on the format you have configured (for more information, check Access Log section).
    • Replace LOGTRUST-RELAY and PORT with your Devo relay server and port.
    • If you choose to send to a secure relay, please make sure you uncomment section SSL of the configuration file.
    • Verify that the file you are going to process and the directory where is it located can be read by the user running rsyslog.

    Please see below a configuration example for the file rotation by using the logrotate tool:

    /etc/logrotate.d/tomcat File

    CATALINA_BASE/logs/catalina.out
    CATALINA_BASE/logs/catalina.log
    CATALINA_BASE/logs/localhost.log
    CATALINA_BASE/logs/localhost_access_log.txt
    {
        rotate 14
        daily
        copytruncate
        missingok
        notifempty
        delaycompress
        compress
        sharedscripts
        lastaction
            service rsyslog stop
            rm -f /var/spool/rsyslog/stat-Tomcat-{catalina,localhost,out,accessClf}
            service rsyslog start
        endscript
    }
    Please note that this configuration involves a restart of rsyslog.
  • Windows environments

    The basic mechanism to send to Devo would be to rely in other tools such as Snare Epilog or MagicLog.

Have we answered your question?

If not, please contact our technical support team via email by clicking the button below.

CONTACT US