- The Devo data analytics platform
- Getting started
- Domain administration
-
Sending data to Devo
-
The Devo In-House Relay
- Installing the Devo Relay
- Configuring the In-House Relay
- Relay migration
- Sending SSL/TLS encrypted events to the Devo relay
- Relay troubleshooting tips (v1.4.2)
-
Event sources
- Unix-like machines
- Windows
- MacOS X
- Cloud services
- Commercial products
- Custom apps
-
Universal Agent
- Pre-integrated query packs
- Data querying in Devo
-
Universal Agent Manager deployment
- Universal Agent Manager - CentOS 7 Deployment
- Universal Agent Manager - CentOS 8 Deployment
- Universal Agent Manager - Debian 9 Deployment
- Universal Agent Manager - Debian 10 Deployment
- Universal Agent Manager - RHEL 7 Deployment
- Universal Agent Manager - RHEL 8 Deployment
- Universal Agent Manager - Ubuntu 18 Deployment
- Universal Agent deployment
- Performance considerations
- Other data collection methods
- Uploading log files
- Devo software
-
The Devo In-House Relay
-
Parsers and collectors
- About Devo tags
- Special Devo tags and data tables
-
List of Devo parsers
- Business & Consumer
- Cloud technologies
- Databases
- Host and Operating Systems
-
Network and application security
- auth.secureauth
- auth.securenvoy
- av.mcafee
- av.sophos
- box.iptables
- edr.cylance
- edr.fireeye.alerts
- edr.minervalabs.events
- edr.paloalto
- endpoint.symantec
- firewall.checkpoint
- firewall.cisco firepower and vpn.cisco
- firewall.fortinet
- firewall.huawei
- firewall.juniper
- firewall.paloalto
- firewall.pfsense
- firewall.sonicwall
- firewall.sophos
- firewall.sophos.xgfirewall
- firewall.stonegate
- firewall.windows
- ids.extrahop
- mail.proofpoint
- nac.aruba
- network.meraki
- network.versa
- network.vmware
- proxy.bluecoat
- proxy.forcepoint
- proxy.squid
- uba.varonis
- vuln.beyondtrust
- vpn.pulsesecure.sa
- Network connectivity
- Web servers
- Technologies supported in CEF syslog format
- Collectors
-
Searching data
- Accessing data tables
-
Building a query
- Data types in Devo
- Build a query in the search window
- Build a query using LINQ
- Working with JSON objects in data tables
- Subqueries
-
Operations reference
-
Aggregation operations
- Average (avg)
- Count (count)
- First (first)
- First not null (nnfirst)
- HyperLogLog++ (hllpp)
- HyperLogLog++ Count Estimation (hllppcount)
- Last (last)
- Last not null (nnlast)
- Maximum (max)
- Median / 2nd quartile / Percentile 50 (median)
- Minimum (min)
- Non-null average (nnavg)
- Non-null standard deviation (biased) (nnstddev)
- Non-null standard deviation (unbiased) (nnustddev)
- Non-null variance (biased) (nnvar)
- Non-null variance (unbiased) (nnuvar)
- Percentile 10 (percentile10)
- Percentile 25 / 1st quartile (percentile25)
- Percentile 5 (percentile5)
- Percentile 75 / 3rd quartile (percentile75)
- Percentile 90 (percentile90)
- Percentile 95 (percentile95)
- Standard deviation (biased) (stddev)
- Standard deviation (unbiased) (ustddev)
- Sum (sum)
- Sum Square (sum2)
- Variance (biased) (var)
- Variance (unbiased) (uvar)
-
Arithmetic group
- Absolute value (abs)
- Addition, sum, plus / Concatenation (add, +)
- Ceiling (ceil)
- Cube root (cbrt)
- Division (div, \)
- Division remainder (rem, %)
- Floor (floor)
- Modulo (mod, %%)
- Multiplication, product (mul, *)
- Power (pow)
- Real division (rdiv, /)
- Rounding (round)
- Sign (signum)
- Square root (sqrt)
- Subtraction, minus / Additive inverse (sub, -)
-
Conversion group
- Duration (duration)
- Format date (formatdate)
- From base16, b16, hex (from16)
- From base64, b64 (from64)
- From UTF8 (fromutf8)
- From Z85, base85 (fromz85)
- Human size (humanSize)
- Make byte array (mkboxar)
- Parse date (parsedate)
- Regular expression, regexp (re)
- Template (template)
- Timestamp (timestamp)
- To base16, b16, hex (to16)
- To base64, b64, hex (to64)
- To BigInt (bigint)
- To boolean (bool)
- To Float (float)
- To image (image)
- To Int (int)
- To IPv4 (ip4)
- To IPv4 net (net4)
- To IPv6 (ip6)
- To IPv6 compatible (compatible)
- To IPv6 mapped (mapped)
- To IPv6 net (net6)
- To IPv6 translated (translated)
- To MAC address (mac)
- To string (str)
- To string (stringify)
- To UTF8 (toutf8)
- To Z85, base85 (toz85)
- Cryptography group
- Date group
- Flow group
- General group
-
Geolocation group
- Coordinates distance (distance)
- Geocoord (geocoord)
- Geographic coordinate system (coordsystem)
- Geohash (geohash)
- Geohash string (geohashstr)
- Geolocated Accuracy Radius with MaxMind GeoIP2 (mm2accuracyradius)
- Geolocated ASN (mmasn)
- Geolocated ASN with MaxMind GeoIP2 (mm2asn)
- Geolocated AS Organization Name with MaxMind GeoIP2 (mm2asorg)
- Geolocated AS owner (mmasowner)
- Geolocated City (mmcity)
- Geolocated City with MaxMind GeoIP2 (mm2city)
- Geolocated Connection Speed (mmspeed)
- Geolocated connection type with MaxMind GeoIP2 (mm2con)
- Geolocated Coordinates (mmcoordinates)
- Geolocated coordinates with MaxMind GeoIP2 (mm2coordinates)
- Geolocated Country (mmcountry)
- Geolocated Country with MaxMind GeoIP2 (mm2country)
- Geolocated ISP (mmisp)
- Geolocated ISP name with MaxMind GeoIP2 (mm2isp)
- Geolocated Latitude (mmlatitude)
- Geolocated Latitude with MaxMind GeoIP2 (mm2latitude)
- Geolocated Level 1 Subdivision with MaxMind GeoIP2 (mm2subdivision1)
- Geolocated Level 2 Subdivision with MaxMind GeoIP2 (mm2subdivision2)
- Geolocated Longitude (mmlongitude)
- Geolocated Longitude with MaxMind GeoIP2 (mm2longitude)
- Geolocated Organization (mmorg)
- Geolocated organization name with MaxMind GeoIP2 (mm2org)
- Geolocated Postal Code (mmpostalcode)
- Geolocated Postal Code with MaxMind GeoIP2 (mm2postalcode)
- Geolocated Region (mmregion)
- Geolocated Region Name (mmregionname)
- ISO-3166-1 Continent Alpha-2 Code (continentalpha2)
- ISO-3166-1 Continent Name (continentname)
- ISO-3166-1 Country Alpha-2 Code (countryalpha2)
- ISO-3166-1 Country Alpha-2 Continent (countrycontinent)
- ISO-3166-1 Country Alpha-3 Code (countryalpha3)
- ISO-3166-1 Country Latitude (countrylatitude)
- ISO-3166-1 Country Longitude (countrylongitude)
- ISO-3166-1 Country Name (countryname)
- Latitude (latitude)
- Latitude and longitude coordinates (latlon)
- Longitude (longitude)
- Parse geocoord format (parsegeo)
- Represent geocoord format (reprgeo)
- Round coordinates (gridlatlon)
- JSON group
- Logic group
-
Mathematical group
- Arc cosine (acos)
- Arc sine (asin)
- Arc tangent (atan)
- Bitwise AND (band, &)
- Bitwise left shift (lshift, <<)
- Bitwise NOT (bnot, ~)
- Bitwise OR (bor, |)
- Bitwise right shift (rshift, >>)
- Bitwise unsigned right shift (urshift, >>>)
- Bitwise XOR (bxor, ^)
- Cosine (cos)
- e (mathematical constant) (e)
- Exponential: base e (exp)
- Hyperbolic cosine (cosh)
- Hyperbolic sine (sinh)
- Hyperbolic tangent (tanh)
- Logarithm: base 2 (log2)
- Logarithm: base 10 (log10)
- Logarithm: natural / arbitrary base (log)
- Pi (mathematical constant) (pi)
- Sine (sin)
- Tangent (tan)
- Meta Analysis group
- Name group
-
Network group
- HTTP Status Description (httpstatusdescription)
- HTTP Status Type (httpstatustype)
- IP Protocol (ipprotocol)
- IP Reputation Score (reputationscore)
- IP Reputation Tags (reputation)
- IPv4 legal use (purpose)
- IPv6 host number (host)
- IPv6 routing number (routing)
- Is IPv4 (ipip4)
- Is Private IPv4 (isprivate)
- Is Public IPv4 (ispublic)
- Squid Black Lists Flags (sbl)
- Order group
-
Packet group
- Ethernet destination MAC address (etherdst)
- Ethernet payload (etherpayload)
- Ethernet source MAC address (ethersrc)
- Ethernet status (etherstatus)
- Ethernet tag (ethertag)
- EtherType (ethertype)
- Has Ethernet frame (hasether)
- Has IPv4 datagram (hasip4)
- Has TCP segment (hastcp)
- Has UDP datagram (hasudp)
- IPv4 destination address (ip4dst)
- IPv4 differentiated services (ip4ds)
- IPv4 explicit congestion notification (ip4ecn)
- IPv4 flags (ip4flags)
- IPv4 fragment offset (ip4fragment)
- IPv4 header checksum (ip4cs)
- IPv4 header length (ip4hl)
- IPv4 identification (ip4ident)
- IPv4 payload (ip4payload)
- IPv4 protocol (ip4proto)
- IPv4 source address (ip4src)
- IPv4 status (ip4status)
- IPv4 time to live (ip4ttl)
- IPv4 total length (ip4len)
- IPv4 type of service (ip4tos)
- TCP ACK (tcpack)
- TCP checksum (tcpcs)
- TCP destination port (tcpdst)
- TCP flags (tcpflags)
- TCP header length (tcphl)
- TCP payload (tcppayload)
- TCP sequence number (tcpseq)
- TCP source port (tcpsrc)
- TCP status (tcpstatus)
- TCP urgent pointer (tcpurg)
- TCP window size (tcpwin)
- UDP checksum (udpcs)
- UDP destination port (udpdst)
- UDP length (udplen)
- UDP payload (udppayload)
- UDP source port (udpsrc)
- UDP status (udpstatus)
- Statistical group
-
String group
- Contains (has, ->)
- Contains - case insensitive (weakhas)
- Contains tokens (toktains)
- Contains tokens - case insensitive (weaktoktains)
- Edit distance: Damerau (damerau)
- Edit distance: Hamming (hamming)
- Edit distance: Levenshtein (levenshtein)
- Edit distance: OSA (osa)
- Ends with (endswith)
- Format number (formatnumber)
- Hostname public suffix (publicsuffix)
- Hostname root domain (rootdomain)
- Hostname root prefix (rootprefix)
- Hostname root suffix (rootsuffix)
- Hostname subdomains (subdomain)
- Hostname top level domain (topleveldomain)
- Is empty (isempty)
- Is in (`in`, <-)
- Is in - case insensitive (weakin)
- Length (length)
- Locate (locate)
- Lower case (lower)
- Matches (matches, ~)
- Peek (peek)
- Replace all (replaceall)
- Replace first (replace)
- Shannon entropy (shannonentropy)
- Split (split)
- Split regexp (splitre)
- Starts with (startswith)
- Substitute (subs)
- Substitute all (subsall)
- Substring (substring)
- Trim both sides (trim)
- Trim the left side (ltrim)
- Trim the right side (rtrim)
- Upper case (upper)
-
Web group
- Absolute URI (absoluteuri)
- Opaque URI (opaqueuri)
- URI authority (uriauthority)
- URI fragment (urifragment)
- URI host (urihost)
- URI path (uripath)
- URI port (uriport)
- URI query (uriquery)
- URI scheme (urischeme)
- URI ssp (urissp)
- URI user (uriuser)
- URL decode (urldecode)
- User Agent Company (uacompany)
- User Agent Company URL (uacompanyurl)
- User Agent Device Icon (uadeviceicon)
- User Agent Device Information URL (uadeviceinfourl)
- User Agent Device Type (uadevicetype)
- User Agent Family (uafamily)
- User Agent Icon (uaicon)
- User Agent Information URL (uainfourl)
- User Agent is Robot (uaisrobot)
- User Agent Name (uaname)
- User Agent OS Company (uaoscompany)
- User Agent OS Company URL (uaoscompanyurl)
- User Agent OS Family (uaosfamily)
- User Agent OS Icon (uaosicon)
- User Agent OS Name (uaosname)
- User Agent OS URL (uaosurl)
- User Agent Type (uatype)
- User Agent URL (uaurl)
- User Agent Version (uaversion)
-
Aggregation operations
-
Working in the search window
-
Generate charts
- Affinity chord diagram
- Availability timeline
- Bipartite chord diagram
- Bubble chart
- Chart aggregation
- Custom date chart aggregation
- Flame graph
- Flat world map by coordinates
- Flat world map by country
- Google animated heat map
- Google area map
- Google heat map
- Graph diagram
- Histogram
- Pew Pew map
- Pie chart
- Pie layered chart
- Punch card
- Robust Random Cut Forest chart
- Sankey diagram
- Scatter plot
- Time heatmap
- Triple exponential chart
- Voronoi treemap
- Data enrichment
- Setting up a data table
- Advanced data operations
- Use case: eCommerce behavior analysis
-
Generate charts
- Managing your queries
- Best practices for data search
- Monitoring tables
- Activeboards
-
Dashboards
- Create a new dashboard
-
Working with dashboard widgets
- Availability timeline widget
- Chord diagram widget
- Circle world map widget
- Color key value widget
- Color world map widget
- Column chart widget
- Comparative chart widget
- Funnel widget
- Gauge meter widget
- Google heatmap widget
- Heat calendar widget
- Line chart widget
- Monitoring widget
- Pie chart widget
- Punch card widget
- Sectored pie chart widget
- Table widget
- Time heatmap widget
- Tree diagram widget
- Voronoi tree widget
- Configuring and sharing dashboards
- Alerts and notifications
- Panels
- Applications
- Tools
- Flow
- Social Intelligence
- API reference
- Release notes
JDK java.util.logging
JDK comes with the java.util.logging (JUL) library which separates event generation from the filter and final consolidation.
The use of the JUL library is described in detail on the Oracle website. For more information, see Java Logging Overview and Javadoc de LogManager.
Log consolidation and distribution are managed by a handler object. The standard JDK comes with handlers that write logs to a console, save them to a file, and more. However, it does not include one for sending the logs using syslog so we will use a third-party object available in the Scoja client library.
This article describes the configuration of the handlers provided by Scoja.
The syslog handler
The handler that the Scoja client library provides for sending log data using syslog is called org.scoja.client.jul.SyslogHandler. It contains properties that enable you to configure syslog elements (such as tag, facility, and level) and transport details (such as protocol, host, and port).
Configuring a single host destination
Configuration for a single destination
handlers = org.scoja.client.jul.SyslogHandler, java.util.logging.ConsoleHandler
org.scoja.client.jul.SyslogHandler.level = INFO
org.scoja.client.jul.SyslogHandler.protocol = stream
org.scoja.client.jul.SyslogHandler.host = relay
org.scoja.client.jul.SyslogHandler.port = 514
org.scoja.client.jul.SyslogHandler.retries = 3
org.scoja.client.jul.SyslogHandler.facility = local0
org.scoja.client.jul.SyslogHandler.levelmap = FINE,CONFIG,INFO,WARNING,950,SEVERE
org.scoja.client.jul.SyslogHandler.program = my.app.mytest
org.scoja.client.jul.SyslogHandler.message = \
${date}|${sequence}|${thread}|${logger}|${level}|${class}|${method}|${message:URL:|}|${exceptionclass}|${exceptionmessage}|${stacktrace}
The following table describes the properties that configure the handler.
Property | Value |
---|---|
.level | The severity level of events to be sent to the host destination. For example, INFO. Any events with a lesser priority level will be discarded. |
.protocol | This indicates the transport protocol to use. For TCP, the value is stream. For UDP, the value is packet. |
.host | The hostname of the destination server. |
.port | The port of the destination server. |
.retries | The number of retries to attempt when there is a connection problem. After this number of retries have been attempted, the event will be discarded. |
.facility | This value combines the event source and the syslog severity level value (0 - 7). |
.levelmap | Comma-separated list of the syslog levels to include in the logs sent to the destination. |
.program | The tag used to identify the log and which will be applied to all log events. |
.message | The user-defined template that dictates how the outgoing log data will be structured in the message body. See the Templates section below for a more detailed explanation. |
Configuring multiple host destinations
In order to send a log simultaneously to multiple host destinations, index and repeat the protocol, host, and port properties, as shown in the following example:
Tho locations configuration
handlers = org.scoja.client.jul.SyslogHandler, java.util.logging.ConsoleHandler
org.scoja.client.jul.SyslogHandler.level = INFO
org.scoja.client.jul.SyslogHandler.protocol-1 = stream
org.scoja.client.jul.SyslogHandler.host-1 = relay
org.scoja.client.jul.SyslogHandler.port-1 = 514
org.scoja.client.jul.SyslogHandler.protocol-2 = packet
org.scoja.client.jul.SyslogHandler.host-2 = syslog
org.scoja.client.jul.SyslogHandler.port-2 = 514
org.scoja.client.jul.SyslogHandler.retries = 3
org.scoja.client.jul.SyslogHandler.facility = local0
org.scoja.client.jul.SyslogHandler.levelmap = FINE,CONFIG,INFO,WARNING,950,SEVERE
org.scoja.client.jul.SyslogHandler.program = my.app.jultest
org.scoja.client.jul.SyslogHandler.message = \
${date}|${sequence}|${thread}|${logger}|${level}|${class}|${method}|${message:URL:|}|${exceptionclass}|${exceptionmessage}|${stacktrace}
- The SyslogHandler handler, explores if there are host-1, host-2, etc. properties.
If they are found, then it will add new destinations, but once one is missing, it stops adding the destinations.
- For each destination, it uses the corresponding protocol-i and port-i. If not found, it uses the protocol and port values or the default values.
Templates
The content of both message and program properties are templates where the dollar sign ($) can be used as a reference to a JUL event element.
The element name and an optional argument that determines the way the element should be handled are contained in braces ({...}) following the dollar sign ($).
Inside the braces, the element and its possible argument are separated by a colon ( : ).
- The element name is mandatory.
- The argument is optional and depends on the element type.
Elements that have no arguments are:
- class - the name of the class that generates the event.
- method - the name of the method that generates the event.
- thread - the name of the thread that generates the event.
- sequence - the sequence number of the event within the ones generated by the same logger.
- level - the name of the JUL level
- level# - same as above but in numerical form
- exception class - the name of the class if there has been an exception.
The elements that may have arguments are:
- logger - the name of the logger. It has a pre-processing argument.
- date - the moment when the JUL event is generated. It has an argument with a date format admissible for java.text.SimpleDateFormat. If omitted, it uses the yyyy-MM-dd HH:mm:ss.SSS format.
- message - the message of the JUL event. It has a pre-processing argument.
- parameter - a parameter of the JUL event. It supports 2 arguments, the parameter index (0 if omitted) and a pre-processor.
- exception message - the exception message. It has a pre-processing argument.
- stacktrace - the stack trace of the exception. It has a pre-processing argument.
Pre-processing arguments are used to format the element, eliminating any characters that may truncate the syslog event. They are always optional. If no pre-processing argument is specified, the element is not re-formatted. These arguments are composed of two values separated by a colon ( : ) :
- First, the name of the pre-processor. This is URL if you want to replace the disruptive characters with a %. This is C if you want to replace the disruptive characters with C escape sequences.
- Second, the dangerous characters to be replaced before the data is sent.
For example, ${message:URL:|} indicates that and | character found in the message element of the event should be replaced with a %.
Examples
Suppose you want to show the exceptions traces.
- You refer to an exception trace with:
${stacktrace}
- The line endings of the tracers usually are characters that cannot go into a syslog event, so you must do some processing to remove them:
${stacktrace:URL:\n}
- Since Scoja handler is designed to send through syslog, it knows that whenever a pre-processing is done and no special characters are indicated, it must include those dangerous characters to syslog, so you can use:
${stacktrace:URL}
Suppose you want to build a message with various elements of the JUL event separated by pipes (|) and that your application generates messages where any character may appear, including line ends and pipes.
- We cannot use ${message} because there might be line ends.
- We cannot use ${message:URL}because there might be pipes.
- You should include the pipe and use:
${message:URL:|}
Extracting info from the message
The application of the syslog tag is critical for Devo to correctly process events. The SyslogHandler handler lets you define the tag using the program property, and this tag is applied to all events processed by the handler. However, if you want to apply multiple tags to different events in a simple log, you can use the MessageSyslogAttributer handler from Scoja. This handler allows the extraction of information from the message, delete it and use it to determine other elements of the syslog event.
The following example shows the use of this handler to determine which tag to apply based on the prefix of the message element.
Configuration with multiple tags
handlers = org.scoja.client.jul.MessageSyslogAttributer, java.util.logging.ConsoleHandler
org.scoja.client.jul.MessageSyslogAttributer.target = org.scoja.client.jul.SyslogHandler
org.scoja.client.jul.MessageSyslogAttributer.pattern = ^([a-zA-Z0-9.]+):
org.scoja.client.jul.MessageSyslogAttributer.keyGroup = 1
org.scoja.client.jul.MessageSyslogAttributer.dropGroup = 0
org.scoja.client.jul.MessageSyslogAttributer.program = my.app.app1.out
org.scoja.client.jul.MessageSyslogAttributer.message = ${message}
org.scoja.client.jul.MessageSyslogAttributer.key-1 = info
org.scoja.client.jul.MessageSyslogAttributer.program-1 = my.app.app1.info
org.scoja.client.jul.MessageSyslogAttributer.key-2 = error
org.scoja.client.jul.MessageSyslogAttributer.program-2 = my.app.app1.error
org.scoja.client.jul.MessageSyslogAttributer.priority-2 = local1.error
org.scoja.client.jul.SyslogHandler.level = INFO
org.scoja.client.jul.SyslogHandler.protocol-1 = stream
org.scoja.client.jul.SyslogHandler.host = relay
org.scoja.client.jul.SyslogHandler.port = 514
org.scoja.client.jul.SyslogHandler.message = \
${date}|${sequence}|${thread}|${logger}|${level}|${class}|${method}|${message:URL:|}|${exceptionclass}|${exceptionmessage}|${stacktrace}
MessageSyslogAttributer is an intermediate handler that processes the JUL event and sends the result to the handler specified by the .target property. This should always be set to org.scoja.client.jul.SyslogHandler since it is the only handler capable of recieving events processed by MessageSyslogAttributer.
MessageSyslogAttributer scans the message, searching for the regular expression specified by the value of the .pattern property. In the example above, .pattern identifies the message prefix up to the first colon (:). If this pattern is found, then MessageSyslogAttributer processes the event as follows:
- It deletes the character sequence captured with the group of parentheses specified in the dropGroup property from the message. If this property is missing, it doesn't delete anything from the message. The parentheses are numbered in the regular expression the usual way and they can be consulted on the javadoc of the Pattern class.
- Extracts the sequence captured with the group of parentheses specified in the keyGroup property (called K).
- Search for the first i in which the key-i property is equal to K.
- Browse all other properties that have that i and writes the values in the JUL event.
As shown in the above example, the prefix is captured until the colon (:). Since dropGroup is 0, everything captured by that expression is removed from the message, so the message is defined as the suffix after the first colon (:), but excluding the colon.
Two key properties are defined, key-1 and key-2, with info and error values, respectively. The K value that we extracted is compared with these values.
- If the result is info, all the JUL event elements with a definition ending in -1 will be changed. In this case, is the program-1 so, only the tag from my.app.app1.info will be changed.
- If the result is error, the program-2 and priority-2 will be converted to my.app.app1.error and local1.error, respectively.
In this case, if you want to send an event to Devo and place it on my.app.app1.info, please use the following code:
Sending to my.app.app1.info
Logger log = Logger.getLogger("MyClass");
log.info("info:rest of the event message");
If you want to send to my.app.app1.error, you should use the following code:
Sending to my.app.app1.error
Logger log = Logger.getLogger("MyClass");
log.info("error:rest of the event message");