• Services & Support
  • Devo.com
  • Contact
    • Contact Us
    • Request a Demo
    • Partner Inquiry
  • Log In
    • USA Devo
    • EU Devo
  • The Devo data analytics platform
    • How Devo indexes data
    • How Devo works
    • Key concepts
  • Getting started
    • Sign up and log in
    • Navigating the Devo app
    • User preferences
    • Devo video tutorials
  • Domain administration
    • Users and roles
      • Managing users
      • Monitoring user activity
      • Managing roles
        • Assign resources to a role
      • Role permissions
      • Role mapping
    • Security credentials
      • Access keys
      • X.509 certificates
      • Authentication tokens
    • Applications gallery
    • Domain preferences
    • User authentication
      • Password
      • SAML
        • Google as an identity provider
        • Okta as an identity provider
        • OneLogin as an identity provider
        • O365/Azure AD as an identity provider
      • OpenID
    • Data processes and feeds
      • Aggregation tasks
      • Injections
      • Permalinks
      • API & OData feeds
  • Sending data to Devo
    • The Devo In-House Relay
      • Installing the Devo Relay
        • Install the Relay on an Ubuntu box (v1.4.2)
        • Install the Relay on a Red Hat or CentOS box (v1.4.2)
        • Install with Docker
      • Configuring the In-House Relay
        • Relay rules
          • Defining a relay rule
          • The 4 predefined relay rules
          • 5 common relay rule scenarios
            • Scenario 1: Apply a fixed tag to all events
            • Scenario 2: Apply a Devo tag based on data found in the inbound event
            • Scenario 3: Filter out unwanted events
            • Scenario 4: Assign dynamic Devo tag using inbound source data
            • Scenario 5: Appending the inbound syslog tag to the outbound Devo tag
          • Using regex in relay rules
        • Customizing In-House Relay settings
        • Managing the relay on the command line
        • Setting up High-Availability with Keepalived (v1.4.2)
        • Relay buffers
      • Relay migration
      • Sending SSL/TLS encrypted events to the Devo relay
      • Relay troubleshooting tips (v1.4.2)
        • Relay troubleshooting tips (v1.4.0)
        • Relay troubleshooting tips (for versions prior to 1.4.0)
    • Event sources
      • Unix-like machines
        • Installing Devo packages for *nix
        • Third-party syslog tools configuration
          • rsyslog
            • Simple sending using rsyslog
            • Secure sending using rsyslog
            • Monitoring files using rsyslog
            • Obsolete legacy format
              • Simple sending using rsyslog (Obsolete legacy format)
              • Secure sending using rsyslog (Obsolete legacy format)
              • Monitoring files using rsyslog (Obsolete legacy format)
          • syslog-ng
            • Simple sending using syslog-ng
            • Secure sending using syslog-ng
            • Monitoring files using syslog-ng
          • syslog/syslogd
        • SELinux configuration conflicts
      • Windows
        • Devo Agent for Windows
        • Configuring WMI for Devo file monitoring
        • NXLog for Windows event collection
      • MacOS X
      • Cloud services
        • AWS S3 Buckets
        • Microsoft Azure
      • Commercial products
      • Custom apps
        • Java apps
          • JDK java.util.logging
          • Scoja client library
          • Sample code
        • Node.js apps
        • Python apps
    • Other data collection methods
      • HTTP endpoint
      • Logstash
      • Fluentd
      • NXLog
    • Uploading log files
    • Devo software
  • Searching data
    • Accessing data tables
      • Run a search using a finder
        • Use a custom finder
          • Create a custom finder
          • Assign a custom finder to a role
          • Edit a custom finder
        • Use the aliased finder
          • Add a query to your aliased finder
      • Run a global search
      • Run a LINQ free text query
      • Run a search with selected columns only
        • Selecting specific columns in LINQ
        • Selecting specific columns with the Finder
        • Selecting unrevealed columns
    • Building a query
      • Data types in Devo
      • Build a query in the search window
        • Filter data
        • Filter column data using the OR selector
        • Create columns
        • Group data
        • Aggregate data
      • Build a query using LINQ
        • LINQ query examples
      • Working with JSON objects in data tables
      • Subqueries
      • Operations reference
        • Aggregation operations
          • Average (avg)
          • Count (count)
          • First (first)
          • First not null (nnfirst)
          • HyperLogLog++ (hllpp)
          • HyperLogLog++ Count Estimation (hllppcount)
          • Last (last)
          • Last not null (nnlast)
          • Maximum (max)
          • Median / 2nd quartile / Percentile 50 (median)
          • Minimum (min)
          • Non-null average (nnavg)
          • Non-null standard deviation (biased) (nnstddev)
          • Non-null standard deviation (unbiased) (nnustddev)
          • Non-null variance (biased) (nnvar)
          • Non-null variance (unbiased) (nnuvar)
          • Percentile 10 (percentile10)
          • Percentile 25 / 1st quartile (percentile25)
          • Percentile 5 (percentile5)
          • Percentile 75 / 3rd quartile (percentile75)
          • Percentile 90 (percentile90)
          • Percentile 95 (percentile95)
          • Standard deviation (biased) (stddev)
          • Standard deviation (unbiased) (ustddev)
          • Sum (sum)
          • Sum Square (sum2)
          • Variance (biased) (var)
          • Variance (unbiased) (uvar)
        • Arithmetic group
          • Absolute value (abs)
          • Addition, sum, plus / Concatenation (add, +)
          • Ceiling (ceil)
          • Cube root (cbrt)
          • Division (div, \)
          • Division remainder (rem, %)
          • Floor (floor)
          • Modulo (mod, %%)
          • Multiplication, product (mul, *)
          • Power (pow)
          • Real division (rdiv, /)
          • Rounding (round)
          • Sign (signum)
          • Square root (sqrt)
          • Subtraction, minus / Additive inverse (sub, -)
        • Conversion group
          • Duration (duration)
          • Format date (formatdate)
          • From base16, b16, hex (from16)
          • From base64, b64 (from64)
          • From UTF8 (fromutf8)
          • From Z85, base85 (fromz85)
          • Human size (humanSize)
          • Make byte array (mkboxar)
          • Parse date (parsedate)
          • Regular expression, regexp (re)
          • Template (template)
          • Timestamp (timestamp)
          • To base16, b16, hex (to16)
          • To base64, b64, hex (to64)
          • To BigInt (bigint)
          • To boolean (bool)
          • To Float (float)
          • To image (image)
          • To Int (int)
          • To IPv4 (ip4)
          • To IPv4 net (net4)
          • To IPv6 (ip6)
          • To IPv6 compatible (compatible)
          • To IPv6 mapped (mapped)
          • To IPv6 net (net6)
          • To IPv6 translated (translated)
          • To MAC address (mac)
          • To string (str)
          • To string (stringify)
          • To UTF8 (toutf8)
          • To Z85, base85 (toz85)
        • Cryptography group
          • MD5 hash function (md5)
          • SHA1 hash function (sha1)
          • SHA256 hash function (sha256)
          • SHA512 hash function (sha512)
        • Date group
          • Day / Day of the month (day)
          • Day of the week (dayofweek)
          • Day of the year (dayofyear)
          • Epoch milliseconds (epoch)
          • Hour (hour)
          • Millisecond (millisecond)
          • Minute (minute)
          • Month (month)
          • Period (period)
          • Second (second)
          • Today (today)
          • Tomorrow (tomorrow)
          • Year (year)
          • Yesterday (yesterday)
        • Flow group
          • Conditional (ifthenelse)
          • Decode, switch (decode)
          • Null value locator (nvl)
        • General group
          • Is not null (isnotnull)
          • Is null (isnull)
        • Geolocation group
          • Coordinates distance (distance)
          • Geocoord (geocoord)
          • Geographic coordinate system (coordsystem)
          • Geohash (geohash)
          • Geohash string (geohashstr)
          • Geolocated Accuracy Radius with MaxMind GeoIP2 (mm2accuracyradius)
          • Geolocated ASN (mmasn)
          • Geolocated ASN with MaxMind GeoIP2 (mm2asn)
          • Geolocated AS Organization Name with MaxMind GeoIP2 (mm2asorg)
          • Geolocated AS owner (mmasowner)
          • Geolocated City (mmcity)
          • Geolocated City with MaxMind GeoIP2 (mm2city)
          • Geolocated Connection Speed (mmspeed)
          • Geolocated connection type with MaxMind GeoIP2 (mm2con)
          • Geolocated Coordinates (mmcoordinates)
          • Geolocated coordinates with MaxMind GeoIP2 (mm2coordinates)
          • Geolocated Country (mmcountry)
          • Geolocated Country with MaxMind GeoIP2 (mm2country)
          • Geolocated ISP (mmisp)
          • Geolocated ISP name with MaxMind GeoIP2 (mm2isp)
          • Geolocated Latitude (mmlatitude)
          • Geolocated Latitude with MaxMind GeoIP2 (mm2latitude)
          • Geolocated Level 1 Subdivision with MaxMind GeoIP2 (mm2subdivision1)
          • Geolocated Level 2 Subdivision with MaxMind GeoIP2 (mm2subdivision2)
          • Geolocated Longitude (mmlongitude)
          • Geolocated Longitude with MaxMind GeoIP2 (mm2longitude)
          • Geolocated Organization (mmorg)
          • Geolocated organization name with MaxMind GeoIP2 (mm2org)
          • Geolocated Postal Code (mmpostalcode)
          • Geolocated Postal Code with MaxMind GeoIP2 (mm2postalcode)
          • Geolocated Region (mmregion)
          • Geolocated Region Name (mmregionname)
          • ISO-3166-1 Continent Alpha-2 Code (continentalpha2)
          • ISO-3166-1 Continent Name (continentname)
          • ISO-3166-1 Country Alpha-2 Code (countryalpha2)
          • ISO-3166-1 Country Alpha-2 Continent (countrycontinent)
          • ISO-3166-1 Country Alpha-3 Code (countryalpha3)
          • ISO-3166-1 Country Latitude (countrylatitude)
          • ISO-3166-1 Country Longitude (countrylongitude)
          • ISO-3166-1 Country Name (countryname)
          • Latitude (latitude)
          • Latitude and longitude coordinates (latlon)
          • Longitude (longitude)
          • Parse geocoord format (parsegeo)
          • Represent geocoord format (reprgeo)
          • Round coordinates (gridlatlon)
        • JSON group
          • Jq evaluation (jqeval)
          • Jq filter compilation (jqcompile)
          • Json value type (label)
          • To json (jsonparse)
        • Logic group
          • And (and)
          • Not (not)
          • Or (or)
        • Mathematical group
          • Arc cosine (acos)
          • Arc sine (asin)
          • Arc tangent (atan)
          • Bitwise AND (band, &)
          • Bitwise left shift (lshift, <<)
          • Bitwise NOT (bnot, ~)
          • Bitwise OR (bor, |)
          • Bitwise right shift (rshift, >>)
          • Bitwise unsigned right shift (urshift, >>>)
          • Bitwise XOR (bxor, ^)
          • Cosine (cos)
          • e (mathematical constant) (e)
          • Exponential: base e (exp)
          • Hyperbolic cosine (cosh)
          • Hyperbolic sine (sinh)
          • Hyperbolic tangent (tanh)
          • Logarithm: base 2 (log2)
          • Logarithm: base 10 (log10)
          • Logarithm: natural / arbitrary base (log)
          • Pi (mathematical constant) (pi)
          • Sine (sin)
          • Tangent (tan)
        • Meta Analysis group
          • Pragma value (pragmavalue)
          • Table name (tablename)
        • Name group
          • Any name matches (anymatches)
          • Glob pattern on names (nameglob)
        • Network group
          • HTTP Status Description (httpstatusdescription)
          • HTTP Status Type (httpstatustype)
          • IP Protocol (ipprotocol)
          • IP Reputation Score (reputationscore)
          • IP Reputation Tags (reputation)
          • IPv4 legal use (purpose)
          • IPv6 host number (host)
          • IPv6 routing number (routing)
          • Is IPv4 (ipip4)
          • Is Private IPv4 (isprivate)
          • Is Public IPv4 (ispublic)
          • Squid Black Lists Flags (sbl)
        • Order group
          • Equal (eq, =)
          • Equal - case insensitive (eqic)
          • Greater or equal (ge, >=)
          • Greater than (gt, >)
          • Less or equal (le, <=)
          • Less than (lt, <)
          • Not equal (ne, /=)
        • Packet group
          • Ethernet destination MAC address (etherdst)
          • Ethernet payload (etherpayload)
          • Ethernet source MAC address (ethersrc)
          • Ethernet status (etherstatus)
          • Ethernet tag (ethertag)
          • EtherType (ethertype)
          • Has Ethernet frame (hasether)
          • Has IPv4 datagram (hasip4)
          • Has TCP segment (hastcp)
          • Has UDP datagram (hasudp)
          • IPv4 destination address (ip4dst)
          • IPv4 differentiated services (ip4ds)
          • IPv4 explicit congestion notification (ip4ecn)
          • IPv4 flags (ip4flags)
          • IPv4 fragment offset (ip4fragment)
          • IPv4 header checksum (ip4cs)
          • IPv4 header length (ip4hl)
          • IPv4 identification (ip4ident)
          • IPv4 payload (ip4payload)
          • IPv4 protocol (ip4proto)
          • IPv4 source address (ip4src)
          • IPv4 status (ip4status)
          • IPv4 time to live (ip4ttl)
          • IPv4 total length (ip4len)
          • IPv4 type of service (ip4tos)
          • TCP ACK (tcpack)
          • TCP checksum (tcpcs)
          • TCP destination port (tcpdst)
          • TCP flags (tcpflags)
          • TCP header length (tcphl)
          • TCP payload (tcppayload)
          • TCP sequence number (tcpseq)
          • TCP source port (tcpsrc)
          • TCP status (tcpstatus)
          • TCP urgent pointer (tcpurg)
          • TCP window size (tcpwin)
          • UDP checksum (udpcs)
          • UDP destination port (udpdst)
          • UDP length (udplen)
          • UDP payload (udppayload)
          • UDP source port (udpsrc)
          • UDP status (udpstatus)
        • Statistical group
          • Approximated estimation (estimation)
          • HyperLogLog++ pack (pack)
          • HyperLogLog++ unpack (unpackhllpp)
        • String group
          • Contains (has, ->)
          • Contains - case insensitive (weakhas)
          • Contains tokens (toktains)
          • Contains tokens - case insensitive (weaktoktains)
          • Edit distance: Damerau (damerau)
          • Edit distance: Hamming (hamming)
          • Edit distance: Levenshtein (levenshtein)
          • Edit distance: OSA (osa)
          • Ends with (endswith)
          • Format number (formatnumber)
          • Hostname public suffix (publicsuffix)
          • Hostname root domain (rootdomain)
          • Hostname root prefix (rootprefix)
          • Hostname root suffix (rootsuffix)
          • Hostname subdomains (subdomain)
          • Hostname top level domain (topleveldomain)
          • Is empty (isempty)
          • Is in (`in`, <-)
          • Is in - case insensitive (weakin)
          • Length (length)
          • Locate (locate)
          • Lower case (lower)
          • Matches (matches, ~)
          • Peek (peek)
          • Replace all (replaceall)
          • Replace first (replace)
          • Shannon entropy (shannonentropy)
          • Split (split)
          • Split regexp (splitre)
          • Starts with (startswith)
          • Substitute (subs)
          • Substitute all (subsall)
          • Substring (substring)
          • Trim both sides (trim)
          • Trim the left side (ltrim)
          • Trim the right side (rtrim)
          • Upper case (upper)
        • Web group
          • Absolute URI (absoluteuri)
          • Opaque URI (opaqueuri)
          • URI authority (uriauthority)
          • URI fragment (urifragment)
          • URI host (urihost)
          • URI path (uripath)
          • URI port (uriport)
          • URI query (uriquery)
          • URI scheme (urischeme)
          • URI ssp (urissp)
          • URI user (uriuser)
          • URL decode (urldecode)
          • User Agent Company (uacompany)
          • User Agent Company URL (uacompanyurl)
          • User Agent Device Icon (uadeviceicon)
          • User Agent Device Information URL (uadeviceinfourl)
          • User Agent Device Type (uadevicetype)
          • User Agent Family (uafamily)
          • User Agent Icon (uaicon)
          • User Agent Information URL (uainfourl)
          • User Agent is Robot (uaisrobot)
          • User Agent Name (uaname)
          • User Agent OS Company (uaoscompany)
          • User Agent OS Company URL (uaoscompanyurl)
          • User Agent OS Family (uaosfamily)
          • User Agent OS Icon (uaosicon)
          • User Agent OS Name (uaosname)
          • User Agent OS URL (uaosurl)
          • User Agent Type (uatype)
          • User Agent URL (uaurl)
          • User Agent Version (uaversion)
    • Working in the search window
      • Generate charts
        • Affinity chord diagram
        • Availability timeline
        • Bipartite chord diagram
        • Bubble chart
        • Chart aggregation
        • Custom date chart aggregation
        • Flame graph
        • Flat world map by coordinates
        • Flat world map by country
        • Google animated heat map
        • Google area map
        • Google heat map
        • Graph diagram
          • Creating a graph diagram
          • Working in the graph diagram
          • Monitor intranet traffic to dangerous websites
        • Histogram
        • Pew Pew map
        • Pie chart
        • Pie layered chart
        • Punch card
        • Robust Random Cut Forest chart
        • Sankey diagram
        • Scatter plot
        • Time heatmap
        • Triple exponential chart
        • Voronoi treemap
      • Data enrichment
        • Upload a lookup table
        • Create a lookup table from a query
        • Add lookup values to your query
        • Manage and edit lookup tables
        • Threat lookups
      • Setting up a data table
        • Modifying the column layout
          • Arrange and resize columns
          • Hide and show columns
          • Change the position of column headers
          • Sort data
          • Setting a default table layout
        • Add a description to a data table
        • Autoparser
          • Autoparse a JSON object
      • Advanced data operations
        • Graphical correlation
          • Cross-Search Graph Diagram
          • Cross-Search Table Join
          • Cross-Search Sankey Diagram
          • Cross-Search Line Chart
        • Custom and union tables
          • Create a custom table
          • Create a union table
          • Manage custom and union tables
        • Set up a data source
        • Inject data to a new table
        • Drill downs
        • Manipulate your data using CyberChef
        • Time series report
      • Use case: eCommerce behavior analysis
        • Step 1. Error analysis using status codes
          • Specific analysis for 404 codes
          • Custom alerts for 404 errors
        • Step 2. Operating system ranking
          • Get the usage share of operating systems
          • Visualize the usage share of operating systems
        • Step 3. Country distribution
          • Build a histogram displaying country distribution
          • Geolocate IP addresses
    • Managing your queries
      • Rename a query
      • Favorite queries
      • Query history
      • Check currently running queries
      • Add a description to your query
      • Block a query
      • Share a query
      • Download query data
      • Close a query
      • Load query data into Excel using the Devo Connector add-in
      • Query priority
    • Best practices for data search
    • Monitoring tables
      • Web application monitoring
      • Alerts monitoring
  • Parsers and collectors
    • About Devo tags
    • Special Devo tags and data tables
    • List of Devo parsers
      • Business & Consumer
      • Cloud technologies
        • cdn.akamai
        • cloud.aws.cloudtrail.events
          • Forwarding the events using Node.js
          • Forwarding the events using Python
        • cloud.aws.cloudwatch.events
        • cloud.office365.siem
      • Databases
        • db.mysql
      • Host and Operating Systems
        • box.unix
        • box.vmware
        • box.win
        • box.win_nxlog
        • box.win_snare
      • Network and application security
        • auth.secureauth
        • auth.securenvoy
        • av.mcafee
        • av.sophos
        • box.iptables
        • edr.cylance
        • edr.fireeye.alerts
        • edr.minervalabs.events
        • endpoint.symantec
        • firewall.checkpoint
        • firewall.cisco firepower and vpn.cisco
        • firewall.fortinet
        • firewall.huawei
        • firewall.juniper
        • firewall.paloalto
          • Sending Palo Alto events to Devo relay using SSL
        • firewall.pfsense
        • firewall.sonicwall
        • firewall.sophos
        • firewall.sophos.xgfirewall
        • firewall.stonegate
        • firewall.windows
        • mail.proofpoint
        • nac.aruba
        • network.meraki
        • network.versa
        • proxy.bluecoat
        • proxy.forcepoint
        • proxy.squid
        • uba.varonis
        • vuln.beyondtrust
        • vpn.pulsesecure.sa
      • Network connectivity
        • netstat.netflow
        • dns.bind
        • dns.windows
        • ftp.iis
      • Web servers
        • web.apache
        • web.apache.mod-security
        • web.iis
        • web.jboss
        • web.nginx
        • web.tomcat
      • Technologies supported in CEF syslog format
    • Collectors
      • AWS collector
      • Google Cloud Platform collector
      • G Suite collectors
        • G Suite Alerts collector
        • G Suite Reports collector
      • Microsoft Azure collector
      • Microsoft Graph collector
      • Okta collectors
        • Okta collector
        • Okta Advanced Server Access collector
      • OneLogin collector
      • Rapid7 InsightVM collector
      • Salesforce collector
  • Activeboards
    • Creating an Activeboard
    • Working with Activeboard widgets
      • Area chart widgets
      • Column chart widgets
      • Donut chart widgets
      • Heatmap widgets
      • Line chart widgets
      • Pie chart widgets
      • Simple value widgets
      • Table widgets
      • Voronoi diagram widgets
    • Working with Activeboard inputs
    • LINQ syntax differences between Activeboards and the search window
    • Setting a time range in Activeboards
    • Working with Activeboards
      • Active Refresh
    • Sharing Activeboards
  • Dashboards
    • Create a new dashboard
    • Working with dashboard widgets
      • Availability timeline widget
      • Chord diagram widget
      • Circle world map widget
      • Color key value widget
      • Color world map widget
      • Column chart widget
      • Comparative chart widget
      • Funnel widget
      • Gauge meter widget
      • Google heatmap widget
      • Heat calendar widget
      • Line chart widget
        • Customize your Line chart
      • Monitoring widget
      • Pie chart widget
      • Punch card widget
      • Sectored pie chart widget
      • Table widget
      • Time heatmap widget
      • Tree diagram widget
      • Voronoi tree widget
    • Configuring and sharing dashboards
  • Alerts and notifications
    • Creating new alerts
      • Alert trigger methods
        • Each alert type
        • Several alert type
        • Low alert type
          • Inactivity alert
        • Rolling alert type
        • Deviation alert type
        • Gradient alert type
      • Create an alert based on triggered alerts
    • Configuring alerts
      • Manage defined alerts
      • Manage sending policies
      • Manage delivery methods
        • Email delivery methods
        • HTTP-JSON delivery methods
        • Service Desk delivery methods
        • Jira delivery methods
        • Pushover delivery methods
        • PagerDuty delivery methods
        • Slack delivery methods
      • Manage anti-flooding policies
      • Make an alert available for panels
      • Pre-installed alert reference
    • Managing triggered alerts
      • Add a comment to a triggered alert
      • Apply a filter for post-processing
  • Panels
    • Create and customize a panel
    • Adding an alert to a panel
    • Adding a query to a panel
    • Using panels
  • Applications
    • Devo Security Operations
      • Overview Dashboard
      • Triage
        • Triaging alerts
        • Triaging investigations
      • Investigations
      • Threat Hunting
      • Use cases
        • Phishing attack
        • Command & Control
        • Alerting system status
    • Devo Stats
      • Working in the Devo Stats application
      • Application tabs and widgets
        • User tab
        • Volume tab
        • Query tab
          • User Query Info
          • CPU Query Info
          • CPU Query Info Multidomain
        • Status tab
    • Security Insights
      • Installing Security Insights
        • Security Insights lookups
      • Configuring Security Insights
      • Navigating Security Insights
        • Overview tab
        • Threats tab
        • Network tab
        • DNS tab
        • Firewall tab
        • Proxy tab
        • Access tab
        • Web tab
        • IDS tab
    • Service operations
      • Basic concepts
      • Installation
      • Global models
      • Technologies configuration
      • Models configuration
      • Service overview
      • Incidents viewer
      • Monitors
      • User experience management
      • Use case for service operations
        • Initial analysis
        • Model configuration
        • Running the model
        • Incidents
    • Systems Monitoring
      • Basic monitoring
      • Advanced monitoring
  • Tools
    • Data Explorer
    • Query App
    • Time Series Analytics
  • Social Intelligence
  • API reference
    • REST API
      • Authorizing REST API requests
      • Running queries with the REST API
        • Forward response to HDFS
        • Forward response to Amazon S3
        • Forward response to SNMP
        • Forward response to email
        • Send requests with Postman
      • Job requests
    • Provisioning API
      • Authorizing Provisioning API requests
      • Common operations
        • User operations
        • Domain operations
        • Certificates
        • Role management
        • Domain resources management
      • Reseller operations
        • Price plans
      • Role specification and examples
    • Alerting API
      • Working with alert definitions
    • Using and managing OData feeds
      • Connecting with Excel
      • Connecting with Tableau
      • Connecting with Power BI
  • Release notes
    • 6.0.0
    • 6.0.1
    • 6.1.0
    • 6.1.1
    • 6.1.2
    • 6.1.3
    • 6.1.4
    • 6.2.0
    • 6.3.0
    • 6.3.1
    • 6.3.2
    • 6.3.3-1
    • 6.4.0
    • 6.4.1
    • 6.4.3
    • 6.5.0
    • 6.5.1
    • 6.5.2
    • 6.6.0
    • 6.6.1
    • 6.6.2
    • 6.7.0
PREVIOUS
JDK java.util.logging
NEXT
Sample code

Sending data to Devo / Event sources / Custom apps / Java apps / Scoja client library

Download as PDF

Scoja client library

The Scoja project has a library for the sending via syslog from Java applications. This library abstracts the protocol's operation, but still lets you select the sending mechanism and configure all the details of a syslog event.

In order to use the library, you have to add the scoja-cc.jar (v1.4.0) and scoja-client.jar (v1.5.1) files to the CLASSPATH.

Example 1 

This is a simple example that sends logs via TCP to a syslog collector listening on port 514 (syslog standard port). Devo In-house Relay listens on that port. 

Introductory example

import org.scoja.client.Syslogger;
import org.scoja.client.ReusingTCPSyslogger;
 
public class Test {
  static final String HOST = "relay";
  static final int PORT = 514;
 
  public static void main(String args[]) throws Exception {
    Syslogger logger = new ReusingTCPSyslogger(HOST, PORT);
    logger.log("Test event");
    logger.close();
  }
}

The ReusingTCPSyslogger class sends the syslog events via TCP to the host and port configured in its constructor. It extends the Syslogger class that accesses the details of the syslog protocol except the transport details.

The log method builds and sends an event with the message that contains its argument. There are several syslogger subclasses for the most common transport protocols including UDP and SSL. There are also subclasses that retry automatically, broadcast, and more in the event of errors.

Using objects like syslogger in the event-sending code makes it easy to change the mechanism you use for sending. 

The syslogger subclasses can be used to send as many events as needed. In fact, it is a good idea to reuse them to send as many events as possible, thereby reducing the costs associated with opening new sessions. We recommend creating the necessary sysloggers at the beginning of the application and keep them open during the execution, using them to send events as needed. However, if the application only needs a syslogger at certain times, you can use the close() method as in Example 1.

All sysloggers are thread-safe, so they can be used in concurrent contexts shared in multiple threads simultaneously.

Details

Let's take syslog events that contain the following fields: when (date), priority (int), and host, tag, and message (all string). 

With the log() method used in the first example, we could only control the message content. However, there's a method you can use to specify the value for each of these fields:

General log method

public void log(Date when, int priority, String host, String tag, String message)
throws LoggingException;

Let's look at two ways the library can set appropriate values for the event fields:

First, we can make the call using "empty" values which tells the library that it needs to replace them with fixed values. The empty values are represented as null, except for priority which is using org.scoja.common.PriorityUtils.UNKNOWN_PRIORITY. In this case, the library will calculate fixed values to replace all fields except message. 

logger.log(null, UNKNOWN_PRIORITY, null, null, message);

Second, we can use multiple versions of the log method with fewer arguments, the first being the one used in the first example:

Simplified log methods

public void log(String message)
throws LoggingException;
 
public void log(int priority, String message)
throws LoggingException;   
 
public void log(int priority, String host, String tag, String message)
throws LoggingException;

You can use values like null or UNKNOWN_PRIORITY in these version of the method as well.

The library can assign a value to an argument in various ways. For example, the event date can be assigned the current time. The host can be assigned from the event source or defined by a method as below:

Explicit host

public Syslogger setHost(String host);

The priority and tag are assigned with values defined using these methods:

Priority and tag definition

public Syslogger setPriority(int priority);
public Syslogger setTag(String tag);

Priority

Syslog priority is a value composed of two parts: facility and level. Facility and level are expressed as numbers that are defined in the syslog specification. Priority is also a number that is build by making certain numerical manipulations on its two parts. The org.scoja.common.PriorityUtils class, names the facilities and legitimate levels with constants, and has a statistic methods to construct a priority from a facility and level and to extract the facility and level from a priority.

  • The facilities are: KERN, USER, MAIL, DAEMON, AUTH, SYSLOG, LPR, NEWS, UUCP, CRON, AUTHPRIV, FTP, LOCAL0, LOCAL1, ..., LOCAL7...
  • The levels are: EMEG, ALERT, CRIT, ERR, WARNING, NOTICE, INFO and DEBUG.

A priority is built using buildPriority (facility,level). The facility and level are extracted from a priority with getFacility(priority) and getLevel(priority).

Example 2

In the following example, we use some of the elements that we've discussed so far. Assume that you want to send to two collectors: one is a Devo Relay listening on TCP port 514 the other is a standard syslog daemon also listening on UDP port 514.

Sending example

import org.scoja.common.PriorityUtils;
import org.scoja.client.Syslogger;
import org.scoja.client.UDPSyslogger;
import org.scoja.client.ReusingTCPSyslogger;
import org.scoja.client.SpreadingSyslogger;
 
public class Test {
  static final String HOST1 = "relay1";
  static final int PORT1 = 514;
  static final String HOST2 = "syslog";
  static final int PORT2 = 514;
 
  public static void main(String[] args) throws Exception {
    Syslogger logger = new SpreadingSyslogger(
      new ReusingTCPSyslogger(HOST1,PORT1),
      new UDPSyslogger(HOST2,PORT2));
    logger.setPriority(PriorityUtils.buildPriority(PriorityUtils.LOCAL0,PriorityUtils.INFO));
    logger.setTag("my.app.app1.activity");
    logger.log("Event body");
    logger.close();
  }
}

ReusingTCPSyslogger sends to the Devo Relay via TCP and UDPSyslogger sends to the syslog daemon via UDP. To avoid generating the event twice (once for each destination), we use SpreadingSyslogger to send the event to the two destinations. This way, it's easy to change or add an event destination.

Errors

The sending of an event can fail. The syslogger will produce a SysloggerException exception that must be captured and treated.A syslogger that failed to send an event probably has run into an inoperative state. You have to restart it by calling the reset() method, which will make it ready to continue sending events.If the error happens due to network problems; the errors will be repeated and you should opt for an alternative action.All this logic (trap errors, retry and choose an alternative action) is encapsulated in syslogger subclass called RetryingSyslogger. This is an example that retries 3 times and, in case of persistent failure, it shows the log through the standard output.

Error recovery

import java.util.Calendar;
import org.scoja.common.PriorityUtils;
import org.scoja.client.LoggingException;
import org.scoja.client.Syslogger;
import org.scoja.client.ReusingTCPSyslogger;
import org.scoja.client.RetryingSyslogger;
import org.scoja.client.LoggingErrorHandler;
 
public class Test {
  static final String HOST = "relay1";
  static final int PORT = 514;
  static final int RETRIES = 3;
 
  public static void main(String[] args) throws Exception {
    Syslogger base = new ReusingTCPSyslogger(HOST,PORT);
 
    LoggingErrorHandler onError = new LoggingErrorHandler() {
      public void log(Syslogger logger, LoggingException error,
          Calendar when, int priority, String host, String tag, String message)
      throws LoggingException {
        ilog(logger, error, when, priority, host, tag, message);
      }
      public void ilog(Syslogger logger, LoggingException error,
          Calendar when, int priority, String host, String tag, String message)
      throws LoggingException {
        System.err.println("Cannot log to " + logger + "; failed with " + error
          + " while sending " + when + " " + priority + " " + host + " " + tag + " " + message);
      }
    };
    base.setTag("my.app.app1.activity");
    Syslogger logger = new RetryingSyslogger(base, RETRIES, onError);
 
    logger.setPriority(PriorityUtils.buildPriority(PriorityUtils.LOCAL0,PriorityUtils.INFO));
    logger.log("Event body");
    logger.close();
  }
} 

The  RetryingSyslogger class is built with a delegate (base) Syslogger, a number of retries (RETRIES) and an action in case the error persists after the retries (onError). In case of error, the action can do several things with the event that caused the error: discard it, throw an exception, send it to another destination with another logger...

When the transport is UDP, there's usually no error or crashes because the problem usually involves packages in some point of the connection network; but with TCP, the link between the client and the collector is stronger: if the collector is heavily loaded or fallen, or there are network problems that makes it inaccessible, you'll see various anomalous behaviour in your application as exceptions or recurring, or, what is worse, as blocks. 

Java's TCP sockets don't have time-outs for the writing, you can only configure a time-out for the connection. The ReusingTCPSyslogger class makes that property visible with the method setConnectionTimeout(millis).

Non-blocking connections

With the Scoja library, it's possible to build a Syslogger with a more sophisticated behaviour on the transport layer. We'll focus on how to build a Syslogger that operates in a non-blocking manner, both during the connection negotiation as well as the sending; so the calls to log(...) methods will just last long enough to copy the data (either to operating systems or internal buffers). Before getting into the example, it's worth mentioning that we'll achieve the prevention of blocking by sacrificing other things. In this case, we'll have to assume that, if something misbehaves, some events will be lost.

Non-blocking send example

import org.scoja.common.PriorityUtils;
import org.scoja.util.ByteFall;
import org.scoja.util.MemoryByteFall;
import org.scoja.client.Syslogger;
import org.scoja.client.ReusingTransportSyslogger;
import org.scoja.trans.Transport;
import org.scoja.trans.tcp.TCPConf;
import org.scoja.trans.nbtcp.NBTCPTransport;
 
public class Test {
  static final String HOST = "relay1";
  static final int PORT = 514;
  static final int BUFFER_SIZE = 64*1024*1024;
 
  public static void main(String[] args) throws Exception {
    Transport<TCPConf> trans = new NBTCPTransport(HOST,PORT);
    ByteFall buffer = new MemoryByteFall(BUFFER_SIZE);
    Syslogger logger = new ReusingTransportSyslogger(trans, buffer);
 
    logger.setPriority(PriorityUtils.buildPriority(PriorityUtils.LOCAL0,PriorityUtils.INFO));
    logger.setTag("my.app.app1.activity");
    for (int i = 0; i < 1000; i++) {
      logger.log("Event body");
      Thread.sleep(1);
    }
    logger.close();
  }
}

The Syslogger construction is now done in 3 steps. A ReusingTransportSyslogger is a Syslogger that doesn't understand about transportation and expects to be given a Transport type object that will take care of that problem. It does know that the sending via a Transport may fail or jam, in which case, it'll save the events in a buffer waiting for the Transport to recover.

In this example, the Transport is a NBTCPTransport (transport via TCP and non-blocking). All of this transport operations are performed in a non-blocking manner, both the connection and the sending. This is not achieved with a separate thread, but putting the socket in a non-blocking mode. It has the advantage that it coexist well with any application server that doesn't introduce new threads, but it has the disadvantage that it can't progress by itself and needs to receive a request to advance in the connection and sending process. The connection is not fully established until it has processed a few events and are visible in the syslog collector. In practice, this procedure is not visible because there are events continuously.

The buffer where data is stored must be limited, if not, a connectivity problem may cause the client to overflow. The MemoryByteFall class implements a buffer in memory, that occupies as much the size as it has been defined and discards the other data when full.

It should be noted that although this solution may lose packages, it has a better behavior than a sending based on UDP. First, it can work in a very congested network without losses, where package retransmission are necessary. In addition, by selecting a suitable size for the buffer, it'll be possible to restart the collector without losing events.

Encrypted sending

Once you've come to the capabilities of Scoja library to configure the transport layer, there's a world of possibilities. Next, we'll explore the sending with SSL.

When working with Devo, there are multiple ways to send events: In-house Relays, public relays, relays in a cloud provider's safety zone, etc. These alternatives are simple intermediaries for Devo's central collectors, they're designed to facilitate the sending; but, sometimes, it can be interesting to have a program that sends directly to the central collectors.

All the logs must get to the central collector through a SSL channel, which forces the double authentication by certificates: the server presents its certificate to the client, but the client must show the certificate to the server. This ensures that the traffic is encrypted when transmitted through internet and that it's not possible to do impersonations in either of the 2 extremes.

We'll adapt the example to send directly to logtrust central collectors via SSL:

Example of sending via SSL

import java.io.File;
import org.scoja.common.PriorityUtils;
import org.scoja.util.ByteFall;
import org.scoja.util.MemoryByteFall;
import org.scoja.client.Syslogger;
import org.scoja.client.ReusingTransportSyslogger;
import org.scoja.trans.Transport;
import org.scoja.trans.tcp.TCPConf;
import org.scoja.trans.nbtcp.NBTCPTransport;
import org.scoja.trans.ssl.SSLConf;
import org.scoja.trans.ssl.SSLTransport;
import static org.scoja.trans.ssl.SSLClientAuthenticationMode.REQUIRE;
import static org.scoja.trans.ssl.SSLUtils.loadTrusts;
import static org.scoja.trans.ssl.SSLUtils.loadKeys;
 
public class TestSSL {
    static final String TRUSTJKS = "logtrust.jks";
    static final String MEJKS = "me.jks";
    static final String HOST = "eu.public.relay.logtrust.net";
    static final int PORT = 443;
    static final int BUFFER_SIZE = 64*1024*1024;
 
    public static void main(String[] args) throws Exception {
        char[] password = args[0].toCharArray();
         
        Transport<TCPConf> tcp = new NBTCPTransport(HOST,PORT);
        Transport<SSLConf> ssl = SSLTransport.tls(tcp);
        SSLConf sslConf = ssl.configuration();
        sslConf.setClientAuth(REQUIRE);
        sslConf.setTrustManagers(loadTrusts(new File(TRUSTJKS)));
        sslConf.setKeyManagers(loadKeys(new File(MEJKS), password));
     
        ByteFall buffer = new MemoryByteFall(BUFFER_SIZE);
        Syslogger logger = new ReusingTransportSyslogger(ssl, buffer);
 
        logger.setPriority(PriorityUtils.buildPriority(PriorityUtils.LOCAL0,PriorityUtils.INFO));
        logger.setTag("my.app.app1.activity");
        for (int i = 0; i < 1000; i++) {
            logger.log("Event body");
            Thread.sleep(1);
        }
        logger.close();
    }
}

As you can see, we've added a new transport build with SSLTransport.tls(tcp). It's a transport element that makes a TLS negotiation over the tcp; which at the same time is a non-blocking TCP transport. The result is a non-blocking SSL transport.

As mentioned above, the communication with Devo's central collectors has to be (necessarily) via SSL with client authentication. To do so, the client must have a private key to communicate with logtrust; you can generate and download that key on the Devo application. In this code, we assume this has been done and is saved in the file me.jks.

It's also desirable to authenticate our central servers (to avoid a man-in.the.middle attack). To do so, you must have the central collectors public certificate, which you can also download from our web application. We assume this has been done and is saved in the file logtrust.jks.

A SSL Transport default configuration doesn't know which type of authentication it has to negotiate, neither the certificates and private keys to work with. All this is indicated with the corresponding sslConf methods. Specifically, the collector certificate and our private key are given with setTrustManagers and setKeyManagers, that wait for an array from javax.net.ssl.TrustManager and javax.net.ssl.KeyManager, respectively. There are different ways to build these arrays. Since we're assuming that we have information on JKS files, it's best to use the loadTrusts and loadKeys methods of the org.scoja.trans.ssl.SSLUtils class.

In the description of the non-blocking sending we've explained that the negotiation of the TCP connection is also made in a non-blocking manner, which usually means that the first elements sent through the Syslogger doesn't arrive immediately and are delayed until the connection is established. For TCP it's usually a couple of events. When SSL Transport operates on a non-blocking layer, it performs its own negotiation and subsequent data sending also in a non-blocking manner. Since the SSL negotiation consumes way more resources than the TCP, with several exchanges that require computation, the initial block of events that can't be sent immediately is usually larger (between 5 and 10 events).

Download as PDF

Did you find what you were looking for?

If not, please let us know what you need. Your feedback will help us to improve.

PREVIOUS
JDK java.util.logging
NEXT
Sample code

Export

See what Devo can do for you. Request a demo!
Discover what's new (Release notes)
  • Services & Support
  • Devo.com
  • Contact
    • Contact Us
    • Request a Demo
    • Partner Inquiry
  • Log In
    • USA Devo
    • EU Devo
  • +1 888 6830910 (USA)
  • +34 900 838 880 (Spain)
Copyright © 2019 Legal Terms Privacy Policy Cookies Policy

Powered by Confluence and Scroll Viewport