• Services & Support
  • Devo.com
  • Contact
    • Contact Us
    • Request a Demo
    • Partner Inquiry
  • Log In
    • USA Devo
    • EU Devo
  • The Devo data analytics platform
    • How Devo indexes data
    • How Devo works
    • Key concepts
  • Getting started
    • Sign up and log in
    • Navigating the Devo app
    • User preferences
    • Devo video tutorials
      • Domain administration videos
      • Sending data to Devo videos
      • Searching data videos
      • Activeboards videos
      • Panels videos
      • Applications videos
  • Domain administration
    • Users and roles
      • Managing users
      • Monitoring user activity
      • Managing roles
        • Assign resources to a role
      • Role permissions
      • Role mapping
    • Security credentials
      • Access keys
      • X.509 certificates
      • Authentication tokens
    • Applications gallery
    • Domain preferences
    • User authentication
      • Password
      • SAML
        • Google as an identity provider
        • Okta as an identity provider
        • OneLogin as an identity provider
        • O365/Azure AD as an identity provider
      • OpenID
    • Data processes and feeds
      • Aggregation tasks
      • Injections
      • Permalinks
      • API & OData feeds
  • Sending data to Devo
    • The Devo In-House Relay
      • Installing the Devo Relay
        • Install the Relay on an Ubuntu box (v1.4.2)
        • Install the Relay on a Red Hat or CentOS box (v1.4.2)
        • Install with Docker
      • Configuring the In-House Relay
        • Relay rules
          • Defining a relay rule
          • The 4 predefined relay rules
          • 5 common relay rule scenarios
            • Scenario 1: Apply a fixed tag to all events
            • Scenario 2: Apply a Devo tag based on data found in the inbound event
            • Scenario 3: Filter out unwanted events
            • Scenario 4: Assign dynamic Devo tag using inbound source data
            • Scenario 5: Appending the inbound syslog tag to the outbound Devo tag
          • Using regex in relay rules
        • Customizing In-House Relay settings
        • Managing the relay on the command line
        • Setting up High-Availability with Keepalived (v1.4.2)
        • Relay buffers
      • Relay migration
      • Sending SSL/TLS encrypted events to the Devo relay
      • Relay troubleshooting tips (v1.4.2)
        • Relay troubleshooting tips (v1.4.0)
        • Relay troubleshooting tips (for versions prior to 1.4.0)
    • Event sources
      • Unix-like machines
        • Installing Devo packages for *nix
        • Third-party syslog tools configuration
          • rsyslog
            • Simple sending using rsyslog
            • Secure sending using rsyslog
            • Monitoring files using rsyslog
            • Obsolete legacy format
              • Simple sending using rsyslog (Obsolete legacy format)
              • Secure sending using rsyslog (Obsolete legacy format)
              • Monitoring files using rsyslog (Obsolete legacy format)
          • syslog-ng
            • Simple sending using syslog-ng
            • Secure sending using syslog-ng
            • Monitoring files using syslog-ng
          • syslog/syslogd
        • SELinux configuration conflicts
      • Windows
        • Devo Agent for Windows
        • Configuring WMI for Devo file monitoring
        • NXLog for Windows event collection
      • MacOS X
      • Cloud services
        • AWS S3 Buckets
      • Commercial products
      • Custom apps
        • Java apps
          • JDK java.util.logging
          • Scoja client library
          • Sample code
        • Node.js apps
        • Python apps
      • Universal Agent
        • Pre-integrated query packs
        • Data querying in Devo
        • Universal Agent Manager deployment
          • Generic deployment guidelines
          • Universal Agent Manager - CentOS 7 Deployment
          • Universal Agent Manager - CentOS 8 Deployment
          • Universal Agent Manager - Debian 9 Deployment
          • Universal Agent Manager - Debian 10 Deployment
          • Universal Agent Manager - RHEL 7 Deployment
          • Universal Agent Manager - RHEL 8 Deployment
          • Universal Agent Manager - Ubuntu 18 Deployment
        • Universal Agent deployment
        • Performance considerations
        • Deployment scenarios
        • Operational guidelines
    • Other data collection methods
      • HTTP endpoint
      • Logstash
      • Fluentd
      • NXLog
    • Uploading log files
    • Devo software
  • Parsers and collectors
    • About Devo tags
    • Special Devo tags and data tables
    • List of Devo parsers
      • Business & Consumer
      • Cloud technologies
        • cdn.akamai
        • cloud.aws.cloudtrail.events
          • Forwarding the events using Node.js
          • Forwarding the events using Python
        • cloud.aws.cloudwatch.events
        • cloud.office365.siem
      • Databases
        • db.mysql
      • Host and Operating Systems
        • box.unix
        • box.vmware
        • box.win
        • box.win_nxlog
        • box.win_snare
      • Network and application security
        • auth.secureauth
        • auth.securenvoy
        • av.mcafee
        • av.sophos
        • box.iptables
        • edr.cylance
        • edr.fireeye.alerts
        • edr.minervalabs.events
        • edr.paloalto
        • endpoint.symantec
        • firewall.checkpoint
        • firewall.cisco firepower and vpn.cisco
        • firewall.fortinet
        • firewall.huawei
        • firewall.juniper
        • firewall.paloalto
          • Sending Palo Alto events to Devo relay using SSL
        • firewall.pfsense
        • firewall.sonicwall
        • firewall.sophos
        • firewall.sophos.xgfirewall
        • firewall.stonegate
        • firewall.windows
        • ids.extrahop
        • mail.proofpoint
        • nac.aruba
        • network.meraki
        • network.versa
        • network.vmware
        • proxy.bluecoat
        • proxy.forcepoint
        • proxy.squid
        • uba.varonis
        • vuln.beyondtrust
        • vpn.pulsesecure.sa
      • Network connectivity
        • netstat.netflow
        • dns.bind
        • dns.windows
        • ftp.iis
      • Web servers
        • web.apache
        • web.apache.mod-security
        • web.iis
        • web.jboss
        • web.nginx
        • web.tomcat
      • Technologies supported in CEF syslog format
    • Collectors
      • AWS collector
      • Google Cloud Platform collector
      • G Suite collectors
        • G Suite Alerts collector
        • G Suite Reports collector
      • Microsoft Azure collector
      • Microsoft Graph collector
      • Office 365 collector
      • Okta collectors
        • Okta collector
        • Okta Advanced Server Access collector
      • OneLogin collector
      • Rapid7 InsightVM collector
      • Salesforce collector
      • Sophos Central collector
  • Searching data
    • Accessing data tables
      • Run a search using a finder
        • Use a custom finder
          • Create a custom finder
          • Assign a custom finder to a role
          • Edit a custom finder
        • Use the aliased finder
          • Add a query to your aliased finder
      • Run a global search
      • Run a LINQ free text query
      • Run a search with selected columns only
        • Selecting specific columns with the Finder
        • Selecting specific columns in LINQ
        • Selecting unrevealed columns
    • Building a query
      • Data types in Devo
      • Build a query in the search window
        • Filter data
        • Filter column data using the OR selector
        • Create columns
        • Group data
        • Aggregate data
      • Build a query using LINQ
        • LINQ query examples
      • Working with JSON objects in data tables
      • Subqueries
      • Operations reference
        • Aggregation operations
          • Average (avg)
          • Count (count)
          • First (first)
          • First not null (nnfirst)
          • HyperLogLog++ (hllpp)
          • HyperLogLog++ Count Estimation (hllppcount)
          • Last (last)
          • Last not null (nnlast)
          • Maximum (max)
          • Median / 2nd quartile / Percentile 50 (median)
          • Minimum (min)
          • Non-null average (nnavg)
          • Non-null standard deviation (biased) (nnstddev)
          • Non-null standard deviation (unbiased) (nnustddev)
          • Non-null variance (biased) (nnvar)
          • Non-null variance (unbiased) (nnuvar)
          • Percentile 10 (percentile10)
          • Percentile 25 / 1st quartile (percentile25)
          • Percentile 5 (percentile5)
          • Percentile 75 / 3rd quartile (percentile75)
          • Percentile 90 (percentile90)
          • Percentile 95 (percentile95)
          • Standard deviation (biased) (stddev)
          • Standard deviation (unbiased) (ustddev)
          • Sum (sum)
          • Sum Square (sum2)
          • Variance (biased) (var)
          • Variance (unbiased) (uvar)
        • Arithmetic group
          • Absolute value (abs)
          • Addition, sum, plus / Concatenation (add, +)
          • Ceiling (ceil)
          • Cube root (cbrt)
          • Division (div, \)
          • Division remainder (rem, %)
          • Floor (floor)
          • Modulo (mod, %%)
          • Multiplication, product (mul, *)
          • Power (pow)
          • Real division (rdiv, /)
          • Rounding (round)
          • Sign (signum)
          • Square root (sqrt)
          • Subtraction, minus / Additive inverse (sub, -)
        • Conversion group
          • Duration (duration)
          • Format date (formatdate)
          • From base16, b16, hex (from16)
          • From base64, b64 (from64)
          • From UTF8 (fromutf8)
          • From Z85, base85 (fromz85)
          • Human size (humanSize)
          • Make byte array (mkboxar)
          • Parse date (parsedate)
          • Regular expression, regexp (re)
          • Template (template)
          • Timestamp (timestamp)
          • To base16, b16, hex (to16)
          • To base64, b64, hex (to64)
          • To BigInt (bigint)
          • To boolean (bool)
          • To Float (float)
          • To image (image)
          • To Int (int)
          • To IPv4 (ip4)
          • To IPv4 net (net4)
          • To IPv6 (ip6)
          • To IPv6 compatible (compatible)
          • To IPv6 mapped (mapped)
          • To IPv6 net (net6)
          • To IPv6 translated (translated)
          • To MAC address (mac)
          • To string (str)
          • To string (stringify)
          • To UTF8 (toutf8)
          • To Z85, base85 (toz85)
        • Cryptography group
          • MD5 hash function (md5)
          • SHA1 hash function (sha1)
          • SHA256 hash function (sha256)
          • SHA512 hash function (sha512)
        • Date group
          • Day / Day of the month (day)
          • Day of the week (dayofweek)
          • Day of the year (dayofyear)
          • Epoch milliseconds (epoch)
          • Hour (hour)
          • Millisecond (millisecond)
          • Minute (minute)
          • Month (month)
          • Period (period)
          • Second (second)
          • Today (today)
          • Tomorrow (tomorrow)
          • Year (year)
          • Yesterday (yesterday)
        • Flow group
          • Conditional (ifthenelse)
          • Decode, switch (decode)
          • Null value locator (nvl)
        • General group
          • Is not null (isnotnull)
          • Is null (isnull)
        • Geolocation group
          • Coordinates distance (distance)
          • Geocoord (geocoord)
          • Geographic coordinate system (coordsystem)
          • Geohash (geohash)
          • Geohash string (geohashstr)
          • Geolocated Accuracy Radius with MaxMind GeoIP2 (mm2accuracyradius)
          • Geolocated ASN (mmasn)
          • Geolocated ASN with MaxMind GeoIP2 (mm2asn)
          • Geolocated AS Organization Name with MaxMind GeoIP2 (mm2asorg)
          • Geolocated AS owner (mmasowner)
          • Geolocated City (mmcity)
          • Geolocated City with MaxMind GeoIP2 (mm2city)
          • Geolocated Connection Speed (mmspeed)
          • Geolocated connection type with MaxMind GeoIP2 (mm2con)
          • Geolocated Coordinates (mmcoordinates)
          • Geolocated coordinates with MaxMind GeoIP2 (mm2coordinates)
          • Geolocated Country (mmcountry)
          • Geolocated Country with MaxMind GeoIP2 (mm2country)
          • Geolocated ISP (mmisp)
          • Geolocated ISP name with MaxMind GeoIP2 (mm2isp)
          • Geolocated Latitude (mmlatitude)
          • Geolocated Latitude with MaxMind GeoIP2 (mm2latitude)
          • Geolocated Level 1 Subdivision with MaxMind GeoIP2 (mm2subdivision1)
          • Geolocated Level 2 Subdivision with MaxMind GeoIP2 (mm2subdivision2)
          • Geolocated Longitude (mmlongitude)
          • Geolocated Longitude with MaxMind GeoIP2 (mm2longitude)
          • Geolocated Organization (mmorg)
          • Geolocated organization name with MaxMind GeoIP2 (mm2org)
          • Geolocated Postal Code (mmpostalcode)
          • Geolocated Postal Code with MaxMind GeoIP2 (mm2postalcode)
          • Geolocated Region (mmregion)
          • Geolocated Region Name (mmregionname)
          • ISO-3166-1 Continent Alpha-2 Code (continentalpha2)
          • ISO-3166-1 Continent Name (continentname)
          • ISO-3166-1 Country Alpha-2 Code (countryalpha2)
          • ISO-3166-1 Country Alpha-2 Continent (countrycontinent)
          • ISO-3166-1 Country Alpha-3 Code (countryalpha3)
          • ISO-3166-1 Country Latitude (countrylatitude)
          • ISO-3166-1 Country Longitude (countrylongitude)
          • ISO-3166-1 Country Name (countryname)
          • Latitude (latitude)
          • Latitude and longitude coordinates (latlon)
          • Longitude (longitude)
          • Parse geocoord format (parsegeo)
          • Represent geocoord format (reprgeo)
          • Round coordinates (gridlatlon)
        • JSON group
          • Jq evaluation (jqeval)
          • Jq filter compilation (jqcompile)
          • Json value type (label)
          • To json (jsonparse)
        • Logic group
          • And (and)
          • Not (not)
          • Or (or)
        • Mathematical group
          • Arc cosine (acos)
          • Arc sine (asin)
          • Arc tangent (atan)
          • Bitwise AND (band, &)
          • Bitwise left shift (lshift, <<)
          • Bitwise NOT (bnot, ~)
          • Bitwise OR (bor, |)
          • Bitwise right shift (rshift, >>)
          • Bitwise unsigned right shift (urshift, >>>)
          • Bitwise XOR (bxor, ^)
          • Cosine (cos)
          • e (mathematical constant) (e)
          • Exponential: base e (exp)
          • Hyperbolic cosine (cosh)
          • Hyperbolic sine (sinh)
          • Hyperbolic tangent (tanh)
          • Logarithm: base 2 (log2)
          • Logarithm: base 10 (log10)
          • Logarithm: natural / arbitrary base (log)
          • Pi (mathematical constant) (pi)
          • Sine (sin)
          • Tangent (tan)
        • Meta Analysis group
          • Pragma value (pragmavalue)
          • Table name (tablename)
        • Name group
          • Any name matches (anymatches)
          • Glob pattern on names (nameglob)
        • Network group
          • HTTP Status Description (httpstatusdescription)
          • HTTP Status Type (httpstatustype)
          • IP Protocol (ipprotocol)
          • IP Reputation Score (reputationscore)
          • IP Reputation Tags (reputation)
          • IPv4 legal use (purpose)
          • IPv6 host number (host)
          • IPv6 routing number (routing)
          • Is IPv4 (ipip4)
          • Is Private IPv4 (isprivate)
          • Is Public IPv4 (ispublic)
          • Squid Black Lists Flags (sbl)
        • Order group
          • Equal (eq, =)
          • Equal - case insensitive (eqic)
          • Greater or equal (ge, >=)
          • Greater than (gt, >)
          • Less or equal (le, <=)
          • Less than (lt, <)
          • Not equal (ne, /=)
        • Packet group
          • Ethernet destination MAC address (etherdst)
          • Ethernet payload (etherpayload)
          • Ethernet source MAC address (ethersrc)
          • Ethernet status (etherstatus)
          • Ethernet tag (ethertag)
          • EtherType (ethertype)
          • Has Ethernet frame (hasether)
          • Has IPv4 datagram (hasip4)
          • Has TCP segment (hastcp)
          • Has UDP datagram (hasudp)
          • IPv4 destination address (ip4dst)
          • IPv4 differentiated services (ip4ds)
          • IPv4 explicit congestion notification (ip4ecn)
          • IPv4 flags (ip4flags)
          • IPv4 fragment offset (ip4fragment)
          • IPv4 header checksum (ip4cs)
          • IPv4 header length (ip4hl)
          • IPv4 identification (ip4ident)
          • IPv4 payload (ip4payload)
          • IPv4 protocol (ip4proto)
          • IPv4 source address (ip4src)
          • IPv4 status (ip4status)
          • IPv4 time to live (ip4ttl)
          • IPv4 total length (ip4len)
          • IPv4 type of service (ip4tos)
          • TCP ACK (tcpack)
          • TCP checksum (tcpcs)
          • TCP destination port (tcpdst)
          • TCP flags (tcpflags)
          • TCP header length (tcphl)
          • TCP payload (tcppayload)
          • TCP sequence number (tcpseq)
          • TCP source port (tcpsrc)
          • TCP status (tcpstatus)
          • TCP urgent pointer (tcpurg)
          • TCP window size (tcpwin)
          • UDP checksum (udpcs)
          • UDP destination port (udpdst)
          • UDP length (udplen)
          • UDP payload (udppayload)
          • UDP source port (udpsrc)
          • UDP status (udpstatus)
        • Statistical group
          • Approximated estimation (estimation)
          • HyperLogLog++ pack (pack)
          • HyperLogLog++ unpack (unpackhllpp)
        • String group
          • Contains (has, ->)
          • Contains - case insensitive (weakhas)
          • Contains tokens (toktains)
          • Contains tokens - case insensitive (weaktoktains)
          • Edit distance: Damerau (damerau)
          • Edit distance: Hamming (hamming)
          • Edit distance: Levenshtein (levenshtein)
          • Edit distance: OSA (osa)
          • Ends with (endswith)
          • Format number (formatnumber)
          • Hostname public suffix (publicsuffix)
          • Hostname root domain (rootdomain)
          • Hostname root prefix (rootprefix)
          • Hostname root suffix (rootsuffix)
          • Hostname subdomains (subdomain)
          • Hostname top level domain (topleveldomain)
          • Is empty (isempty)
          • Is in (`in`, <-)
          • Is in - case insensitive (weakin)
          • Length (length)
          • Locate (locate)
          • Lower case (lower)
          • Matches (matches, ~)
          • Peek (peek)
          • Replace all (replaceall)
          • Replace first (replace)
          • Shannon entropy (shannonentropy)
          • Split (split)
          • Split regexp (splitre)
          • Starts with (startswith)
          • Substitute (subs)
          • Substitute all (subsall)
          • Substring (substring)
          • Trim both sides (trim)
          • Trim the left side (ltrim)
          • Trim the right side (rtrim)
          • Upper case (upper)
        • Web group
          • Absolute URI (absoluteuri)
          • Opaque URI (opaqueuri)
          • URI authority (uriauthority)
          • URI fragment (urifragment)
          • URI host (urihost)
          • URI path (uripath)
          • URI port (uriport)
          • URI query (uriquery)
          • URI scheme (urischeme)
          • URI ssp (urissp)
          • URI user (uriuser)
          • URL decode (urldecode)
          • User Agent Company (uacompany)
          • User Agent Company URL (uacompanyurl)
          • User Agent Device Icon (uadeviceicon)
          • User Agent Device Information URL (uadeviceinfourl)
          • User Agent Device Type (uadevicetype)
          • User Agent Family (uafamily)
          • User Agent Icon (uaicon)
          • User Agent Information URL (uainfourl)
          • User Agent is Robot (uaisrobot)
          • User Agent Name (uaname)
          • User Agent OS Company (uaoscompany)
          • User Agent OS Company URL (uaoscompanyurl)
          • User Agent OS Family (uaosfamily)
          • User Agent OS Icon (uaosicon)
          • User Agent OS Name (uaosname)
          • User Agent OS URL (uaosurl)
          • User Agent Type (uatype)
          • User Agent URL (uaurl)
          • User Agent Version (uaversion)
    • Working in the search window
      • Generate charts
        • Affinity chord diagram
        • Availability timeline
        • Bipartite chord diagram
        • Bubble chart
        • Chart aggregation
        • Custom date chart aggregation
        • Flame graph
        • Flat world map by coordinates
        • Flat world map by country
        • Google animated heat map
        • Google area map
        • Google heat map
        • Graph diagram
          • Creating a graph diagram
          • Working in the graph diagram
          • Monitor intranet traffic to dangerous websites
        • Histogram
        • Pew Pew map
        • Pie chart
        • Pie layered chart
        • Punch card
        • Robust Random Cut Forest chart
        • Sankey diagram
        • Scatter plot
        • Time heatmap
        • Triple exponential chart
        • Voronoi treemap
      • Data enrichment
        • Upload a lookup table
        • Create a lookup table from a query
        • Add lookup values to your query
        • Manage and edit lookup tables
        • Threat lookups
      • Setting up a data table
        • Modifying the column layout
          • Arrange and resize columns
          • Hide and show columns
          • Change the position of column headers
          • Sort data
          • Setting a default table layout
        • Add a description to a data table
        • Autoparser
          • Autoparse a JSON object
      • Advanced data operations
        • Graphical correlation
          • Cross-Search Graph Diagram
          • Cross-Search Table Join
          • Cross-Search Sankey Diagram
          • Cross-Search Line Chart
        • Custom and union tables
          • Create a custom table
          • Create a union table
          • Manage custom and union tables
        • Set up a data source
        • Inject data to a new table
        • Drill downs
        • Manipulate your data using CyberChef
        • Time series report
      • Use case: eCommerce behavior analysis
        • Step 1. Error analysis using status codes
          • Specific analysis for 404 codes
          • Custom alerts for 404 errors
        • Step 2. Operating system ranking
          • Get the usage share of operating systems
          • Visualize the usage share of operating systems
        • Step 3. Country distribution
          • Build a histogram displaying country distribution
          • Geolocate IP addresses
    • Managing your queries
      • Rename a query
      • Favorite queries
      • Query history
      • Check currently running queries
      • Add a description to your query
      • Block a query
      • Share a query
      • Download query data
      • Close a query
      • Load query data into Excel using the Devo Connector add-in
      • Query priority
    • Best practices for data search
    • Monitoring tables
      • Web application monitoring
      • Alerts monitoring
  • Activeboards
    • Creating an Activeboard
    • Working with Activeboard widgets
      • Area chart widgets
      • Column chart widgets
      • Donut chart widgets
      • Heatmap widgets
      • Line chart widgets
      • Pie chart widgets
      • Simple value widgets
      • Table widgets
      • Voronoi diagram widgets
    • Working with Activeboard inputs
    • LINQ syntax differences between Activeboards and the search window
    • Setting a time range in Activeboards
    • Working with Activeboards
      • Active Refresh
    • Sharing Activeboards
  • Dashboards
    • Create a new dashboard
    • Working with dashboard widgets
      • Availability timeline widget
      • Chord diagram widget
      • Circle world map widget
      • Color key value widget
      • Color world map widget
      • Column chart widget
      • Comparative chart widget
      • Funnel widget
      • Gauge meter widget
      • Google heatmap widget
      • Heat calendar widget
      • Line chart widget
        • Customize your Line chart
      • Monitoring widget
      • Pie chart widget
      • Punch card widget
      • Sectored pie chart widget
      • Table widget
      • Time heatmap widget
      • Tree diagram widget
      • Voronoi tree widget
    • Configuring and sharing dashboards
  • Alerts and notifications
    • Creating new alerts
      • Alert trigger methods
        • Each alert type
        • Several alert type
        • Low alert type
          • Inactivity alert
        • Rolling alert type
        • Deviation alert type
        • Gradient alert type
      • Create an alert based on triggered alerts
    • Configuring alerts
      • Manage defined alerts
      • Manage sending policies
      • Manage delivery methods
        • Email delivery methods
        • HTTP-JSON delivery methods
        • Service Desk delivery methods
        • Jira delivery methods
        • Pushover delivery methods
        • PagerDuty delivery methods
        • Slack delivery methods
      • Manage anti-flooding policies
      • Make an alert available for panels
      • Pre-installed alert reference
    • Managing triggered alerts
      • Add a comment to a triggered alert
      • Apply a filter for post-processing
  • Panels
    • Create and customize a panel
    • Adding an alert to a panel
    • Adding a query to a panel
    • Using panels
  • Applications
    • Devo Security Operations
      • Overview Dashboard
      • Triage
        • Triaging alerts
        • Triaging investigations
      • Investigations
      • Threat Hunting
      • Use cases
        • Phishing attack
        • Command & Control
        • Alerting system status
    • Devo Stats
      • Working in the Devo Stats application
      • Application tabs and widgets
        • User tab
        • Volume tab
        • Query tab
          • User Query Info
          • CPU Query Info
          • CPU Query Info Multidomain
        • Status tab
    • Security Insights
      • Installing Security Insights
        • Security Insights lookups
      • Configuring Security Insights
      • Navigating Security Insights
        • Overview tab
        • Threats tab
        • Network tab
        • DNS tab
        • Firewall tab
        • Proxy tab
        • Access tab
        • Web tab
        • IDS tab
    • Service operations
      • Basic concepts
      • Installation
      • Global models
      • Technologies configuration
      • Models configuration
      • Service overview
      • Incidents viewer
      • Monitors
      • User experience management
      • Use case for service operations
        • Initial analysis
        • Model configuration
        • Running the model
        • Incidents
    • Systems Monitoring
      • Basic monitoring
      • Advanced monitoring
  • Tools
    • Data Explorer
    • Query App
    • Time Series Analytics
  • Flow
    • Creating a Flow
    • Working with your Flow
      • Basic operations
      • Working with units
      • Working with links
      • Events window
      • Modify your Flow layout
    • Unit types
      • Control
        • Batch Detector
        • Delayer
        • Gate Delayer
      • Core
        • Filter
        • Join
        • Map
        • Reducer
        • Switch
      • Data
        • Lookup
        • Memory
      • DB
        • Neo4j Source
        • Neo4j Sink
        • OrientDB Sink
      • Devo
        • Devo Full Query
        • Devo Managed Query
        • Devo Sink
        • Devo Source
      • Event
        • Detacher
        • Generator
        • Repeater
        • Tick
      • HTTP
        • HTTP call
      • Log
        • Syslog Sender
      • MQ
        • Kafka Sender
        • Rabbit Ack
        • Rabbit Receiver
        • Rabbit Sender
      • Text
        • CSV Parser
        • JQ
        • JSON Parser
        • Regex
        • Text Formatter
        • XML Parser
        • XPath
      • Sink
        • Email Sink
        • Slack Sink
      • Window
        • Count Window Collector
        • Time Window Collector
    • Flow use cases
      • Alert with external web service
      • Simple Each alert
      • Simple ETL
  • Social Intelligence
  • API reference
    • REST API
      • Authorizing REST API requests
      • Running queries with the REST API
        • Forward response to Amazon S3
        • Forward response to email
        • Forward response to HDFS
        • Forward response to Kafka
        • Forward response to SNMP
        • Send requests with Postman
      • Job requests
    • Provisioning API
      • Authorizing Provisioning API requests
      • Common operations
        • User operations
        • Domain operations
        • Certificates
        • Role management
        • Domain resources management
      • Reseller operations
        • Price plans
      • Role specification and examples
    • Alerting API
      • Working with alert definitions
    • Using and managing OData feeds
      • Connecting with Excel
      • Connecting with Tableau
      • Connecting with Power BI
  • Release notes
    • 6.0.0
    • 6.0.1
    • 6.1.0
    • 6.1.1
    • 6.1.2
    • 6.1.3
    • 6.1.4
    • 6.2.0
    • 6.3.0
    • 6.3.1
    • 6.3.2
    • 6.3.3-1
    • 6.4.0
    • 6.4.1
    • 6.4.3
    • 6.5.0
    • 6.5.1
    • 6.5.2
    • 6.6.0
    • 6.6.1
    • 6.6.2
    • 6.7.0
    • 6.7.1
    • 6.7.2
    • 6.7.3
PREVIOUS
Sending SSL/TLS encrypted events to the Devo relay
NEXT
Relay troubleshooting tips (v1.4.0)

Sending data to Devo / The Devo In-House Relay / Relay troubleshooting tips (v1.4.2)

Download as PDF

Relay troubleshooting tips (v1.4.2)

These tips are based on version 1.4.2 of the Devo Relay. 

The Devo In-house Relay software is based on a Syslog engine called Scoja written in Java. Also, there are some processes written in Python that perform other tasks.

Recommendation

This software is prepared to be installed in any Linux system, but Devo recommends using Ubuntu.

Once running, the Scoja process listens for TCP/UDP connections on ports and, depending on the port, it modifies the Syslog events and forwards them to the Devo central collector. The main role of the relay is to tag events based on a defined rule, then forward the tagged events to Devo. 

The relay comes with a few ports that are preconfigured with rules for managing specific types of events. For example, port 13000 is preconfigured to forward the events received directly to Devo without any other processing. This port should be used for events that are properly tagged in the data source. You can also create custom rules to tag and/or filter the events sent to the associated port on the relay. Each rule is stored in one of these two folders:

  • /opt/devo/scoja-server/conf/rules/ → Rules managed from the Devo web application.
  • /opt/devo/scoja-server/conf/unrules/ → Rules not managed from the Devo web application. These rules are directly managed from the relay machine itself.

About the relay logs files

First, a word about the logs generated by the Devo Relay that can be used to investigate and identify problems with the relay.

The main logs files are:

  • /var/log/scoja.log → Main activity log of the Scoja engine. For example, if there is an error starting the process, it will be logged in this file.
  • /var/log/lt-relay.log → Contains events related to rules and changes to the relay configuration.

In addition, daily log files are generated and stored in:

/var/logt/local/<year>/<month>/<day>/<relayName>/syslog/scoja/

  • main.log → main process log file
  • source.log → logs related to the sources configured
  • thread.log → threads opened for each source
  • stats.log → statistics about received and sent events
  • target.log  → logs related to the sending process

These five daily logs are compressed at night by a cron task.

Troubleshooting checklist

Make sure only one Scoja process (Java) is running 

# ps auxwww|grep java|grep scoja|grep -v grep
root     20312  0.9 14.1 5656324 2260448 ?     Sl   Oct05  56:23 java -Xms2G -Xmx2G -XX:+UseConcMarkSweepGC -classpath /opt/devo/scoja-server/lib/scoja-server-1.4.0.jar:/opt/devo/scoja-server/lib/scoja-client-1.4.0.jar:/opt/devo/scoja-server/lib/scoja-beep-1.4.0.jar:/opt/devo/scoja-server/lib/scoja-rpc-1.4.0.jar:/opt/devo/scoja-server/lib/scoja-compression-1.4.0.jar:/opt/devo/scoja-server/lib/scoja-rpc-xc-1.4.0.jar:/opt/devo/scoja-server/lib/scoja-cc-1.4.0.jar:/opt/devo/scoja-server/lib/jython-standalone-2.7.2b3.jar:/opt/devo/scoja-server/lib/log4j-1.2.17.jar org.scoja.server.Scoja -r 5s -G /opt/devo/scoja-server/conf/all-me.conf -j /opt/devo/scoja-server/conf/all-var.conf

If there is more than one process running, you should kill all of them using the command kill -9 <PID>, then start the relay again:

sudo /etc/init.d/devo-scoja-relay start

If the Java process is still not appearing after restarting the relay, check the scoja.log file for errors.

Check the scoja.log file for errors

Use cat to display the contents of the scoja.log:

scoja.log

# cat /var/log/scoja.log
Oct  5 13:12:09 relay-awsus-testing syslog.scoja.configuration: (Re)Loading configuration file /opt/devo/scoja-server/conf/all-me.conf
Oct  5 13:12:09 relay-awsus-testing syslog.scoja.thread: Starting thread 1 for Queue "internal queue" with 1 queued events, 1 max threads, 1 threads started, 0 threads awaiting
Oct  5 13:12:09 relay-awsus-testing syslog.scoja.configuration: (Re)Loading configuration file /opt/devo/scoja-server/conf/parameters.conf
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.configuration: Configuration file /opt/devo/scoja-server/conf/parameters.conf successfully parsed
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.configuration: Configuration file /opt/devo/scoja-server/conf/parameters.conf successfully (re)loaded
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.configuration: (Re)Loading configuration file /opt/devo/scoja-server/conf/me.conf
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.configuration: Configuration file /opt/devo/scoja-server/conf/me.conf successfully parsed
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.configuration: Configuration file /opt/devo/scoja-server/conf/me.conf successfully (re)loaded
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.configuration: Configuration file /opt/devo/scoja-server/conf/all-me.conf successfully (re)loaded
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.configuration: (Re)Loading configuration file /opt/devo/scoja-server/conf/all-var.conf
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.configuration: Configuration file /opt/devo/scoja-server/conf/all-var.conf successfully parsed
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.configuration: Configuration file /opt/devo/scoja-server/conf/all-var.conf successfully (re)loaded
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.source.measurer: Starting Measurer source
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.thread: Starting thread 1 for Measurer source
...
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.thread: Starting thread 1 for UDP source listening at 0.0.0.0:13000
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.thread: Starting thread 2 for UDP source listening at 0.0.0.0:13000
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.source.net.tcp: Starting Selecting TCP source listening at 0.0.0.0:13000
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.source.net.udp: Opening server socket for UDP source listening at 0.0.0.0:13000
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.thread: Starting thread 1 for Selecting TCP source listening at 0.0.0.0:13000
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.thread: Starting thread 2 for Selecting TCP source listening at 0.0.0.0:13000
...
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.main: Reloader started to run every 5.0 seconds
Oct  5 13:12:11 relay-awsus-testing syslog.scoja.main: Scoja started

If the log file contains the error /opt/logtrust/scoja/scoja.sh: 60: java: not found, this means that it is unable to find the Java interpreter. In this case, consider creating a symbolic link or editing the JAVA_HOME environment variable.

Confirm that the default ports are listening

  • Ports 13000-13002 are listening on both TCP and UDP.
  • Port 12999 is listening on UDP and is prepared to receive external Netflow traffic (Netflow version < 9). 
  • Port 5141 is for internal Netflow traffic obtained using fprobe.
  • Port 5140 is listening on TCP and UDP to handle internal relay logs. 
# netstat -atunp | grep java | grep -v ESTABLISHED
tcp6       0      0 :::13000                :::*                    LISTEN      887/java
tcp6       0      0 :::13001                :::*                    LISTEN      887/java
tcp6       0      0 :::13002                :::*                    LISTEN      887/java
tcp6       0      0 127.0.0.1:5140          :::*                    LISTEN      887/java
udp6       0      0 :::12999                :::*                                887/java
udp6       0      0 :::13000                :::*                                887/java
udp6       0      0 :::13001                :::*                                887/java
udp6       0      0 :::13002                :::*                                887/java
udp6       0      0 127.0.0.1:5140          :::*                                887/java
udp6       0      0 127.0.0.1:5141          :::*                                887/java

Ensure that the client certificate is installed

The client certificate file, client.jks, should be installed in the relay's /opt/devo/scoja-server/conf/keys/ folder. 

# ls -al /opt/devo/scoja-server/conf/keys
total 16
drwx------ 2 root root 4096 Jan 28 10:24 .
drwxr-xr-x 5 root root 4096 Jan 28 10:24 ..
-rw-r--r-- 1 root root 8048 Jan 28 10:24 client.jks

If the certificate download fails, you can force the download again:

  1. Log into the Devo web application.
  2. Go to Administration → Relays and select the name of the relay in the list. The Rules screen appears. 
  3. Select the Force Generate New Certificate checkbox, then click the Apply Configuration button.

Ensure that the relay configuration is updating correctly

A cron task runs every minute to check if any changes to the relay configuration have been made in Devo. The result of this check appears in the lt-relay.log file. 

# cat /var/log/lt-relay.log:
2016-01-28 10:16:01.374445 [ERROR] Property 'api.key' not found in conf file
2016-01-28 10:17:01.510618 [ERROR] Property 'api.key' not found in conf file
2016-01-28 10:18:01.651220 [ERROR] Property 'api.key' not found in conf file
2016-01-28 10:19:01.791669 [ERROR] Property 'api.key' not found in conf file
2016-01-28 10:20:01.932002 [ERROR] Property 'api.key' not found in conf file
2016-01-28 10:21:02.073736 [ERROR] Property 'api.key' not found in conf file
2016-01-28 10:22:01.208316 [ERROR] Property 'api.key' not found in conf file
2016-01-28 10:23:00.488298 [SUCCESS] Relay activation is still pending, waiting for user to activate the Relay
2016-01-28 10:23:01.158783 [SUCCESS] Relay activation is still pending, waiting for user to activate the Relay
2016-01-28 10:24:01.331976 [SUCCESS] Relay is active, deploying new configuration set
2016-01-28 10:25:01.794021 [SUCCESS] No changes in the Relay Configuration
2016-01-28 10:26:01.975251 [SUCCESS] No changes in the Relay Configuration
...
  • The "Property 'api.key' not found in conf file" message indicates that the devo-relay-configure process has not been executed yet and there is no API key defined in the relay.
  • The "Relay activation is still pending" message indicates that the relay is registered but has not yet been activated in the Devo web application.
  • The "Relay is active" message confirms that the relay has been activated and new configuration has been downloaded.

Once activated, you will be able to define the relay rules using the web application. The relay will begin checking the configuration for updates every 1 minute.

  • The "No changes in Relay Configuration" message occurs every minute to confirm that no changes were detected in the relay configuration.

During the first configuration, the certificate is also downloaded. If the certificate download fails, you must force the download again.

Send test events to confirm relay is receiving and forwarding correctly

We can use netcat to send 100 events to the relay port 13000' to confirm that it is receiving and forwarding events to Devo. Enter this command on the relay machine:

# for i in `seq 1 100`; do (echo "<14>Jan  1 00:00:00 xxx test.keep.free: test event $i"|nc -w0 localhost 13000); done

To confirm that the events were processed correctly, open Devo, go to Data Search, and locate the test.keep.free table. This table should contain 100 events. 

If there are no events logged in the test.keep.free table, check the unknown.unknown table. If there was a problem applying the tags, events will be logged in this table.

Check if the firewall is disabled 

CentOS 7

# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)
[vagrant@localhost ~]$ systemctl disable firewalld
 
 
# systemctl disable firewalld

Check the relay's internet proxy configuration

If the Devo relay uses a proxy to access the Internet, you can review the configuration for possible errors.

The internet proxy is configured in /opt/devo/scoja-server/conf/local.py. In the sample configuration below, the proxy IP address is 12.12.12.12 and the port is 8080.

...
# LogTrust collectors
nextHops = [
#hop(host,port[,proxy,proxyPort])
hop(host="eu.elb.relay.logtrust.net", port=443, proxy="12.12.12.12", proxyPort=8080),
]
...

The hop host should be the hostname for the Devo region you are using and the port must always be 443. These are the hostnames by region.

  • Europe: eu.elb.relay.logtrust.net
  • USA: us.elb.relay.logtrust.net
  • Spain: es.elb.relay.logtrust.net

Some common problems and solutions

Problem 1

Scoja (java process) Out of Memory

Solution: In case the issue is related to the memory buffer there are two possible solutions: increase allocated memory for scoja or decrease memory buffer size (it can be done from the web).

In order to increase allocated memory, check Managing the relay on the command line (v1.4.2). Once updated,  restart the relay.

Labels

  • latest
Download as PDF

Did you find what you were looking for?

If not, please let us know what you need. Your feedback will help us to improve.

PREVIOUS
Sending SSL/TLS encrypted events to the Devo relay
NEXT
Relay troubleshooting tips (v1.4.0)

Export

See what Devo can do for you. Request a demo!
Discover what's new (Release notes)
  • Services & Support
  • Devo.com
  • Contact
    • Contact Us
    • Request a Demo
    • Partner Inquiry
  • Log In
    • USA Devo
    • EU Devo
  • +1 888 6830910 (USA)
  • +34 900 838 880 (Spain)
Copyright © 2019 Legal Terms Privacy Policy Cookies Policy

Powered by Confluence and Scroll Viewport