- The Devo data analytics platform
- Getting started
- Domain administration
-
Sending data to Devo
-
The Devo In-House Relay
- Installing the Devo Relay
- Configuring the In-House Relay
- Relay migration
- Sending SSL/TLS encrypted events to the Devo relay
- Relay troubleshooting tips (v1.4.2)
-
Event sources
- Unix-like machines
- Windows
- MacOS X
- Cloud services
- Commercial products
- Custom apps
-
Universal Agent
- Deployment scenarios
- Pre-integrated query packs
- Data querying in Devo
-
Universal Agent Manager deployment
- Generic deployment guidelines
- Universal Agent Manager - CentOS 7 Deployment
- Universal Agent Manager - CentOS 8 Deployment
- Universal Agent Manager - Debian 9 Deployment
- Universal Agent Manager - Debian 10 Deployment
- Universal Agent Manager - RHEL 7 Deployment
- Universal Agent Manager - RHEL 8 Deployment
- Universal Agent Manager - Ubuntu 18 Deployment
- Universal Agent deployment
- Universal Agent Manager user manual
- Operational guidelines
- Performance considerations
- Other data collection methods
- Uploading log files
- Devo software
-
The Devo In-House Relay
-
Parsers and collectors
- About Devo tags
- Special Devo tags and data tables
-
List of Devo parsers
- Business & Consumer
- Cloud technologies
- Databases
- Host and Operating Systems
-
Network and application security
- auth.cisco
- auth.secureauth
- auth.securenvoy
- av.mcafee
- av.sophos
- box.iptables
- edr.carbonblack
- edr.cylance
- edr.fireeye.alerts
- edr.minervalabs.events
- edr.paloalto
- endpoint.symantec
- firewall.checkpoint
- firewall.cisco firepower and vpn.cisco
- firewall.fortinet
- firewall.huawei
- firewall.juniper
- firewall.paloalto
- firewall.pfsense
- firewall.sonicwall
- firewall.sophos
- firewall.sophos.xgfirewall
- firewall.stonegate
- firewall.windows
- ids.extrahop
- mail.proofpoint
- nac.aruba
- network.meraki
- network.versa
- network.vmware
- proxy.bluecoat
- proxy.forcepoint
- proxy.squid
- proxy.zscaler
- uba.varonis
- vuln.beyondtrust
- vpn.pulsesecure.sa
- vpn.zscaler
- Network connectivity
- Web servers
- Technologies supported in CEF syslog format
- Collectors
-
Searching data
- Accessing data tables
-
Building a query
- Data types in Devo
- Build a query in the search window
- Build a query using LINQ
- Working with JSON objects in data tables
- Subqueries
-
Operations reference
-
Aggregation operations
- Average (avg)
- Count (count)
- First (first)
- First not null (nnfirst)
- HyperLogLog++ (hllpp)
- HyperLogLog++ Count Estimation (hllppcount)
- Last (last)
- Last not null (nnlast)
- Maximum (max)
- Median / 2nd quartile / Percentile 50 (median)
- Minimum (min)
- Non-null average (nnavg)
- Non-null standard deviation (biased) (nnstddev)
- Non-null standard deviation (unbiased) (nnustddev)
- Non-null variance (biased) (nnvar)
- Non-null variance (unbiased) (nnuvar)
- Percentile 10 (percentile10)
- Percentile 25 / 1st quartile (percentile25)
- Percentile 5 (percentile5)
- Percentile 75 / 3rd quartile (percentile75)
- Percentile 90 (percentile90)
- Percentile 95 (percentile95)
- Standard deviation (biased) (stddev)
- Standard deviation (unbiased) (ustddev)
- Sum (sum)
- Sum Square (sum2)
- Variance (biased) (var)
- Variance (unbiased) (uvar)
-
Arithmetic group
- Absolute value (abs)
- Addition, sum, plus / Concatenation (add, +)
- Ceiling (ceil)
- Cube root (cbrt)
- Division (div, \)
- Division remainder (rem, %)
- Floor (floor)
- Modulo (mod, %%)
- Multiplication, product (mul, *)
- Power (pow)
- Real division (rdiv, /)
- Rounding (round)
- Sign (signum)
- Square root (sqrt)
- Subtraction, minus / Additive inverse (sub, -)
-
Conversion group
- Duration (duration)
- Format date (formatdate)
- From base16, b16, hex (from16)
- From base64, b64 (from64)
- From UTF8 (fromutf8)
- From Z85, base85 (fromz85)
- Human size (humanSize)
- Make byte array (mkboxar)
- Parse date (parsedate)
- Regular expression, regexp (re)
- Template (template)
- Timestamp (timestamp)
- To base16, b16, hex (to16)
- To base64, b64, hex (to64)
- To BigInt (bigint)
- To boolean (bool)
- To Float (float)
- To image (image)
- To Int (int)
- To IPv4 (ip4)
- To IPv4 net (net4)
- To IPv6 (ip6)
- To IPv6 compatible (compatible)
- To IPv6 mapped (mapped)
- To IPv6 net (net6)
- To IPv6 translated (translated)
- To MAC address (mac)
- To string (str)
- To string (stringify)
- To UTF8 (toutf8)
- To Z85, base85 (toz85)
- Cryptography group
- Date group
- Flow group
- General group
-
Geolocation group
- Coordinates distance (distance)
- Geocoord (geocoord)
- Geographic coordinate system (coordsystem)
- Geohash (geohash)
- Geohash string (geohashstr)
- Geolocated Accuracy Radius with MaxMind GeoIP2 (mm2accuracyradius)
- Geolocated ASN (mmasn)
- Geolocated ASN with MaxMind GeoIP2 (mm2asn)
- Geolocated AS Organization Name with MaxMind GeoIP2 (mm2asorg)
- Geolocated AS owner (mmasowner)
- Geolocated City (mmcity)
- Geolocated City with MaxMind GeoIP2 (mm2city)
- Geolocated Connection Speed (mmspeed)
- Geolocated connection type with MaxMind GeoIP2 (mm2con)
- Geolocated Coordinates (mmcoordinates)
- Geolocated coordinates with MaxMind GeoIP2 (mm2coordinates)
- Geolocated Country (mmcountry)
- Geolocated Country with MaxMind GeoIP2 (mm2country)
- Geolocated ISP (mmisp)
- Geolocated ISP name with MaxMind GeoIP2 (mm2isp)
- Geolocated Latitude (mmlatitude)
- Geolocated Latitude with MaxMind GeoIP2 (mm2latitude)
- Geolocated Level 1 Subdivision with MaxMind GeoIP2 (mm2subdivision1)
- Geolocated Level 2 Subdivision with MaxMind GeoIP2 (mm2subdivision2)
- Geolocated Longitude (mmlongitude)
- Geolocated Longitude with MaxMind GeoIP2 (mm2longitude)
- Geolocated Organization (mmorg)
- Geolocated organization name with MaxMind GeoIP2 (mm2org)
- Geolocated Postal Code (mmpostalcode)
- Geolocated Postal Code with MaxMind GeoIP2 (mm2postalcode)
- Geolocated Region (mmregion)
- Geolocated Region Name (mmregionname)
- ISO-3166-1 Continent Alpha-2 Code (continentalpha2)
- ISO-3166-1 Continent Name (continentname)
- ISO-3166-1 Country Alpha-2 Code (countryalpha2)
- ISO-3166-1 Country Alpha-2 Continent (countrycontinent)
- ISO-3166-1 Country Alpha-3 Code (countryalpha3)
- ISO-3166-1 Country Latitude (countrylatitude)
- ISO-3166-1 Country Longitude (countrylongitude)
- ISO-3166-1 Country Name (countryname)
- Latitude (latitude)
- Latitude and longitude coordinates (latlon)
- Longitude (longitude)
- Parse geocoord format (parsegeo)
- Represent geocoord format (reprgeo)
- Round coordinates (gridlatlon)
- JSON group
- Logic group
-
Mathematical group
- Arc cosine (acos)
- Arc sine (asin)
- Arc tangent (atan)
- Bitwise AND (band, &)
- Bitwise left shift (lshift, <<)
- Bitwise NOT (bnot, ~)
- Bitwise OR (bor, |)
- Bitwise right shift (rshift, >>)
- Bitwise unsigned right shift (urshift, >>>)
- Bitwise XOR (bxor, ^)
- Cosine (cos)
- e (mathematical constant) (e)
- Exponential: base e (exp)
- Hyperbolic cosine (cosh)
- Hyperbolic sine (sinh)
- Hyperbolic tangent (tanh)
- Logarithm: base 2 (log2)
- Logarithm: base 10 (log10)
- Logarithm: natural / arbitrary base (log)
- Pi (mathematical constant) (pi)
- Sine (sin)
- Tangent (tan)
- Meta Analysis group
- Name group
-
Network group
- HTTP Status Description (httpstatusdescription)
- HTTP Status Type (httpstatustype)
- IP Protocol (ipprotocol)
- IP Reputation Score (reputationscore)
- IP Reputation Tags (reputation)
- IPv4 legal use (purpose)
- IPv6 host number (host)
- IPv6 routing number (routing)
- Is IPv4 (ipip4)
- Is Private IPv4 (isprivate)
- Is Public IPv4 (ispublic)
- Squid Black Lists Flags (sbl)
- Order group
-
Packet group
- Ethernet destination MAC address (etherdst)
- Ethernet payload (etherpayload)
- Ethernet source MAC address (ethersrc)
- Ethernet status (etherstatus)
- Ethernet tag (ethertag)
- EtherType (ethertype)
- Has Ethernet frame (hasether)
- Has IPv4 datagram (hasip4)
- Has TCP segment (hastcp)
- Has UDP datagram (hasudp)
- IPv4 destination address (ip4dst)
- IPv4 differentiated services (ip4ds)
- IPv4 explicit congestion notification (ip4ecn)
- IPv4 flags (ip4flags)
- IPv4 fragment offset (ip4fragment)
- IPv4 header checksum (ip4cs)
- IPv4 header length (ip4hl)
- IPv4 identification (ip4ident)
- IPv4 payload (ip4payload)
- IPv4 protocol (ip4proto)
- IPv4 source address (ip4src)
- IPv4 status (ip4status)
- IPv4 time to live (ip4ttl)
- IPv4 total length (ip4len)
- IPv4 type of service (ip4tos)
- TCP ACK (tcpack)
- TCP checksum (tcpcs)
- TCP destination port (tcpdst)
- TCP flags (tcpflags)
- TCP header length (tcphl)
- TCP payload (tcppayload)
- TCP sequence number (tcpseq)
- TCP source port (tcpsrc)
- TCP status (tcpstatus)
- TCP urgent pointer (tcpurg)
- TCP window size (tcpwin)
- UDP checksum (udpcs)
- UDP destination port (udpdst)
- UDP length (udplen)
- UDP payload (udppayload)
- UDP source port (udpsrc)
- UDP status (udpstatus)
- Statistical group
-
String group
- Contains (has, ->)
- Contains - case insensitive (weakhas)
- Contains tokens (toktains)
- Contains tokens - case insensitive (weaktoktains)
- Edit distance: Damerau (damerau)
- Edit distance: Hamming (hamming)
- Edit distance: Levenshtein (levenshtein)
- Edit distance: OSA (osa)
- Ends with (endswith)
- Format number (formatnumber)
- Hostname public suffix (publicsuffix)
- Hostname root domain (rootdomain)
- Hostname root prefix (rootprefix)
- Hostname root suffix (rootsuffix)
- Hostname subdomains (subdomain)
- Hostname top level domain (topleveldomain)
- Is empty (isempty)
- Is in (`in`, <-)
- Is in - case insensitive (weakin)
- Length (length)
- Locate (locate)
- Lower case (lower)
- Matches (matches, ~)
- Peek (peek)
- Replace all (replaceall)
- Replace first (replace)
- Shannon entropy (shannonentropy)
- Split (split)
- Split regexp (splitre)
- Starts with (startswith)
- Substitute (subs)
- Substitute all (subsall)
- Substring (substring)
- Trim both sides (trim)
- Trim the left side (ltrim)
- Trim the right side (rtrim)
- Upper case (upper)
-
Web group
- Absolute URI (absoluteuri)
- Opaque URI (opaqueuri)
- URI authority (uriauthority)
- URI fragment (urifragment)
- URI host (urihost)
- URI path (uripath)
- URI port (uriport)
- URI query (uriquery)
- URI scheme (urischeme)
- URI ssp (urissp)
- URI user (uriuser)
- URL decode (urldecode)
- User Agent Company (uacompany)
- User Agent Company URL (uacompanyurl)
- User Agent Device Icon (uadeviceicon)
- User Agent Device Information URL (uadeviceinfourl)
- User Agent Device Type (uadevicetype)
- User Agent Family (uafamily)
- User Agent Icon (uaicon)
- User Agent Information URL (uainfourl)
- User Agent is Robot (uaisrobot)
- User Agent Name (uaname)
- User Agent OS Company (uaoscompany)
- User Agent OS Company URL (uaoscompanyurl)
- User Agent OS Family (uaosfamily)
- User Agent OS Icon (uaosicon)
- User Agent OS Name (uaosname)
- User Agent OS URL (uaosurl)
- User Agent Type (uatype)
- User Agent URL (uaurl)
- User Agent Version (uaversion)
-
Aggregation operations
-
Working in the search window
-
Generate charts
- Affinity chord diagram
- Availability timeline
- Bipartite chord diagram
- Bubble chart
- Chart aggregation
- Custom date chart aggregation
- Flame graph
- Flat world map by coordinates
- Flat world map by country
- Google animated heat map
- Google area map
- Google heat map
- Graph diagram
- Histogram
- Pew Pew map
- Pie chart
- Pie layered chart
- Punch card
- Robust Random Cut Forest chart
- Sankey diagram
- Scatter plot
- Time heatmap
- Triple exponential chart
- Voronoi treemap
- Data enrichment
- Setting up a data table
- Advanced data operations
- Use case: eCommerce behavior analysis
-
Generate charts
- Managing your queries
- Best practices for data search
- Monitoring tables
- Activeboards
-
Dashboards
-
Working with dashboard widgets
- Availability timeline widget
- Chord diagram widget
- Circle world map widget
- Color key value widget
- Color world map widget
- Column chart widget
- Comparative chart widget
- Funnel widget
- Gauge meter widget
- Google heatmap widget
- Heat calendar widget
- Line chart widget
- Monitoring widget
- Pie chart widget
- Punch card widget
- Sectored pie chart widget
- Table widget
- Time heatmap widget
- Tree diagram widget
- Voronoi tree widget
- Configuring and sharing dashboards
-
Working with dashboard widgets
- Alerts and notifications
- Panels
- Applications
- Tools
- Flow
- Social Intelligence
- API reference
- Release notes
Logstash
Logstash is an open source tool for collecting, parsing and storing logs for future use. It ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite repository (in this case, Devo).
The procedures in the article assume a general working knowledge of this tool. To resolve questions about using Logstash, please visit the Logstash Reference Guide on the Elastic website.
In this article you will learn about:
About Logstash configuration files
For every data source that sends events to Logstash, there must be a corresponding pipeline configuration (.conf) file in the /etc/logstash/conf.d directory.
In order to forward a source's events from Logstash onward to a Devo Relay or to Devo directly, you will have to edit its corresponding pipeline configuration file. Specifically, by creating a syslog-type output section in the configuration file. This requires the logstash-output-syslog plugin. Make sure that this Logstash plugin is installed before you use these instructions.
Sending from Logstash to a Devo relay
To forward events to a Devo Relay, Logstash must be installed on the same machine as the relay.
To set up the forwarding, simply create a syslog plugin in the output section of the configuration file. Here, we not only specify the routing information but we also apply the Devo tag to the events. Here's an excerpt of a sample configuration file showing the output section.
input {...}
output {
syslog {
facility => "local7"
severity => "informational"
host => "localhost"
port => 13000 ### forwarding port 13000
sourcehost => "syslogHostname" ### syslog message hostname
appname => "av.mcafee.epo.events" ### Devo tag
protocol => "tcp"
}
}
You can get details about each of the parameters contained in this example in the Logstash Reference Guide, but note that:
- The port is 13000. This is because the events will be delivered to the relay already tagged and no further processing is required. More about default relay rules here.
- The appname parameter is where we specify the Devo tag to apply to these events.
Once you've edited the .conf file, reactivate the configuration with:
./bin/logstash -f <filename>.conf
Sending from Logstash to the Devo Cloud
In this case, we will be sending data over the internet so it is necessary to establish a secure channel using the Devo domain's SSL certificates. Download the certificate files from Administration → Relays and save them to a directory on the machine where Logstash is installed.
The configuration of the syslog plugin directs the events to the Devo Cloud (for your region, in this case Europe), applies the Devo tag, and references the SSL certificates.
output {
syslog {
facility => "local7"
severity => "informational"
host => "collector-eu.devo.io"
port => "443"
appname => "my.app.logstash.test" #SPECIFY THE DEVO TAG HERE
protocol => "ssl-tcp"
ssl_cert => "domain.crt"
ssl_key => "domain.key"
ssl_cacert => "chain.crt"
}
}
Again, you can read more about each of the parameters contained in this example in the Logstash Reference Guide, but note that:
- The host specifies the address of the Devo Cloud for the region you are using. It should be one of:
- USA: collector-us.devo.io
- Europe: collector-eu.devo.io
- Spain: collector-es.devo.io
- The port is 443 because this is the inbound port used for sending to the Devo Cloud.
- The appname parameter is where we specify the Devo tag to apply to these events.
Once you've edited the .conf file, reactivate the configuration with:
./bin/logstash -f <filename>.conf
Some configuration examples
Below we provide some sample configuration files for some cases where Logstash can be used to forward events from different kinds of data sources.
Forwarding from a database with JDBC
This configuration will send an event to Devo each time a new record appears in the users table of the specified database.
In this example, the input section uses the jdbc plugin to collect input:
- from a database specified by the jdbc_connection_string parameter
- with a frequency defined by the schedule parameter
- using the SQL statement set in the statement parameter
Read more about the jdbc input parameters here.
The output section forwards all new records as events to the Devo Cloud.
input {
jdbc {
jdbc_driver_library => "/Users/Bob/logstash/ojdbc6.jar"
jdbc_driver_class => "Java::oracle.jdbc.OracleDriver"
jdbc_connection_string => "jdbc:oracle:thin:@192.168.1.33:1521:test"
jdbc_user => "system"
jdbc_password => "password"
schedule => "* * * * *"
tracking_column => id
use_column_value => true
statement => "select * from users where id > :sql_last_value"
}
}
output {
syslog {
facility => "local7"
host => "ecollector-eu.devo.io"
port => "443"
severity => "informational"
appname => "my.app.oracle.test"
protocol => "ssl-tcp"
ssl_cert => "/Users/Bob/logstash/ca/domain.crt"
ssl_key => "/Users/Bob/logstash/ca/domain.key"
ssl_cacert => "/Users/Bob/logstash/ca/chain.crt"
codec => line {
format => "%{name},%{id}"
}
}
}
Forwarding keyboard input
This configuration will send an event to Devo each time a user types something using the keyboard and then presses Enter.
In this example, the input section uses the stdin plugin to collect input from the keyboard. The output section sends this input to the Devo Cloud using SSL-TCP. Security is enabled through the use of authentication certificates.
input {
stdin { }
}
output {
syslog {
facility => "local7"
severity => "informational"
host => "collector-eu.devo.io"
port => 443
protocol => "ssl-tcp"
sourcehost => "macbook_121"
appname => "my.app.test.logstash"
ssl_cacert => "chain.crt"
ssl_cert => "domain.crt"
ssl_key => "domain.key"
}
}
Forwarding from a file
This configuration will send events to Devo each time a file is updated with new information.
In this example, the input section uses the file plugin to collect input from a file. The output section sends this input to the Devo Cloud using SSL-TCP. Security is enabled through the use of authentication certificates.
input {
file {
path => "/Users/Ramon/logstash/logs/*"
start_position => "beginning"
sincedb_path => "/Users/Ramon/logstash/dbfile"
}
}
output {
syslog {
facility => "local7"
severity => "informational"
host => "collector-eu.devo.io"
port => "443"
appname => "my.app.ramon.logstash"
protocol => "ssl-tcp"
ssl_cert => "/Users/Ramon/logstash/ca/domain.crt"
ssl_key => "/Users/Ramon/logstash/ca/domain.key"
ssl_cacert => "/Users/Ramon/logstash/ca/chain.crt"
}
}
Forwarding from an Apache Kafka topic
This configuration will send events to Devo that are read from an Apache Kafka topic.
In this example, the input section uses the kafka plugin to collect input from a Kafka topic. The output section sends this input to the Devo Cloud using SSL-TCP. Security is enabled through the use of authentication certificates. It also sends events to standard output.
input {
kafka {
group_id => "test-kafka"
topics => ["devotest"]
bootstrap_servers => "localhost:9092"
}
}
output {
syslog {
facility => "local7"
severity => "informational"
host => "collector-us.devo.io"
port => "443"
appname => "my.app.kafka.topic"
protocol => "ssl-tcp"
ssl_cert => "/home/devo/domain.crt"
ssl_key => "/home/devo/domain.key"
ssl_cacert => "/home/devo/chain.crt"
}
stdout{codec => rubydebug }
}