- The Devo data analytics platform
- Getting started
- Domain administration
-
Sending data to Devo
-
The Devo In-House Relay
- Installing the Devo Relay
- Configuring the In-House Relay
- Relay migration
- Sending SSL/TLS encrypted events to the Devo relay
- Relay troubleshooting tips (v1.4.2)
- Event sources
- Other data collection methods
- Uploading log files
- Devo software
-
The Devo In-House Relay
-
Searching data
- Accessing data tables
-
Building a query
- Data types in Devo
- Build a query in the search window
- Build a query using LINQ
- Working with JSON objects in data tables
- Subqueries
-
Operations reference
-
Aggregation operations
- Average (avg)
- Count (count)
- First (first)
- First not null (nnfirst)
- HyperLogLog++ (hllpp)
- HyperLogLog++ Count Estimation (hllppcount)
- Last (last)
- Last not null (nnlast)
- Maximum (max)
- Median / 2nd quartile / Percentile 50 (median)
- Minimum (min)
- Non-null average (nnavg)
- Non-null standard deviation (biased) (nnstddev)
- Non-null standard deviation (unbiased) (nnustddev)
- Non-null variance (biased) (nnvar)
- Non-null variance (unbiased) (nnuvar)
- Percentile 10 (percentile10)
- Percentile 25 / 1st quartile (percentile25)
- Percentile 5 (percentile5)
- Percentile 75 / 3rd quartile (percentile75)
- Percentile 90 (percentile90)
- Percentile 95 (percentile95)
- Standard deviation (biased) (stddev)
- Standard deviation (unbiased) (ustddev)
- Sum (sum)
- Sum Square (sum2)
- Variance (biased) (var)
- Variance (unbiased) (uvar)
-
Arithmetic group
- Absolute value (abs)
- Addition, sum, plus / Concatenation (add, +)
- Ceiling (ceil)
- Cube root (cbrt)
- Division (div, \)
- Division remainder (rem, %)
- Floor (floor)
- Modulo (mod, %%)
- Multiplication, product (mul, *)
- Power (pow)
- Real division (rdiv, /)
- Rounding (round)
- Sign (signum)
- Square root (sqrt)
- Subtraction, minus / Additive inverse (sub, -)
-
Conversion group
- Duration (duration)
- Format date (formatdate)
- From base16, b16, hex (from16)
- From base64, b64 (from64)
- From UTF8 (fromutf8)
- From Z85, base85 (fromz85)
- Human size (humanSize)
- Make byte array (mkboxar)
- Parse date (parsedate)
- Regular expression, regexp (re)
- Template (template)
- Timestamp (timestamp)
- To base16, b16, hex (to16)
- To base64, b64, hex (to64)
- To BigInt (bigint)
- To boolean (bool)
- To Float (float)
- To image (image)
- To Int (int)
- To IPv4 (ip4)
- To IPv4 net (net4)
- To IPv6 (ip6)
- To IPv6 compatible (compatible)
- To IPv6 mapped (mapped)
- To IPv6 net (net6)
- To IPv6 translated (translated)
- To MAC address (mac)
- To string (str)
- To string (stringify)
- To UTF8 (toutf8)
- To Z85, base85 (toz85)
- Cryptography group
- Date group
- Flow group
- General group
-
Geolocation group
- Coordinates distance (distance)
- Geocoord (geocoord)
- Geographic coordinate system (coordsystem)
- Geohash (geohash)
- Geohash string (geohashstr)
- Geolocated Accuracy Radius with MaxMind GeoIP2 (mm2accuracyradius)
- Geolocated ASN (mmasn)
- Geolocated ASN with MaxMind GeoIP2 (mm2asn)
- Geolocated AS Organization Name with MaxMind GeoIP2 (mm2asorg)
- Geolocated AS owner (mmasowner)
- Geolocated City (mmcity)
- Geolocated City with MaxMind GeoIP2 (mm2city)
- Geolocated Connection Speed (mmspeed)
- Geolocated connection type with MaxMind GeoIP2 (mm2con)
- Geolocated Coordinates (mmcoordinates)
- Geolocated coordinates with MaxMind GeoIP2 (mm2coordinates)
- Geolocated Country (mmcountry)
- Geolocated Country with MaxMind GeoIP2 (mm2country)
- Geolocated ISP (mmisp)
- Geolocated ISP name with MaxMind GeoIP2 (mm2isp)
- Geolocated Latitude (mmlatitude)
- Geolocated Latitude with MaxMind GeoIP2 (mm2latitude)
- Geolocated Level 1 Subdivision with MaxMind GeoIP2 (mm2subdivision1)
- Geolocated Level 2 Subdivision with MaxMind GeoIP2 (mm2subdivision2)
- Geolocated Longitude (mmlongitude)
- Geolocated Longitude with MaxMind GeoIP2 (mm2longitude)
- Geolocated Organization (mmorg)
- Geolocated organization name with MaxMind GeoIP2 (mm2org)
- Geolocated Postal Code (mmpostalcode)
- Geolocated Postal Code with MaxMind GeoIP2 (mm2postalcode)
- Geolocated Region (mmregion)
- Geolocated Region Name (mmregionname)
- ISO-3166-1 Continent Alpha-2 Code (continentalpha2)
- ISO-3166-1 Continent Name (continentname)
- ISO-3166-1 Country Alpha-2 Code (countryalpha2)
- ISO-3166-1 Country Alpha-2 Continent (countrycontinent)
- ISO-3166-1 Country Alpha-3 Code (countryalpha3)
- ISO-3166-1 Country Latitude (countrylatitude)
- ISO-3166-1 Country Longitude (countrylongitude)
- ISO-3166-1 Country Name (countryname)
- Latitude (latitude)
- Latitude and longitude coordinates (latlon)
- Longitude (longitude)
- Parse geocoord format (parsegeo)
- Represent geocoord format (reprgeo)
- Round coordinates (gridlatlon)
- JSON group
- Logic group
-
Mathematical group
- Arc cosine (acos)
- Arc sine (asin)
- Arc tangent (atan)
- Bitwise AND (band, &)
- Bitwise left shift (lshift, <<)
- Bitwise NOT (bnot, ~)
- Bitwise OR (bor, |)
- Bitwise right shift (rshift, >>)
- Bitwise unsigned right shift (urshift, >>>)
- Bitwise XOR (bxor, ^)
- Cosine (cos)
- e (mathematical constant) (e)
- Exponential: base e (exp)
- Hyperbolic cosine (cosh)
- Hyperbolic sine (sinh)
- Hyperbolic tangent (tanh)
- Logarithm: base 2 (log2)
- Logarithm: base 10 (log10)
- Logarithm: natural / arbitrary base (log)
- Pi (mathematical constant) (pi)
- Sine (sin)
- Tangent (tan)
- Meta Analysis group
- Name group
-
Network group
- HTTP Status Description (httpstatusdescription)
- HTTP Status Type (httpstatustype)
- IP Protocol (ipprotocol)
- IP Reputation Score (reputationscore)
- IP Reputation Tags (reputation)
- IPv4 legal use (purpose)
- IPv6 host number (host)
- IPv6 routing number (routing)
- Is IPv4 (ipip4)
- Is Private IPv4 (isprivate)
- Is Public IPv4 (ispublic)
- Squid Black Lists Flags (sbl)
- Order group
-
Packet group
- Ethernet destination MAC address (etherdst)
- Ethernet payload (etherpayload)
- Ethernet source MAC address (ethersrc)
- Ethernet status (etherstatus)
- Ethernet tag (ethertag)
- EtherType (ethertype)
- Has Ethernet frame (hasether)
- Has IPv4 datagram (hasip4)
- Has TCP segment (hastcp)
- Has UDP datagram (hasudp)
- IPv4 destination address (ip4dst)
- IPv4 differentiated services (ip4ds)
- IPv4 explicit congestion notification (ip4ecn)
- IPv4 flags (ip4flags)
- IPv4 fragment offset (ip4fragment)
- IPv4 header checksum (ip4cs)
- IPv4 header length (ip4hl)
- IPv4 identification (ip4ident)
- IPv4 payload (ip4payload)
- IPv4 protocol (ip4proto)
- IPv4 source address (ip4src)
- IPv4 status (ip4status)
- IPv4 time to live (ip4ttl)
- IPv4 total length (ip4len)
- IPv4 type of service (ip4tos)
- TCP ACK (tcpack)
- TCP checksum (tcpcs)
- TCP destination port (tcpdst)
- TCP flags (tcpflags)
- TCP header length (tcphl)
- TCP payload (tcppayload)
- TCP sequence number (tcpseq)
- TCP source port (tcpsrc)
- TCP status (tcpstatus)
- TCP urgent pointer (tcpurg)
- TCP window size (tcpwin)
- UDP checksum (udpcs)
- UDP destination port (udpdst)
- UDP length (udplen)
- UDP payload (udppayload)
- UDP source port (udpsrc)
- UDP status (udpstatus)
- Statistical group
-
String group
- Contains (has, ->)
- Contains - case insensitive (weakhas)
- Contains tokens (toktains)
- Contains tokens - case insensitive (weaktoktains)
- Edit distance: Damerau (damerau)
- Edit distance: Hamming (hamming)
- Edit distance: Levenshtein (levenshtein)
- Edit distance: OSA (osa)
- Ends with (endswith)
- Format number (formatnumber)
- Hostname public suffix (publicsuffix)
- Hostname root domain (rootdomain)
- Hostname root prefix (rootprefix)
- Hostname root suffix (rootsuffix)
- Hostname subdomains (subdomain)
- Hostname top level domain (topleveldomain)
- Is empty (isempty)
- Is in (`in`, <-)
- Is in - case insensitive (weakin)
- Length (length)
- Locate (locate)
- Lower case (lower)
- Matches (matches, ~)
- Peek (peek)
- Replace all (replaceall)
- Replace first (replace)
- Shannon entropy (shannonentropy)
- Split (split)
- Split regexp (splitre)
- Starts with (startswith)
- Substitute (subs)
- Substitute all (subsall)
- Substring (substring)
- Trim both sides (trim)
- Trim the left side (ltrim)
- Trim the right side (rtrim)
- Upper case (upper)
-
Web group
- Absolute URI (absoluteuri)
- Opaque URI (opaqueuri)
- URI authority (uriauthority)
- URI fragment (urifragment)
- URI host (urihost)
- URI path (uripath)
- URI port (uriport)
- URI query (uriquery)
- URI scheme (urischeme)
- URI ssp (urissp)
- URI user (uriuser)
- URL decode (urldecode)
- User Agent Company (uacompany)
- User Agent Company URL (uacompanyurl)
- User Agent Device Icon (uadeviceicon)
- User Agent Device Information URL (uadeviceinfourl)
- User Agent Device Type (uadevicetype)
- User Agent Family (uafamily)
- User Agent Icon (uaicon)
- User Agent Information URL (uainfourl)
- User Agent is Robot (uaisrobot)
- User Agent Name (uaname)
- User Agent OS Company (uaoscompany)
- User Agent OS Company URL (uaoscompanyurl)
- User Agent OS Family (uaosfamily)
- User Agent OS Icon (uaosicon)
- User Agent OS Name (uaosname)
- User Agent OS URL (uaosurl)
- User Agent Type (uatype)
- User Agent URL (uaurl)
- User Agent Version (uaversion)
-
Aggregation operations
-
Working in the search window
-
Generate charts
- Affinity chord diagram
- Availability timeline
- Bipartite chord diagram
- Bubble chart
- Chart aggregation
- Custom date chart aggregation
- Flame graph
- Flat world map by coordinates
- Flat world map by country
- Google animated heat map
- Google area map
- Google heat map
- Graph diagram
- Histogram
- Pew Pew map
- Pie chart
- Pie layered chart
- Punch card
- Robust Random Cut Forest chart
- Sankey diagram
- Scatter plot
- Time heatmap
- Triple exponential chart
- Voronoi treemap
- Data enrichment
- Setting up a data table
- Advanced data operations
- Use case: eCommerce behavior analysis
-
Generate charts
- Managing your queries
- Best practices for data search
- Monitoring tables
-
Parsers and collectors
- About Devo tags
- Special Devo tags and data tables
-
List of Devo parsers
- Business & Consumer
- Cloud technologies
- Databases
- Host and Operating Systems
-
Network and application security
- auth.secureauth
- auth.securenvoy
- av.mcafee
- av.sophos
- box.iptables
- edr.cylance
- edr.fireeye.alerts
- edr.minervalabs.events
- endpoint.symantec
- firewall.checkpoint
- firewall.cisco firepower and vpn.cisco
- firewall.fortinet
- firewall.huawei
- firewall.juniper
- firewall.paloalto
- firewall.pfsense
- firewall.sonicwall
- firewall.sophos
- firewall.sophos.xgfirewall
- firewall.stonegate
- firewall.windows
- mail.proofpoint
- nac.aruba
- network.meraki
- network.versa
- proxy.bluecoat
- proxy.forcepoint
- proxy.squid
- uba.varonis
- vuln.beyondtrust
- vpn.pulsesecure.sa
- Network connectivity
- Web servers
- Technologies supported in CEF syslog format
- Collectors
- Activeboards
-
Dashboards
- Create a new dashboard
-
Working with dashboard widgets
- Availability timeline widget
- Chord diagram widget
- Circle world map widget
- Color key value widget
- Color world map widget
- Column chart widget
- Comparative chart widget
- Funnel widget
- Gauge meter widget
- Google heatmap widget
- Heat calendar widget
- Line chart widget
- Monitoring widget
- Pie chart widget
- Punch card widget
- Sectored pie chart widget
- Table widget
- Time heatmap widget
- Tree diagram widget
- Voronoi tree widget
- Configuring and sharing dashboards
- Alerts and notifications
- Panels
- Applications
- Tools
- Social Intelligence
- API reference
- Release notes
Searching data / Working in the search window / Data enrichment / Create a lookup table from a query
Create a lookup table from a query
Query lookup types
Users with the required permissions can use the content of a query to create a lookup directly from the search window. You can create two different types of lookup tables using query data: static query lookups and dynamic query lookups.
Static query lookup
These lookup tables are created using query data from a specific period of time. You cannot create this type of lookup if you enable real-time in your query –you must always choose a period of time using the time range selector in the query toolbar. These lookups work exactly the same as uploaded lookup tables, since both contain a fixed set of data that you can use to enrich a raw data table.
The lookup will include the last events in the specified time range containing all the unique values in the key column selected.
Dynamic query lookup
These lookup tables are also created with query data, but they differ from static query lookups in that they are constantly populated with new sets of data. By default, they are updated every 5 minutes since the creation date. If a new event arrives where the key column value has different row values, the old ones will be overwritten.
If your query groups data, the dynamic query lookup will be updated according to the grouping time indicated. For example, if your query groups data every 1 hour, the lookup will be populated with a new set of data every 1 hour. Note that if the grouping period is less than 5 minutes, the lookup will be updated every 5 minutes.
You can create a dynamic query lookup with the real-time option activated, but you can also define a specific time range. If you do this, the first data set until it is updated will be the data in the time frame indicated. For example, if you set Last week as the time range, the data in the last seven days will be used to populate the lookup, and then will be populated with new data following the rules explained above.
Check the following video to start learning about query lookups:
Create a query lookup
To define a lookup table using the content of a query, follow these steps:
Note that this feature is only enabled in certain domains by default. If you need to use it and is not enabled in your domain, contact the Devo support team.
- Go to Data Search and access the required table.
- Choose the required period of time using the time range selector and click Apply Interval.
- In the data table toolbar, click the gear icon and then select Table Operations → New Query Lookup.
The New Query Lookup window appears. Enter the required information.
Name Give a name to the lookup table. Data columns Drag to this field the data columns you want to add to the lookup table. The valid column types are:
bool
str
int (int4/int8)
float (float4/float8)
ip4
ip6
timestamp
Mode Choose either Static query or Dynamic query. Key column Select one of the columns added as key column. Learn more about this here.
Time range lookup Check this option to use a timestamp type column in your table to determine the value to be matched with the key column according to the date. Learn more about time range lookups in the section below. Repeated values in the key column
If the column selected as key upon the creation of a static query lookup contains repeated values, only the last value in the column will be kept, and the rest of the events with that value repeated will be discarded. In the case of a dynamic query lookup, repeated key values will be replaced by the most recent ones.
Click Save settings. You will receive a notification when the file is uploaded.
Query lookup tables are stored along with uploaded lookup tables, in Data Search → Lookup Management. The query lookup will be ready when the indicator in the Status column turns green. The Type column shows Upload, Static Query or Dynamic Query to indicate the lookup table type. Learn more in Manage and edit lookup tables.
Time range lookups
Both static and dynamic query lookups can be created as a time range lookup. When using this option, the same entry of your key column will be matched with different results depending on the corresponding time range between the dates specified in a timestamp column of your query. This way, you can match the same value in your query with different values in your lookup, which may come in handy in different situations.
To create a time range lookup, you must check the Time range lookup checkbox in the creation process of the query lookup, as explained above. Once you do this, the Time range column dropdown menu will appear, and you must choose the timestamp-type column you want to use among the ones added into the field on the left.
Choosing your timestamp column
If the type of the column is not timestamp-type, it will not appear in the Time range column dropdown. Remember that you can check the type of a column by clicking the icon that appears when hovering over a column header.
You can use the Timestamp (timestamp) operation to transform strings representing dates or epoch times in milliseconds (integers) into timestamp data type.
For example, imagine you have the following query, which shows the cities to which a user (Mike) has been calling in different time ranges. As you can see, Mike has been talking with colleagues in Madrid from 08:00 to 13:00; with users in New York from 13:00 to 17:00; and with users in Paris from 17:00.
After defining a time range lookup based on this data, we want to define a new column in a query that contains a user column. For every Mike value in our query, we want to include the corresponding city depending on the time range specified in our lookup. As you can see in the following capture, the new column shows the corresponding city according to the time in the eventdate column. For example, you can see the value New York in events sent at 16:00, since the time range lookup said that New York corresponds to the time range that goes from 13:00 to 17:00, as explained above.
To see the creation process of a time range lookup in action, check the following video: