- The Devo data analytics platform
- Getting started
- Domain administration
-
Sending data to Devo
-
The Devo In-House Relay
- Installing the Devo Relay
- Configuring the In-House Relay
- Relay migration
- Sending SSL/TLS encrypted events to the Devo relay
- Relay troubleshooting tips (v1.4.2)
-
Event sources
- Unix-like machines
- Windows
- MacOS X
- Cloud services
- Commercial products
- Custom apps
-
Universal Agent
- Deployment scenarios
- Pre-integrated query packs
- Data querying in Devo
-
Universal Agent Manager deployment
- Generic deployment guidelines
- Universal Agent Manager - CentOS 7 Deployment
- Universal Agent Manager - CentOS 8 Deployment
- Universal Agent Manager - Debian 9 Deployment
- Universal Agent Manager - Debian 10 Deployment
- Universal Agent Manager - RHEL 7 Deployment
- Universal Agent Manager - RHEL 8 Deployment
- Universal Agent Manager - Ubuntu 18 Deployment
- Universal Agent deployment
- Universal Agent Manager user manual
- Operational guidelines
- Performance considerations
- Other data collection methods
- Uploading log files
- Devo software
-
The Devo In-House Relay
-
Parsers and collectors
- About Devo tags
- Special Devo tags and data tables
-
List of Devo parsers
- Business & Consumer
- Cloud technologies
- Databases
- Host and Operating Systems
-
Network and application security
- auth.cisco
- auth.secureauth
- auth.securenvoy
- av.mcafee
- av.sophos
- box.iptables
- edr.carbonblack
- edr.cylance
- edr.fireeye.alerts
- edr.minervalabs.events
- edr.paloalto
- endpoint.symantec
- firewall.checkpoint
- firewall.cisco firepower and vpn.cisco
- firewall.fortinet
- firewall.huawei
- firewall.juniper
- firewall.paloalto
- firewall.pfsense
- firewall.sonicwall
- firewall.sophos
- firewall.sophos.xgfirewall
- firewall.stonegate
- firewall.windows
- ids.extrahop
- mail.proofpoint
- nac.aruba
- network.meraki
- network.versa
- network.vmware
- proxy.bluecoat
- proxy.forcepoint
- proxy.squid
- proxy.zscaler
- uba.varonis
- vuln.beyondtrust
- vpn.pulsesecure.sa
- vpn.zscaler
- Network connectivity
- Web servers
- Technologies supported in CEF syslog format
- Collectors
-
Searching data
- Accessing data tables
-
Building a query
- Data types in Devo
- Build a query in the search window
- Build a query using LINQ
- Working with JSON objects in data tables
- Subqueries
-
Operations reference
-
Aggregation operations
- Average (avg)
- Count (count)
- First (first)
- First not null (nnfirst)
- HyperLogLog++ (hllpp)
- HyperLogLog++ Count Estimation (hllppcount)
- Last (last)
- Last not null (nnlast)
- Maximum (max)
- Median / 2nd quartile / Percentile 50 (median)
- Minimum (min)
- Non-null average (nnavg)
- Non-null standard deviation (biased) (nnstddev)
- Non-null standard deviation (unbiased) (nnustddev)
- Non-null variance (biased) (nnvar)
- Non-null variance (unbiased) (nnuvar)
- Percentile 10 (percentile10)
- Percentile 25 / 1st quartile (percentile25)
- Percentile 5 (percentile5)
- Percentile 75 / 3rd quartile (percentile75)
- Percentile 90 (percentile90)
- Percentile 95 (percentile95)
- Standard deviation (biased) (stddev)
- Standard deviation (unbiased) (ustddev)
- Sum (sum)
- Sum Square (sum2)
- Variance (biased) (var)
- Variance (unbiased) (uvar)
-
Arithmetic group
- Absolute value (abs)
- Addition, sum, plus / Concatenation (add, +)
- Ceiling (ceil)
- Cube root (cbrt)
- Division (div, \)
- Division remainder (rem, %)
- Floor (floor)
- Modulo (mod, %%)
- Multiplication, product (mul, *)
- Power (pow)
- Real division (rdiv, /)
- Rounding (round)
- Sign (signum)
- Square root (sqrt)
- Subtraction, minus / Additive inverse (sub, -)
-
Conversion group
- Duration (duration)
- Format date (formatdate)
- From base16, b16, hex (from16)
- From base64, b64 (from64)
- From UTF8 (fromutf8)
- From Z85, base85 (fromz85)
- Human size (humanSize)
- Make byte array (mkboxar)
- Parse date (parsedate)
- Regular expression, regexp (re)
- Template (template)
- Timestamp (timestamp)
- To base16, b16, hex (to16)
- To base64, b64, hex (to64)
- To BigInt (bigint)
- To boolean (bool)
- To Float (float)
- To image (image)
- To Int (int)
- To IPv4 (ip4)
- To IPv4 net (net4)
- To IPv6 (ip6)
- To IPv6 compatible (compatible)
- To IPv6 mapped (mapped)
- To IPv6 net (net6)
- To IPv6 translated (translated)
- To MAC address (mac)
- To string (str)
- To string (stringify)
- To UTF8 (toutf8)
- To Z85, base85 (toz85)
- Cryptography group
- Date group
- Flow group
- General group
-
Geolocation group
- Coordinates distance (distance)
- Geocoord (geocoord)
- Geographic coordinate system (coordsystem)
- Geohash (geohash)
- Geohash string (geohashstr)
- Geolocated Accuracy Radius with MaxMind GeoIP2 (mm2accuracyradius)
- Geolocated ASN (mmasn)
- Geolocated ASN with MaxMind GeoIP2 (mm2asn)
- Geolocated AS Organization Name with MaxMind GeoIP2 (mm2asorg)
- Geolocated AS owner (mmasowner)
- Geolocated City (mmcity)
- Geolocated City with MaxMind GeoIP2 (mm2city)
- Geolocated Connection Speed (mmspeed)
- Geolocated connection type with MaxMind GeoIP2 (mm2con)
- Geolocated Coordinates (mmcoordinates)
- Geolocated coordinates with MaxMind GeoIP2 (mm2coordinates)
- Geolocated Country (mmcountry)
- Geolocated Country with MaxMind GeoIP2 (mm2country)
- Geolocated ISP (mmisp)
- Geolocated ISP name with MaxMind GeoIP2 (mm2isp)
- Geolocated Latitude (mmlatitude)
- Geolocated Latitude with MaxMind GeoIP2 (mm2latitude)
- Geolocated Level 1 Subdivision with MaxMind GeoIP2 (mm2subdivision1)
- Geolocated Level 2 Subdivision with MaxMind GeoIP2 (mm2subdivision2)
- Geolocated Longitude (mmlongitude)
- Geolocated Longitude with MaxMind GeoIP2 (mm2longitude)
- Geolocated Organization (mmorg)
- Geolocated organization name with MaxMind GeoIP2 (mm2org)
- Geolocated Postal Code (mmpostalcode)
- Geolocated Postal Code with MaxMind GeoIP2 (mm2postalcode)
- Geolocated Region (mmregion)
- Geolocated Region Name (mmregionname)
- ISO-3166-1 Continent Alpha-2 Code (continentalpha2)
- ISO-3166-1 Continent Name (continentname)
- ISO-3166-1 Country Alpha-2 Code (countryalpha2)
- ISO-3166-1 Country Alpha-2 Continent (countrycontinent)
- ISO-3166-1 Country Alpha-3 Code (countryalpha3)
- ISO-3166-1 Country Latitude (countrylatitude)
- ISO-3166-1 Country Longitude (countrylongitude)
- ISO-3166-1 Country Name (countryname)
- Latitude (latitude)
- Latitude and longitude coordinates (latlon)
- Longitude (longitude)
- Parse geocoord format (parsegeo)
- Represent geocoord format (reprgeo)
- Round coordinates (gridlatlon)
- JSON group
- Logic group
-
Mathematical group
- Arc cosine (acos)
- Arc sine (asin)
- Arc tangent (atan)
- Bitwise AND (band, &)
- Bitwise left shift (lshift, <<)
- Bitwise NOT (bnot, ~)
- Bitwise OR (bor, |)
- Bitwise right shift (rshift, >>)
- Bitwise unsigned right shift (urshift, >>>)
- Bitwise XOR (bxor, ^)
- Cosine (cos)
- e (mathematical constant) (e)
- Exponential: base e (exp)
- Hyperbolic cosine (cosh)
- Hyperbolic sine (sinh)
- Hyperbolic tangent (tanh)
- Logarithm: base 2 (log2)
- Logarithm: base 10 (log10)
- Logarithm: natural / arbitrary base (log)
- Pi (mathematical constant) (pi)
- Sine (sin)
- Tangent (tan)
- Meta Analysis group
- Name group
-
Network group
- HTTP Status Description (httpstatusdescription)
- HTTP Status Type (httpstatustype)
- IP Protocol (ipprotocol)
- IP Reputation Score (reputationscore)
- IP Reputation Tags (reputation)
- IPv4 legal use (purpose)
- IPv6 host number (host)
- IPv6 routing number (routing)
- Is IPv4 (ipip4)
- Is Private IPv4 (isprivate)
- Is Public IPv4 (ispublic)
- Squid Black Lists Flags (sbl)
- Order group
-
Packet group
- Ethernet destination MAC address (etherdst)
- Ethernet payload (etherpayload)
- Ethernet source MAC address (ethersrc)
- Ethernet status (etherstatus)
- Ethernet tag (ethertag)
- EtherType (ethertype)
- Has Ethernet frame (hasether)
- Has IPv4 datagram (hasip4)
- Has TCP segment (hastcp)
- Has UDP datagram (hasudp)
- IPv4 destination address (ip4dst)
- IPv4 differentiated services (ip4ds)
- IPv4 explicit congestion notification (ip4ecn)
- IPv4 flags (ip4flags)
- IPv4 fragment offset (ip4fragment)
- IPv4 header checksum (ip4cs)
- IPv4 header length (ip4hl)
- IPv4 identification (ip4ident)
- IPv4 payload (ip4payload)
- IPv4 protocol (ip4proto)
- IPv4 source address (ip4src)
- IPv4 status (ip4status)
- IPv4 time to live (ip4ttl)
- IPv4 total length (ip4len)
- IPv4 type of service (ip4tos)
- TCP ACK (tcpack)
- TCP checksum (tcpcs)
- TCP destination port (tcpdst)
- TCP flags (tcpflags)
- TCP header length (tcphl)
- TCP payload (tcppayload)
- TCP sequence number (tcpseq)
- TCP source port (tcpsrc)
- TCP status (tcpstatus)
- TCP urgent pointer (tcpurg)
- TCP window size (tcpwin)
- UDP checksum (udpcs)
- UDP destination port (udpdst)
- UDP length (udplen)
- UDP payload (udppayload)
- UDP source port (udpsrc)
- UDP status (udpstatus)
- Statistical group
-
String group
- Contains (has, ->)
- Contains - case insensitive (weakhas)
- Contains tokens (toktains)
- Contains tokens - case insensitive (weaktoktains)
- Edit distance: Damerau (damerau)
- Edit distance: Hamming (hamming)
- Edit distance: Levenshtein (levenshtein)
- Edit distance: OSA (osa)
- Ends with (endswith)
- Format number (formatnumber)
- Hostname public suffix (publicsuffix)
- Hostname root domain (rootdomain)
- Hostname root prefix (rootprefix)
- Hostname root suffix (rootsuffix)
- Hostname subdomains (subdomain)
- Hostname top level domain (topleveldomain)
- Is empty (isempty)
- Is in (`in`, <-)
- Is in - case insensitive (weakin)
- Length (length)
- Locate (locate)
- Lower case (lower)
- Matches (matches, ~)
- Peek (peek)
- Replace all (replaceall)
- Replace first (replace)
- Shannon entropy (shannonentropy)
- Split (split)
- Split regexp (splitre)
- Starts with (startswith)
- Substitute (subs)
- Substitute all (subsall)
- Substring (substring)
- Trim both sides (trim)
- Trim the left side (ltrim)
- Trim the right side (rtrim)
- Upper case (upper)
-
Web group
- Absolute URI (absoluteuri)
- Opaque URI (opaqueuri)
- URI authority (uriauthority)
- URI fragment (urifragment)
- URI host (urihost)
- URI path (uripath)
- URI port (uriport)
- URI query (uriquery)
- URI scheme (urischeme)
- URI ssp (urissp)
- URI user (uriuser)
- URL decode (urldecode)
- User Agent Company (uacompany)
- User Agent Company URL (uacompanyurl)
- User Agent Device Icon (uadeviceicon)
- User Agent Device Information URL (uadeviceinfourl)
- User Agent Device Type (uadevicetype)
- User Agent Family (uafamily)
- User Agent Icon (uaicon)
- User Agent Information URL (uainfourl)
- User Agent is Robot (uaisrobot)
- User Agent Name (uaname)
- User Agent OS Company (uaoscompany)
- User Agent OS Company URL (uaoscompanyurl)
- User Agent OS Family (uaosfamily)
- User Agent OS Icon (uaosicon)
- User Agent OS Name (uaosname)
- User Agent OS URL (uaosurl)
- User Agent Type (uatype)
- User Agent URL (uaurl)
- User Agent Version (uaversion)
-
Aggregation operations
-
Working in the search window
-
Generate charts
- Affinity chord diagram
- Availability timeline
- Bipartite chord diagram
- Bubble chart
- Chart aggregation
- Custom date chart aggregation
- Flame graph
- Flat world map by coordinates
- Flat world map by country
- Google animated heat map
- Google area map
- Google heat map
- Graph diagram
- Histogram
- Pew Pew map
- Pie chart
- Pie layered chart
- Punch card
- Robust Random Cut Forest chart
- Sankey diagram
- Scatter plot
- Time heatmap
- Triple exponential chart
- Voronoi treemap
- Data enrichment
- Setting up a data table
- Advanced data operations
- Use case: eCommerce behavior analysis
-
Generate charts
- Managing your queries
- Best practices for data search
- Monitoring tables
- Activeboards
-
Dashboards
-
Working with dashboard widgets
- Availability timeline widget
- Chord diagram widget
- Circle world map widget
- Color key value widget
- Color world map widget
- Column chart widget
- Comparative chart widget
- Funnel widget
- Gauge meter widget
- Google heatmap widget
- Heat calendar widget
- Line chart widget
- Monitoring widget
- Pie chart widget
- Punch card widget
- Sectored pie chart widget
- Table widget
- Time heatmap widget
- Tree diagram widget
- Voronoi tree widget
- Configuring and sharing dashboards
-
Working with dashboard widgets
- Alerts and notifications
- Panels
- Applications
- Tools
- Flow
- Social Intelligence
- API reference
- Release notes
Microsoft Azure collector
Service description
Microsoft Azure is an ever-expanding set of cloud computing services to help your organization meet its business challenges. Azure gives you the freedom to build, manage, and deploy applications on a massive, global network using your preferred tools and frameworks.
Data source description
You can use the Microsoft Azure collector to send the following types of data to your Devo domain. Once the gathered information arrives at Devo, it will be processed and included in different tables in the associated Devo domain.
Virtual Machine metrics
With the advantages of the Azure API, you can obtain basic metrics about your deployed Virtual Machines, gather them on our platform, and transform it as required.
Virtual Machine metric events are sent to the cloud.azure.vm.metrics_simple Devo table.
Event Hub Services
Thanks to the Azure Event Hub resource combined with the Diagnostic Settings option offered by Azure, you can easily centralize all the supported services in the Event Hub and send them to Devo. Although Event Hub is the service used to centralize data, it also generates data that can be sent to itself.
The collector will retrieve all the data received in the Event Hub. You must take this into account before configuring the required diagnostic settings so that not all data is sent to the same Event Hub.
Microsoft Azure services
Listed in the table below are some service names, details, and how the Devo platform treats the data.
Service name | Description | Devo data tables |
---|---|---|
Active Directory | Provides single sign-on and multi-factor authentication to help protect your users from 99.9 percent of cybersecurity attacks. | cloud.azure.eh.events |
Activity Logs | Provides insight into subscription-level events that have occurred in Azure. | cloud.azure.eh.events |
Automation Accounts | Delivers a cloud-based automation and configuration service that supports consistent management across your Azure and non-Azure environments. | cloud.azure.eh.events cloud.azure.eh.metrics |
Event Hubs | Big data streaming platform and event ingestion service. | cloud.azure.eh.events cloud.azure.eh.metrics |
Key Vault | Tool for securely storing and accessing secrets. | cloud.azure.eh.events cloud.azure.eh.metrics |
Load Balancer | Tool for evenly distributing load (incoming network traffic) across a group of backend resources or servers. | cloud.azure.eh.events cloud.azure.eh.metrics |
Network Security Groups | A network security group contains security rules that allow or deny inbound network traffic to, or outbound network traffic from, several types of Azure resources. | cloud.azure.eh.events |
Setup
The Microsoft Azure collector centralizes the data with an Event Hub using the Azure SDK. To use it, you need to configure the resources in the Azure Portal and set the right permissions to access the information.
Virtual Machine metrics
Getting credentials
To log in to the Azure subscription, the collector uses a Service Principal object. You need to get the subscription ID, Active Directory ID, Application ID (service principal identificator) and the secret (service principal "password"). To get them, follow these steps:
Log in to your Azure account and search for Azure Active Directory.
- Now, click App registrations in the left menu and click the app (or Service Principal) that you are going to use.
- In the Overview area, find the Application (client) ID and the Directory (tenant) ID.
Now click Certificates & Secrets on the menu and create a new secret.
Don't forget to save the secret, it will be only shown upon creation.
- Get the subscription ID searching for Subscriptions on the home page.
- Find the correct subscription and note down the subscription ID.
Setting up permissions
You need Administration permissions to follow these steps.
After creating the App registration (or Service Principal), go to the desired Resource Group (or subscription if you want to retrieve metrics from all the available virtual machines).
- Select Access control (IAM) in the left menu and click Add.
- Select at least the Reader role and choose the previously created App registration.
Confirm the changes.
Event Hub events
Getting credentials
- In your Azure account, search the Event Hubs service and click on it.
Create an Event Hub resource per region (repeat the steps below for each region):
Click Add.
Fill the mandatory fields keeping in mind that the Event Hub must be in the same region as the resources that you are going to monitor (and only need one per region).
The Throughput Units option refers to the ingress/egress limit in MB/s (each unit is 1 MB/s or 1000 events/second ingress, 2 MB/s, or 4096 events/second egress). You should adjust it according to the data volume (this can be modified later).
The previous steps create an EventHub namespace; now go to Event Hubs, search the created one and click on it.
Now click on the + Event Hub button and create a new resource. You only need to fill the Name and Partition Count fields (the Partition Count field will divide the data into different partitions to make it easier to read large volumes of data).
Write down the EventHub name to be used later in the configuration file.
Once the Event Hub is created in the namespace, click it and select Consumer Group in the left menu.
Here you will see the Event Hub consumer groups. This will be used by the collector (or other applications) for reading data from the Event Hub. Write down the Consumer group name that you will use later in the configuration file.
Now, in the Event Hub Namespace, click on Shared access policies, search the default policy named RootManageSharedAccessKey and click it.
- Copy and write down the primary (or secondary) connection string to be used later in the configuration file.
Setting up the Event Hubs
Now, search the Monitor service and click on it.
Click the Diagnostic Settings option in the left area.
A list of the deployed resources will be shown. Search for the resources that you want to monitor, select them, and click Add diagnostic setting.
- Type a name for the rule and check the required category details (logs will be sent to the cloud.azure.eh.events table, and metrics will be sent to the cloud.azure.eh.metrics table).
- Check Stream to an Event Hub, and select the corresponding Event hub namespace, Event hub name and Event hub policy name.
- Click Save to finish the process.
Run the collector
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.
Structure
The following directory structure will be required as part of the setup procedure (it can be created under any directory):
<any_directory>
└── devo-collectors/
└── azure/
├── certs/
│ ├── chain.crt
│ ├── <your_domain>.key
│ └── <your_domain>.crt
└── config/
└── config-azure.yaml
Devo credentials
In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <any directory>/devo-collectors/azure/certs
. Learn more about security credentials in Devo here.
Editing the config-azure.yaml file
In the config-azure.yaml file, replace the <app_id>
, <active_directory_id>
, <subscription_id>
and <secret>
values and enter the ones that you got in the previous steps. In the <short_unique_identifier>
placeholder, enter the value that you choose.
config-azure.yaml
globals:
debug: false
id: not_used
name: azure
persistence:
type: filesystem # File system persistence ON
config:
directory_name: state # Directory where the persistence will be saved in case of using filesystem
outputs:
devo_1: # Cloud Devo config EU (for US use us.elb.relay.logtrust.net)
type: devo_platform
config:
address: eu.elb.relay.logtrust.net
port: 443
type: SSL
chain: chain.crt
cert: <your_domain>.crt
key: <your_domain>.key
inputs:
azure:
id: <short_unique_identifier> # The value of this field will be used internally for having independent persistence areas
enabled: true
requests_per_second: 5 # Setup how many request API por second
autoconfig:
enable: true # Autoconfig always True for this collector
refresh_interval_in_seconds: 600 # Setting up the time for executing autoconfig interval. 60sec is recommended.
credentials:
app_id: "<app_id_value>" # Service principal identificator. Obtained in the Setup steps.
active_directory_id: "<active_directory_id_value>" # Active directory identificator. Obtained in the Setup steps.
subscription_id: "<subscription_id_value>" # Subscription identificator. Obtained in the Setup steps.
secret: "<secret_value>" # Service principal secret. Obtained in the Setup steps.
services: # Services available for this collector.
my_service_1: # Service name, has only aclarative purpose, can be any string.
request_period_in_seconds: 60 # Setting up time interval between API requests. 60sec is recommended.
types: # Kind of data that will be retrieved (can be "eh_services" or "vm_metrics", both are shown in this example)
- eh_services
queues: # List of eventHubs that will be readed. Needed because you can read more than one EventHub in the same Namespace
queue_a: # Queue name. As service name has only aclarative purpose, can be any string.
event_hub_name: "<event_hub_name_value>" # Name of the EventHub that will be readed, obtained in the Setup steps.
connection_str: Endpoint=sb://<domain_name>.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<shared_access_key_value> # Connection string of the EventHub that will be readed, obtained in the Setup steps.
consumer_group: "<consumer_group_value>" # Consumer group of the EventHub that will be used, obtained in the Setup steps.
tag: my.app.azure.sample # Optional. Custom tag if you want to send the event to other Devo table.
my_service_2:
request_period_in_seconds: 60
types:
- vm_metrics
The tag
field is optional and is only available in the eh_services services type.
If you need to use a custom tag for generated messages, it can be done using the property tag
inside any queue name, next to event_hub_name
or connection_str.
For example tag: my.app.azure.{service_name}
Download the Docker image
The collector should be deployed as a Docker container. Click here to download the Docker image of the collector as a .tgz file.
Use the following command to add the Docker image to the system:
gunzip -c collector-azure-docker-image-<version>.tgz | docker load
Once the Docker image is imported, it will show the real name of the Docker image (including version info).
The Docker image can be deployed on the following services:
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/azure/
docker run \
--name collector-azure \
--volume $PWD/certs:/devo-collector/certs \
--volume $PWD/config:/devo-collector/config \
--volume $PWD/state:/devo-collector/state \
--env CONFIG_FILE=config-azure.yaml \
--rm -it docker.devo.internal/collector/azure:<version>
Replace <version>
with the proper version.
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/azure/
directory.
docker-compose.yaml
version: '3'
services:
collector-azure:
build:
context: .
dockerfile: Dockerfile
image: docker.devo.internal/collector/azure:${IMAGE_VERSION:-latest}
container_name: collector-azure
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
- ./state:/devo-collector/state
environment:
- CONFIG_FILE=${CONFIG_FILE:-config-azure.yaml}
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/azure/
directory:
IMAGE_VERSION=<version> docker-compose up -d
Replace <version>
with the proper version.