- The Devo data analytics platform
- Getting started
- Domain administration
-
Sending data to Devo
-
The Devo In-House Relay
- Installing the Devo Relay
- Configuring the In-House Relay
- Relay migration
- Sending SSL/TLS encrypted events to the Devo relay
- Relay troubleshooting tips (v1.4.2)
- Event sources
- Other data collection methods
- Uploading log files
- Devo software
-
The Devo In-House Relay
-
Searching data
- Accessing data tables
-
Building a query
- Data types in Devo
- Build a query in the search window
- Build a query using LINQ
- Working with JSON objects in data tables
- Subqueries
-
Operations reference
-
Aggregation operations
- Average (avg)
- Count (count)
- First (first)
- First not null (nnfirst)
- HyperLogLog++ (hllpp)
- HyperLogLog++ Count Estimation (hllppcount)
- Last (last)
- Last not null (nnlast)
- Maximum (max)
- Median / 2nd quartile / Percentile 50 (median)
- Minimum (min)
- Non-null average (nnavg)
- Non-null standard deviation (biased) (nnstddev)
- Non-null standard deviation (unbiased) (nnustddev)
- Non-null variance (biased) (nnvar)
- Non-null variance (unbiased) (nnuvar)
- Percentile 10 (percentile10)
- Percentile 25 / 1st quartile (percentile25)
- Percentile 5 (percentile5)
- Percentile 75 / 3rd quartile (percentile75)
- Percentile 90 (percentile90)
- Percentile 95 (percentile95)
- Standard deviation (biased) (stddev)
- Standard deviation (unbiased) (ustddev)
- Sum (sum)
- Sum Square (sum2)
- Variance (biased) (var)
- Variance (unbiased) (uvar)
-
Arithmetic group
- Absolute value (abs)
- Addition, sum, plus / Concatenation (add, +)
- Ceiling (ceil)
- Cube root (cbrt)
- Division (div, \)
- Division remainder (rem, %)
- Floor (floor)
- Modulo (mod, %%)
- Multiplication, product (mul, *)
- Power (pow)
- Real division (rdiv, /)
- Rounding (round)
- Sign (signum)
- Square root (sqrt)
- Subtraction, minus / Additive inverse (sub, -)
-
Conversion group
- Duration (duration)
- Format date (formatdate)
- From base16, b16, hex (from16)
- From base64, b64 (from64)
- From UTF8 (fromutf8)
- From Z85, base85 (fromz85)
- Human size (humanSize)
- Make byte array (mkboxar)
- Parse date (parsedate)
- Regular expression, regexp (re)
- Template (template)
- Timestamp (timestamp)
- To base16, b16, hex (to16)
- To base64, b64, hex (to64)
- To BigInt (bigint)
- To boolean (bool)
- To Float (float)
- To image (image)
- To Int (int)
- To IPv4 (ip4)
- To IPv4 net (net4)
- To IPv6 (ip6)
- To IPv6 compatible (compatible)
- To IPv6 mapped (mapped)
- To IPv6 net (net6)
- To IPv6 translated (translated)
- To MAC address (mac)
- To string (str)
- To string (stringify)
- To UTF8 (toutf8)
- To Z85, base85 (toz85)
- Cryptography group
- Date group
- Flow group
- General group
-
Geolocation group
- Coordinates distance (distance)
- Geocoord (geocoord)
- Geographic coordinate system (coordsystem)
- Geohash (geohash)
- Geohash string (geohashstr)
- Geolocated Accuracy Radius with MaxMind GeoIP2 (mm2accuracyradius)
- Geolocated ASN (mmasn)
- Geolocated ASN with MaxMind GeoIP2 (mm2asn)
- Geolocated AS Organization Name with MaxMind GeoIP2 (mm2asorg)
- Geolocated AS owner (mmasowner)
- Geolocated City (mmcity)
- Geolocated City with MaxMind GeoIP2 (mm2city)
- Geolocated Connection Speed (mmspeed)
- Geolocated connection type with MaxMind GeoIP2 (mm2con)
- Geolocated Coordinates (mmcoordinates)
- Geolocated coordinates with MaxMind GeoIP2 (mm2coordinates)
- Geolocated Country (mmcountry)
- Geolocated Country with MaxMind GeoIP2 (mm2country)
- Geolocated ISP (mmisp)
- Geolocated ISP name with MaxMind GeoIP2 (mm2isp)
- Geolocated Latitude (mmlatitude)
- Geolocated Latitude with MaxMind GeoIP2 (mm2latitude)
- Geolocated Level 1 Subdivision with MaxMind GeoIP2 (mm2subdivision1)
- Geolocated Level 2 Subdivision with MaxMind GeoIP2 (mm2subdivision2)
- Geolocated Longitude (mmlongitude)
- Geolocated Longitude with MaxMind GeoIP2 (mm2longitude)
- Geolocated Organization (mmorg)
- Geolocated organization name with MaxMind GeoIP2 (mm2org)
- Geolocated Postal Code (mmpostalcode)
- Geolocated Postal Code with MaxMind GeoIP2 (mm2postalcode)
- Geolocated Region (mmregion)
- Geolocated Region Name (mmregionname)
- ISO-3166-1 Continent Alpha-2 Code (continentalpha2)
- ISO-3166-1 Continent Name (continentname)
- ISO-3166-1 Country Alpha-2 Code (countryalpha2)
- ISO-3166-1 Country Alpha-2 Continent (countrycontinent)
- ISO-3166-1 Country Alpha-3 Code (countryalpha3)
- ISO-3166-1 Country Latitude (countrylatitude)
- ISO-3166-1 Country Longitude (countrylongitude)
- ISO-3166-1 Country Name (countryname)
- Latitude (latitude)
- Latitude and longitude coordinates (latlon)
- Longitude (longitude)
- Parse geocoord format (parsegeo)
- Represent geocoord format (reprgeo)
- Round coordinates (gridlatlon)
- JSON group
- Logic group
-
Mathematical group
- Arc cosine (acos)
- Arc sine (asin)
- Arc tangent (atan)
- Bitwise AND (band, &)
- Bitwise left shift (lshift, <<)
- Bitwise NOT (bnot, ~)
- Bitwise OR (bor, |)
- Bitwise right shift (rshift, >>)
- Bitwise unsigned right shift (urshift, >>>)
- Bitwise XOR (bxor, ^)
- Cosine (cos)
- e (mathematical constant) (e)
- Exponential: base e (exp)
- Hyperbolic cosine (cosh)
- Hyperbolic sine (sinh)
- Hyperbolic tangent (tanh)
- Logarithm: base 2 (log2)
- Logarithm: base 10 (log10)
- Logarithm: natural / arbitrary base (log)
- Pi (mathematical constant) (pi)
- Sine (sin)
- Tangent (tan)
- Meta Analysis group
- Name group
-
Network group
- HTTP Status Description (httpstatusdescription)
- HTTP Status Type (httpstatustype)
- IP Protocol (ipprotocol)
- IP Reputation Score (reputationscore)
- IP Reputation Tags (reputation)
- IPv4 legal use (purpose)
- IPv6 host number (host)
- IPv6 routing number (routing)
- Is IPv4 (ipip4)
- Is Private IPv4 (isprivate)
- Is Public IPv4 (ispublic)
- Squid Black Lists Flags (sbl)
- Order group
-
Packet group
- Ethernet destination MAC address (etherdst)
- Ethernet payload (etherpayload)
- Ethernet source MAC address (ethersrc)
- Ethernet status (etherstatus)
- Ethernet tag (ethertag)
- EtherType (ethertype)
- Has Ethernet frame (hasether)
- Has IPv4 datagram (hasip4)
- Has TCP segment (hastcp)
- Has UDP datagram (hasudp)
- IPv4 destination address (ip4dst)
- IPv4 differentiated services (ip4ds)
- IPv4 explicit congestion notification (ip4ecn)
- IPv4 flags (ip4flags)
- IPv4 fragment offset (ip4fragment)
- IPv4 header checksum (ip4cs)
- IPv4 header length (ip4hl)
- IPv4 identification (ip4ident)
- IPv4 payload (ip4payload)
- IPv4 protocol (ip4proto)
- IPv4 source address (ip4src)
- IPv4 status (ip4status)
- IPv4 time to live (ip4ttl)
- IPv4 total length (ip4len)
- IPv4 type of service (ip4tos)
- TCP ACK (tcpack)
- TCP checksum (tcpcs)
- TCP destination port (tcpdst)
- TCP flags (tcpflags)
- TCP header length (tcphl)
- TCP payload (tcppayload)
- TCP sequence number (tcpseq)
- TCP source port (tcpsrc)
- TCP status (tcpstatus)
- TCP urgent pointer (tcpurg)
- TCP window size (tcpwin)
- UDP checksum (udpcs)
- UDP destination port (udpdst)
- UDP length (udplen)
- UDP payload (udppayload)
- UDP source port (udpsrc)
- UDP status (udpstatus)
- Statistical group
-
String group
- Contains (has, ->)
- Contains - case insensitive (weakhas)
- Contains tokens (toktains)
- Contains tokens - case insensitive (weaktoktains)
- Edit distance: Damerau (damerau)
- Edit distance: Hamming (hamming)
- Edit distance: Levenshtein (levenshtein)
- Edit distance: OSA (osa)
- Ends with (endswith)
- Format number (formatnumber)
- Hostname public suffix (publicsuffix)
- Hostname root domain (rootdomain)
- Hostname root prefix (rootprefix)
- Hostname root suffix (rootsuffix)
- Hostname subdomains (subdomain)
- Hostname top level domain (topleveldomain)
- Is empty (isempty)
- Is in (`in`, <-)
- Is in - case insensitive (weakin)
- Length (length)
- Locate (locate)
- Lower case (lower)
- Matches (matches, ~)
- Peek (peek)
- Replace all (replaceall)
- Replace first (replace)
- Shannon entropy (shannonentropy)
- Split (split)
- Split regexp (splitre)
- Starts with (startswith)
- Substitute (subs)
- Substitute all (subsall)
- Substring (substring)
- Trim both sides (trim)
- Trim the left side (ltrim)
- Trim the right side (rtrim)
- Upper case (upper)
-
Web group
- Absolute URI (absoluteuri)
- Opaque URI (opaqueuri)
- URI authority (uriauthority)
- URI fragment (urifragment)
- URI host (urihost)
- URI path (uripath)
- URI port (uriport)
- URI query (uriquery)
- URI scheme (urischeme)
- URI ssp (urissp)
- URI user (uriuser)
- URL decode (urldecode)
- User Agent Company (uacompany)
- User Agent Company URL (uacompanyurl)
- User Agent Device Icon (uadeviceicon)
- User Agent Device Information URL (uadeviceinfourl)
- User Agent Device Type (uadevicetype)
- User Agent Family (uafamily)
- User Agent Icon (uaicon)
- User Agent Information URL (uainfourl)
- User Agent is Robot (uaisrobot)
- User Agent Name (uaname)
- User Agent OS Company (uaoscompany)
- User Agent OS Company URL (uaoscompanyurl)
- User Agent OS Family (uaosfamily)
- User Agent OS Icon (uaosicon)
- User Agent OS Name (uaosname)
- User Agent OS URL (uaosurl)
- User Agent Type (uatype)
- User Agent URL (uaurl)
- User Agent Version (uaversion)
-
Aggregation operations
-
Working in the search window
-
Generate charts
- Affinity chord diagram
- Availability timeline
- Bipartite chord diagram
- Bubble chart
- Chart aggregation
- Custom date chart aggregation
- Flame graph
- Flat world map by coordinates
- Flat world map by country
- Google animated heat map
- Google area map
- Google heat map
- Graph diagram
- Histogram
- Pew Pew map
- Pie chart
- Pie layered chart
- Punch card
- Robust Random Cut Forest chart
- Sankey diagram
- Scatter plot
- Time heatmap
- Triple exponential chart
- Voronoi treemap
- Data enrichment
- Setting up a data table
- Advanced data operations
- Use case: eCommerce behavior analysis
-
Generate charts
- Managing your queries
- Best practices for data search
- Monitoring tables
-
Parsers and collectors
- About Devo tags
- Special Devo tags and data tables
-
List of Devo parsers
- Business & Consumer
- Cloud technologies
- Databases
- Host and Operating Systems
-
Network and application security
- auth.secureauth
- auth.securenvoy
- av.mcafee
- av.sophos
- box.iptables
- edr.cylance
- edr.fireeye.alerts
- edr.minervalabs.events
- endpoint.symantec
- firewall.checkpoint
- firewall.cisco firepower and vpn.cisco
- firewall.fortinet
- firewall.huawei
- firewall.juniper
- firewall.paloalto
- firewall.pfsense
- firewall.sonicwall
- firewall.sophos
- firewall.sophos.xgfirewall
- firewall.stonegate
- firewall.windows
- mail.proofpoint
- nac.aruba
- network.meraki
- network.versa
- proxy.bluecoat
- proxy.forcepoint
- proxy.squid
- uba.varonis
- vuln.beyondtrust
- vpn.pulsesecure.sa
- Network connectivity
- Web servers
- Technologies supported in CEF syslog format
- Collectors
- Activeboards
-
Dashboards
- Create a new dashboard
-
Working with dashboard widgets
- Availability timeline widget
- Chord diagram widget
- Circle world map widget
- Color key value widget
- Color world map widget
- Column chart widget
- Comparative chart widget
- Funnel widget
- Gauge meter widget
- Google heatmap widget
- Heat calendar widget
- Line chart widget
- Monitoring widget
- Pie chart widget
- Punch card widget
- Sectored pie chart widget
- Table widget
- Time heatmap widget
- Tree diagram widget
- Voronoi tree widget
- Configuring and sharing dashboards
- Alerts and notifications
- Panels
- Applications
- Tools
- Social Intelligence
- API reference
- Release notes
Microsoft Graph collector
Service description
Microsoft Graph provides many services such as Microsoft 365, Office 365, Outlook, and others. At this moment, the Microsoft Graph collector only deals with security alerts and scores retrieved from the Microsoft products. This empowers customers to streamline security operations and better defend against increasing cyber threats. The Microsoft Graph collector includes the two key entities described in the following sections:
Alerts
Alerts are potential security issues within a customer's tenant that Microsoft or partner security solutions have identified and flagged for action or notification. With the Microsoft Graph alerts entity, you can unify and streamline management of security issues across all integrated solutions.
Alerts Security Providers:
Azure Security Center
Azure Active Directory Identity Protection
Microsoft Cloud App Security
Microsoft Defender Advanced Threat Protection
Azure Advanced Threat Protection
Cloud App Security (Update coming soon form MS Graph)
Azure Information Protection
Azure Sentinel
Secure Scores
Microsoft Secure Score is a security analytics solution that gives you visibility into your security portfolio and how to improve it. With a single score, you can better understand what you have done to reduce your risk in Microsoft solutions. You can also compare your score with other organizations and see how your score has been trending over time.
The Microsoft Graph secure score and secureScoreControlProfile entities help you balance your organization's security and productivity needs while enabling the appropriate mix of security features. You can also project what your score would be after you adopt security features.
Data source description
Currently, the Microsoft Graph collector generates security activities for these resources. The collector processes the Microsoft Graph responses and sends them to the Devo platform, which will categorize all the information received on tables along rows and columns on your Devo domain.
Microsoft Graph resources
Listed in the table below are the application names, details, and how the Devo platform treats the data and to which tables sends it:
Application name | Details | Devo data tables |
---|---|---|
alerts | Represents potential security issues within a customer's tenant that Microsoft or partner security solutions have identified. Use alerts to unify and streamline security issue management across all integrated solutions. | cloud.msgraph.security.alerts |
secureScore | Represents a tenant's secure score per day of scoring data, at the tenant and control level. By default, 90 days of data is held. | cloud.msgraph.security.scores |
secureScoreControlProfile | Represents a tenant's secure score per control data. By default, it returns all controls for a tenant and can explicitly pull individual controls. | cloud.msgraph.security.scorecontrol |
For more info about Microsoft Graph API, visit Microsoft Graph Reference.
Setup
The Microsoft Graph data collector works over the Microsoft products, such as Microsoft Azure Directory. To active the alerts and secure score resources from the Microsoft Graph API, a subscription on Microsoft Azure Directory followed by an app registration should be created, as well as configuring the resources with the right permissions for the best performance of the collector.
Setting up permissions on the subscription
- Go to the Azure portal and click Azure Activity Directory.
Click App registrations → New registration to create a new app.
On the Register an Application page, give your application a name.
On Supported Accounts Type, select the third option (Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox) )
On Redirect URI (optional), leave it blank (as default) and click Register.
After registering the app, it will be displayed in a list on the App registration page. Click your app to give it permissions and configure it. You’ll see the app on the dashboard with some important information, docs, and endpoints.
- On the left menu, click Authentication → Add a platform → Mobile and desktop applications.
Mark the 3 redirect URIs and click configure.
On the left menu, click API permissions and check if you already have Microsoft Graph on the API/ Permission list. If not, click Add permission and add Microsoft Graph.
Now select Application permissions and search for Security. Check all the boxes available for the service. Then, repeat the same process with Audit and User. If you have done everything correctly, your permissions will display as shown on the green box line. Then Grant admin consent for the applications.
Troubleshooting
If you get this error “Unable to save changes. One or more of the following permission(s) are currently not supported: SecurityEvents.ReadWrite.All, SecurityEvents.Read.All, SecurityActions.Read.All, SecurityActions.ReadWrite.All. Please remove these permission(s) and retry your request. [O6b9]” you might not have set up the permission correctly. Make sure that your configuration is exactly the same as in the green box in the capture above.
Authentication
After applying the permissions, select Certificates & secrets → New client secret, enter the desired name, and copy the token.
The token will display just once. You might have to create another in case you don’t copy it.
Getting the credentials
After creating the token, go to Overview to get your Tenant ID and Client ID. This information will be used on the collector server to run the application.
Run the collector
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.
Structure
The following directory structure should be created for being used when running the Microsoft Graph collector:
<any_directory>
└── devo-collectors/
└── msgraph/
├── certs/
│ ├── chain.crt
│ ├── <your_domain>.key
│ └── <your_domain>.crt
└── config/
└── config-msgraph.yaml
Devo credentials
In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <any directory>/devo-collectors/msgraph/certs
. Learn more about security credentials in Devo here.
Editing the config-msgraph.yaml file
In the config-msgraph.yaml file, replace the <short_unique_identifier>
, <tenant_id_value>
, <client_id_value>
, and <client_secret_value>
values and enter the ones that you got in the previous steps. See inline comments to better understand the file.
config-msgraph.yaml
globals:
debug: False
id: not_used
name: microsoft_graph
persistence:
# type: redis # Redis persistence (uncomment lines to activate it and set the proper values, also comment the filesystem)
# config:
# host: 192.168.56.10
# port: 6379
# password:
# db: 0
# socket_timeout: 5
type: filesystem # File system persistence ON
config:
directory_name: state # Directory where the persistence will be saved in case of using filesystem
outputs:
devo_1: # Cloud Devo config EU (for US use us.elb.relay.logtrust.net)
type: devo_platform
config:
address: eu.elb.relay.logtrust.net
port: 443
type: SSL
chain: chain.crt
cert: <certificate>.crt
key: <certificate>.key
inputs:
microsoft_graph:
id: <short_unique_identifier> # The value of this field will be used internally for having independent persistence areas
enabled: true
requests_per_second: 5 # Setup how many request API por second
autoconfig:
enable: true # Autoconfig always True for this collector
refresh_interval_in_seconds: 60 # Setting up the time for executing autoconfig interval. 60sec is recommended.
credentials:
tenant_id: <tenant_id_value> # Azure Directory Tenant ID
client_id: <client_id_value> # Application client ID
client_secret: <client_secret_value> # Application client secret
services: # Services available for this collector are Alerts, Secure Score and Secure score control profile
alerts:
request_period_in_seconds: 60 # Setting up time interval between API requests. 60sec is recommended.
# start_time: "9999-12-31T23:59:59.999Z" # Collector Initial time. if this is blank or the data format is not valid, "time now" will be auto filled.
secure_scores:
request_period_in_seconds: 86400 # 86400 seconds => 1day in seconds, this is the recommended value, but it can be changed
# start_time: "9999-12-31T23:59:59Z"
secure_score_control_profile:
request_period_in_seconds: 86400 # 86400 seconds => 1day in seconds, this is the recommended value, but it can be changed
If you need to use a custom tag for generated messages, it can be done using the property tag inside any of the available services: alerts, secure_scores, and/or secure_score_control_profile. For example: tag: my.app.msgraph_security.{service_name}
The start_time
fields are optional. Follow these formats if you need to use them:
Alerts “start_time” format: 0000-00-00T00:00:000Z
Secure Scores “start_time” format: 0000-00-00T00:00:00Z
Download the Docker image
The collector should be deployed as a Docker container. Click here to download the Docker image of the collector as a .tgz file.
Use the following command to add the Docker image to the system:
gunzip -c collector-msgraph-docker-image-<version>.tgz | docker load
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace "<version>
" with a proper value.
The Docker image can be deployed on the following services:
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/msgraph/
docker run \
--name collector-msgraph \
--volume $PWD/certs:/devo-collector/certs \
--volume $PWD/config:/devo-collector/config \
--volume $PWD/state:/devo-collector/state \
--env CONFIG_FILE=config-msgraph.yaml \
--rm -it docker.devo.internal/collector/msgraph:<version>
Replace <version>
with a proper value.
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/msgraph/
directory.
docker-compose.yaml
version: '3'
services:
collector-msgraph:
build:
context: .
dockerfile: Dockerfile
image: docker.devo.internal/collector/msgraph:${IMAGE_VERSION:-latest}
container_name: collector-msgraph
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
environment:
- CONFIG_FILE=${CONFIG_FILE:-config-msgraph.yaml}
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/msgraph/
directory:
IMAGE_VERSION=<version> docker-compose up -d
Labels
- latest