- The Devo data analytics platform
- Getting started
- Domain administration
-
Sending data to Devo
-
The Devo In-House Relay
- Installing the Devo Relay
- Configuring the In-House Relay
- Relay migration
- Sending SSL/TLS encrypted events to the Devo relay
- Relay troubleshooting tips (v1.4.2)
- Event sources
- Other data collection methods
- Uploading log files
- Devo software
-
The Devo In-House Relay
-
Searching data
- Accessing data tables
-
Building a query
- Data types in Devo
- Build a query in the search window
- Build a query using LINQ
- Working with JSON objects in data tables
- Subqueries
-
Operations reference
-
Aggregation operations
- Average (avg)
- Count (count)
- First (first)
- First not null (nnfirst)
- HyperLogLog++ (hllpp)
- HyperLogLog++ Count Estimation (hllppcount)
- Last (last)
- Last not null (nnlast)
- Maximum (max)
- Median / 2nd quartile / Percentile 50 (median)
- Minimum (min)
- Non-null average (nnavg)
- Non-null standard deviation (biased) (nnstddev)
- Non-null standard deviation (unbiased) (nnustddev)
- Non-null variance (biased) (nnvar)
- Non-null variance (unbiased) (nnuvar)
- Percentile 10 (percentile10)
- Percentile 25 / 1st quartile (percentile25)
- Percentile 5 (percentile5)
- Percentile 75 / 3rd quartile (percentile75)
- Percentile 90 (percentile90)
- Percentile 95 (percentile95)
- Standard deviation (biased) (stddev)
- Standard deviation (unbiased) (ustddev)
- Sum (sum)
- Sum Square (sum2)
- Variance (biased) (var)
- Variance (unbiased) (uvar)
-
Arithmetic group
- Absolute value (abs)
- Addition, sum, plus / Concatenation (add, +)
- Ceiling (ceil)
- Cube root (cbrt)
- Division (div, \)
- Division remainder (rem, %)
- Floor (floor)
- Modulo (mod, %%)
- Multiplication, product (mul, *)
- Power (pow)
- Real division (rdiv, /)
- Rounding (round)
- Sign (signum)
- Square root (sqrt)
- Subtraction, minus / Additive inverse (sub, -)
-
Conversion group
- Duration (duration)
- Format date (formatdate)
- From base16, b16, hex (from16)
- From base64, b64 (from64)
- From UTF8 (fromutf8)
- From Z85, base85 (fromz85)
- Human size (humanSize)
- Make byte array (mkboxar)
- Parse date (parsedate)
- Regular expression, regexp (re)
- Template (template)
- Timestamp (timestamp)
- To base16, b16, hex (to16)
- To base64, b64, hex (to64)
- To BigInt (bigint)
- To boolean (bool)
- To Float (float)
- To image (image)
- To Int (int)
- To IPv4 (ip4)
- To IPv4 net (net4)
- To IPv6 (ip6)
- To IPv6 compatible (compatible)
- To IPv6 mapped (mapped)
- To IPv6 net (net6)
- To IPv6 translated (translated)
- To MAC address (mac)
- To string (str)
- To string (stringify)
- To UTF8 (toutf8)
- To Z85, base85 (toz85)
- Cryptography group
- Date group
- Flow group
- General group
-
Geolocation group
- Coordinates distance (distance)
- Geocoord (geocoord)
- Geographic coordinate system (coordsystem)
- Geohash (geohash)
- Geohash string (geohashstr)
- Geolocated Accuracy Radius with MaxMind GeoIP2 (mm2accuracyradius)
- Geolocated ASN (mmasn)
- Geolocated ASN with MaxMind GeoIP2 (mm2asn)
- Geolocated AS Organization Name with MaxMind GeoIP2 (mm2asorg)
- Geolocated AS owner (mmasowner)
- Geolocated City (mmcity)
- Geolocated City with MaxMind GeoIP2 (mm2city)
- Geolocated Connection Speed (mmspeed)
- Geolocated connection type with MaxMind GeoIP2 (mm2con)
- Geolocated Coordinates (mmcoordinates)
- Geolocated coordinates with MaxMind GeoIP2 (mm2coordinates)
- Geolocated Country (mmcountry)
- Geolocated Country with MaxMind GeoIP2 (mm2country)
- Geolocated ISP (mmisp)
- Geolocated ISP name with MaxMind GeoIP2 (mm2isp)
- Geolocated Latitude (mmlatitude)
- Geolocated Latitude with MaxMind GeoIP2 (mm2latitude)
- Geolocated Level 1 Subdivision with MaxMind GeoIP2 (mm2subdivision1)
- Geolocated Level 2 Subdivision with MaxMind GeoIP2 (mm2subdivision2)
- Geolocated Longitude (mmlongitude)
- Geolocated Longitude with MaxMind GeoIP2 (mm2longitude)
- Geolocated Organization (mmorg)
- Geolocated organization name with MaxMind GeoIP2 (mm2org)
- Geolocated Postal Code (mmpostalcode)
- Geolocated Postal Code with MaxMind GeoIP2 (mm2postalcode)
- Geolocated Region (mmregion)
- Geolocated Region Name (mmregionname)
- ISO-3166-1 Continent Alpha-2 Code (continentalpha2)
- ISO-3166-1 Continent Name (continentname)
- ISO-3166-1 Country Alpha-2 Code (countryalpha2)
- ISO-3166-1 Country Alpha-2 Continent (countrycontinent)
- ISO-3166-1 Country Alpha-3 Code (countryalpha3)
- ISO-3166-1 Country Latitude (countrylatitude)
- ISO-3166-1 Country Longitude (countrylongitude)
- ISO-3166-1 Country Name (countryname)
- Latitude (latitude)
- Latitude and longitude coordinates (latlon)
- Longitude (longitude)
- Parse geocoord format (parsegeo)
- Represent geocoord format (reprgeo)
- Round coordinates (gridlatlon)
- JSON group
- Logic group
-
Mathematical group
- Arc cosine (acos)
- Arc sine (asin)
- Arc tangent (atan)
- Bitwise AND (band, &)
- Bitwise left shift (lshift, <<)
- Bitwise NOT (bnot, ~)
- Bitwise OR (bor, |)
- Bitwise right shift (rshift, >>)
- Bitwise unsigned right shift (urshift, >>>)
- Bitwise XOR (bxor, ^)
- Cosine (cos)
- e (mathematical constant) (e)
- Exponential: base e (exp)
- Hyperbolic cosine (cosh)
- Hyperbolic sine (sinh)
- Hyperbolic tangent (tanh)
- Logarithm: base 2 (log2)
- Logarithm: base 10 (log10)
- Logarithm: natural / arbitrary base (log)
- Pi (mathematical constant) (pi)
- Sine (sin)
- Tangent (tan)
- Meta Analysis group
- Name group
-
Network group
- HTTP Status Description (httpstatusdescription)
- HTTP Status Type (httpstatustype)
- IP Protocol (ipprotocol)
- IP Reputation Score (reputationscore)
- IP Reputation Tags (reputation)
- IPv4 legal use (purpose)
- IPv6 host number (host)
- IPv6 routing number (routing)
- Is IPv4 (ipip4)
- Is Private IPv4 (isprivate)
- Is Public IPv4 (ispublic)
- Squid Black Lists Flags (sbl)
- Order group
-
Packet group
- Ethernet destination MAC address (etherdst)
- Ethernet payload (etherpayload)
- Ethernet source MAC address (ethersrc)
- Ethernet status (etherstatus)
- Ethernet tag (ethertag)
- EtherType (ethertype)
- Has Ethernet frame (hasether)
- Has IPv4 datagram (hasip4)
- Has TCP segment (hastcp)
- Has UDP datagram (hasudp)
- IPv4 destination address (ip4dst)
- IPv4 differentiated services (ip4ds)
- IPv4 explicit congestion notification (ip4ecn)
- IPv4 flags (ip4flags)
- IPv4 fragment offset (ip4fragment)
- IPv4 header checksum (ip4cs)
- IPv4 header length (ip4hl)
- IPv4 identification (ip4ident)
- IPv4 payload (ip4payload)
- IPv4 protocol (ip4proto)
- IPv4 source address (ip4src)
- IPv4 status (ip4status)
- IPv4 time to live (ip4ttl)
- IPv4 total length (ip4len)
- IPv4 type of service (ip4tos)
- TCP ACK (tcpack)
- TCP checksum (tcpcs)
- TCP destination port (tcpdst)
- TCP flags (tcpflags)
- TCP header length (tcphl)
- TCP payload (tcppayload)
- TCP sequence number (tcpseq)
- TCP source port (tcpsrc)
- TCP status (tcpstatus)
- TCP urgent pointer (tcpurg)
- TCP window size (tcpwin)
- UDP checksum (udpcs)
- UDP destination port (udpdst)
- UDP length (udplen)
- UDP payload (udppayload)
- UDP source port (udpsrc)
- UDP status (udpstatus)
- Statistical group
-
String group
- Contains (has, ->)
- Contains - case insensitive (weakhas)
- Contains tokens (toktains)
- Contains tokens - case insensitive (weaktoktains)
- Edit distance: Damerau (damerau)
- Edit distance: Hamming (hamming)
- Edit distance: Levenshtein (levenshtein)
- Edit distance: OSA (osa)
- Ends with (endswith)
- Format number (formatnumber)
- Hostname public suffix (publicsuffix)
- Hostname root domain (rootdomain)
- Hostname root prefix (rootprefix)
- Hostname root suffix (rootsuffix)
- Hostname subdomains (subdomain)
- Hostname top level domain (topleveldomain)
- Is empty (isempty)
- Is in (`in`, <-)
- Is in - case insensitive (weakin)
- Length (length)
- Locate (locate)
- Lower case (lower)
- Matches (matches, ~)
- Peek (peek)
- Replace all (replaceall)
- Replace first (replace)
- Shannon entropy (shannonentropy)
- Split (split)
- Split regexp (splitre)
- Starts with (startswith)
- Substitute (subs)
- Substitute all (subsall)
- Substring (substring)
- Trim both sides (trim)
- Trim the left side (ltrim)
- Trim the right side (rtrim)
- Upper case (upper)
-
Web group
- Absolute URI (absoluteuri)
- Opaque URI (opaqueuri)
- URI authority (uriauthority)
- URI fragment (urifragment)
- URI host (urihost)
- URI path (uripath)
- URI port (uriport)
- URI query (uriquery)
- URI scheme (urischeme)
- URI ssp (urissp)
- URI user (uriuser)
- URL decode (urldecode)
- User Agent Company (uacompany)
- User Agent Company URL (uacompanyurl)
- User Agent Device Icon (uadeviceicon)
- User Agent Device Information URL (uadeviceinfourl)
- User Agent Device Type (uadevicetype)
- User Agent Family (uafamily)
- User Agent Icon (uaicon)
- User Agent Information URL (uainfourl)
- User Agent is Robot (uaisrobot)
- User Agent Name (uaname)
- User Agent OS Company (uaoscompany)
- User Agent OS Company URL (uaoscompanyurl)
- User Agent OS Family (uaosfamily)
- User Agent OS Icon (uaosicon)
- User Agent OS Name (uaosname)
- User Agent OS URL (uaosurl)
- User Agent Type (uatype)
- User Agent URL (uaurl)
- User Agent Version (uaversion)
-
Aggregation operations
-
Working in the search window
-
Generate charts
- Affinity chord diagram
- Availability timeline
- Bipartite chord diagram
- Bubble chart
- Chart aggregation
- Custom date chart aggregation
- Flame graph
- Flat world map by coordinates
- Flat world map by country
- Google animated heat map
- Google area map
- Google heat map
- Graph diagram
- Histogram
- Pew Pew map
- Pie chart
- Pie layered chart
- Punch card
- Robust Random Cut Forest chart
- Sankey diagram
- Scatter plot
- Time heatmap
- Triple exponential chart
- Voronoi treemap
- Data enrichment
- Setting up a data table
- Advanced data operations
- Use case: eCommerce behavior analysis
-
Generate charts
- Managing your queries
- Best practices for data search
- Monitoring tables
-
Parsers and collectors
- About Devo tags
- Special Devo tags and data tables
-
List of Devo parsers
- Business & Consumer
- Cloud technologies
- Databases
- Host and Operating Systems
-
Network and application security
- auth.secureauth
- auth.securenvoy
- av.mcafee
- av.sophos
- box.iptables
- edr.cylance
- edr.fireeye.alerts
- edr.minervalabs.events
- endpoint.symantec
- firewall.checkpoint
- firewall.cisco firepower and vpn.cisco
- firewall.fortinet
- firewall.huawei
- firewall.juniper
- firewall.paloalto
- firewall.pfsense
- firewall.sonicwall
- firewall.sophos
- firewall.sophos.xgfirewall
- firewall.stonegate
- firewall.windows
- mail.proofpoint
- nac.aruba
- network.meraki
- network.versa
- proxy.bluecoat
- proxy.forcepoint
- proxy.squid
- uba.varonis
- vuln.beyondtrust
- vpn.pulsesecure.sa
- Network connectivity
- Web servers
- Technologies supported in CEF syslog format
- Collectors
- Activeboards
-
Dashboards
- Create a new dashboard
-
Working with dashboard widgets
- Availability timeline widget
- Chord diagram widget
- Circle world map widget
- Color key value widget
- Color world map widget
- Column chart widget
- Comparative chart widget
- Funnel widget
- Gauge meter widget
- Google heatmap widget
- Heat calendar widget
- Line chart widget
- Monitoring widget
- Pie chart widget
- Punch card widget
- Sectored pie chart widget
- Table widget
- Time heatmap widget
- Tree diagram widget
- Voronoi tree widget
- Configuring and sharing dashboards
- Alerts and notifications
- Panels
- Applications
- Tools
- Social Intelligence
- API reference
- Release notes
G Suite Alerts collector
Service description
The G Suite Alert Center manages alerts on potential issues within your domain. Apps you develop can use the Alert Center API to retrieve alerts in order to respond to them. Apps can also use the API to create and retrieve alert feedback. For example, a monitoring app could retrieve new alerts, prioritize them, and then notify members of your organization when action is needed.
Data source description
The G Suite API generates account activities for these applications and sources. The G suite collector that we provide processes the Google API responses and sends them to the Devo platform. Data will be categorized in different tables in your Devo domain, as you can check in the following table.
G Suite Alert Center
Listed in the table below are the alerts sources, types, the data that G Suite classifies and how Devo platform treats it.
Alert source | Alert type | Devo data tables |
---|---|---|
Domain wide takeout | Customer takeout initiated | cloud.gsuite.alerts.customer_takeout_initiated |
Gmail phishing | Malware reclassification | cloud.gsuite.alerts.malware_reclassification |
Misconfigured whitelist | cloud.gsuite.alerts.misconfigured_whitelist | |
Phishing reclassification | cloud.gsuite.alerts.phishing_reclassification | |
Suspicious message reported | cloud.gsuite.alerts.suspicious_message_reported | |
User reported phishing | cloud.gsuite.alerts.user_reported_phishing | |
User reported spam spike | cloud.gsuite.alerts.user_reported_spam_spike | |
Google identity | Leaked password | cloud.gsuite.alerts.eaked_password |
Suspicious login | cloud.gsuite.alerts.suspicious_login | |
Suspicious login (less secure app) | cloud.gsuite.alerts.suspicious_login_less_secure_app | |
Suspicious programmatic login | cloud.gsuite.alerts.suspicious_programmatic_login | |
User suspended | cloud.gsuite.alerts.user_suspended | |
User suspended (spam) | cloud.gsuite.alerts.user_suspended_spam | |
User suspended (spam through relay) | cloud.gsuite.alerts.user_suspended_spam_through_relay | |
User suspended (suspicious activity) | cloud.gsuite.alerts.user_suspended_suspicious_activity | |
Google Operations | Google Operations | cloud.gsuite.alerts.google_operations |
State Sponsored Attack | Government attack warning | cloud.gsuite.alerts.government_attack_warning |
Mobile device management | Device compromised | cloud.gsuite.alerts.device_compromised |
Suspicious activity | cloud.gsuite.alerts.suspicious_activity | |
AppMaker Editor | AppMaker Default Cloud SQL setup | cloud.gsuite.alerts.appmaker_default_cloud_sql_setup |
Security Center rules | Activity Rule | cloud.gsuite.alerts.activity_rules |
For more information about sources and types, visit the G Suite Alert Center API documentation.
Setup
The G Suite Alerts collector needs to be configured on Google APIs Console and the G Suite Admin Console for getting credentials and for giving the G Suite Alert Center API the right permissions so that the collector data retrieves alerts properly. Follow the instructions below to learn how to configure the services.
Credentials
Follow the next steps to create the Service Account that will be used to collect the alerts and enable the necessary API and scopes to use it.
- Create a project in the Google APIs console.
Go to the Library, search for G Suite Alert Center API, and enable it.
- Go to the Credentials section, then click on Manage service accounts.
Click Create service account, download the credentials file in JSON format and move it to
<any_directory>/devo-collector/gsuite-alerts/credentials
in the directory.
G Suite Admin Console
Grant domain access to the application in G Suite:
- Go to the G Suite Admin Console.
- Click Security and select Advanced Settings.
- Go to Authentication → Manage API client.
- Add the previously created service account. Enter the Service Account Client ID as the Client Name and in the One or More API Scopes field, enter https://www.googleapis.com/auth/apps.alerts
Grant access to the Alert Center to the user who will delegate the permissions.
Go to the G Suite Admin Console.
Click Admin Roles and select Create a new role.
Search for Alert Center in the privileges list and scroll down to view the View Access checkbox. Check it and save the new role.
Search the user and assign the new role to it.
Run the collector
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.
Structure
The following directory structure will be required as part of the setup procedure (it can be created under any directory):
<any_directory>
└── devo-collectors/
└── gsuite-alerts/
├── certs/
│ ├── chain.crt
│ ├── <your_domain>.key
│ └── <your_domain>.crt
├── credentials/
│ └── credentials-gsuite-alerts.json
└── config/
└── config-gsuite-alerts.yaml
Devo credentials
In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <any_directory>/devo-collectors/gsuite-alerts/certs/
. Learn more about security credentials in Devo here.
Editing the config-gsuite-alerts.yaml file
In the config-gsuite-alerts.yaml file, replace the <delegated_email_value>
and <source_id_value>
values and enter the ones that you got in the previous steps. In the <short_unique_identifier>
placeholder, enter the value that you choose.
config-gsuite-alerts.yaml
globals:
debug: True # <- Setup as True or False for debugging mode
id: not_used
name: gsuite
persistence: # <- Persistence setup filesystem
type: filesystem
config:
directory_name: state # <- Persistence directory
outputs:
devo_1:
type: devo_platform
config:
address: eu.elb.relay.logtrust.net # <- Devo platform address EU/US
port: 443
type: SSL
chain: chain.crt
cert: <your_domain>.crt # <- Please, replace with the certificate from your Devo domain (Administration>Credentials>x.509)
key: <your_domain>.key # <- Please, replace with the certificate from your Devo domain (Administration>Credentials>x.509)
inputs:
gsuite_alerts:
id: <short_unique_identifier> # <- "input_id", used for internal identifications
enabled: true # <- G Suite alerts service enabled
requests_per_second: 5 # <- Setting up requests per second. 5 recommended.
autoconfig: # <- "autoconfiguration" will be executed (connector doesn't support this attribute, set is "true" by default).
enabled: true # <- Autocofig setting up - True or False
refresh_interval_in_seconds: 180 # <- Time wait in second between requests - 180s recommended.
credentials:
filename: credentials-gsuite-alerts.json # <- Service Account credentials json file that you named on the getting credentials section
delegated_email: <delegated_email_value> # <- Email that will be used to delegate G Suite Alerts Viewer permissions to the Service Account
source_id: <source_id_value> # <- This value will be used for adding to message "tag" as fourth level
services: # <- List with the Alerts that you want to collect
customer_takeout_initiated:
request_period_in_seconds: 60 # <- Controls waiting time for to the next request
start_time: "9999-12-31T23:59:59.999Z"
malware_reclassification:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
misconfigured_whitelist:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
phishing_reclassification:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
suspicious_message_reported:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
user_reported_phishing:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
user_reported_spam_spike:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
leaked_password:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
suspicious_login:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
suspicious_login_less_secure_app:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
suspicious_programmatic_login:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
user_suspended:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
user_suspended_spam:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
user_suspended_spam_through_relay:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
user_suspended_suspicious_activity:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
google_operations:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
government_attack_warning:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
device_compromised:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
suspicious_activity:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
appmaker_default_cloud_sql_setup:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
activity_rule:
request_period_in_seconds: 60
start_time: "9999-12-31T23:59:59.999Z"
The start_time
fields are optional. If you would like to establish any value, the required format is 0000-00-00T00:00:00.000Z
Download the Docker image
The collector should be deployed as a Docker container. Click here to download the Docker image of the collector as a .tgz file.
Use the following command to add the Docker image to the system:
gunzip -c collector-gsuite-docker-image.tgz | docker load
Once the Docker image is imported, it will show the real name of the Docker image (including version info).
The Docker image can be deployed on the following services:
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/gsuite-alerts/
docker run \
--name collector-gsuite-alerts \
--volume $PWD/certs:/devo-collector/certs \
--volume $PWD/config:/devo-collector/config \
--volume $PWD/state:/devo-collector/state \
--env CONFIG_FILE=config-gsuite-alerts.yaml \
--rm -it docker.devo.internal/collector/gsuite:<version>
Replace <version>
with the proper version.
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/gsuite-alerts/
directory.
docker-compose.yaml
version: '3'
services:
collector-gsuite-alerts:
build:
context: .
dockerfile: Dockerfile
image: docker.devo.internal/collector/gsuite:${IMAGE_VERSION:-latest}
container_name: collector-gsuite-alerts
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
- ./state:/devo-collector/state
environment:
- CONFIG_FILE=${CONFIG_FILE:-config-gsuite-alerts.yaml}
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/gsuite-alerts/
directory:
IMAGE_VERSION=<version> docker-compose up -d
Replace <version>
with the proper version.
Activeboards
Click here to download a preconfigured Activeboard that makes use of this collector and try in your Devo domain.
To start working with it, follow these instructions:
Create a new Activeboard in your domain. Learn how to do it here.
In Edit mode, click the ellipsis button and select Edit raw configuration.
Open the downloaded file, select all the text, and copy it into the clipboard.
Paste the contents of the file in the raw editor. Make sure you replace the existing configuration completely.
Click Save changes. The Activeboard should show up immediately.
Disclaimer
The API limits the number of requests for your APIs Console project. The API project's maximum number of requests per second (project QPS) is 5 QPS and the maximum number of requests per day (project QPD) is 150,000 QPD across the account. If these limits are exceeded, the server returns an HTTP 503 status code.