- The Devo data analytics platform
- Getting started
- Domain administration
-
Sending data to Devo
-
The Devo In-House Relay
- Installing the Devo Relay
- Configuring the In-House Relay
- Relay migration
- Sending SSL/TLS encrypted events to the Devo relay
- Relay troubleshooting tips (v1.4.2)
- Event sources
- Other data collection methods
- Uploading log files
- Devo software
-
The Devo In-House Relay
-
Searching data
- Accessing data tables
-
Building a query
- Data types in Devo
- Build a query in the search window
- Build a query using LINQ
- Working with JSON objects in data tables
- Subqueries
-
Operations reference
-
Aggregation operations
- Average (avg)
- Count (count)
- First (first)
- First not null (nnfirst)
- HyperLogLog++ (hllpp)
- HyperLogLog++ Count Estimation (hllppcount)
- Last (last)
- Last not null (nnlast)
- Maximum (max)
- Median / 2nd quartile / Percentile 50 (median)
- Minimum (min)
- Non-null average (nnavg)
- Non-null standard deviation (biased) (nnstddev)
- Non-null standard deviation (unbiased) (nnustddev)
- Non-null variance (biased) (nnvar)
- Non-null variance (unbiased) (nnuvar)
- Percentile 10 (percentile10)
- Percentile 25 / 1st quartile (percentile25)
- Percentile 5 (percentile5)
- Percentile 75 / 3rd quartile (percentile75)
- Percentile 90 (percentile90)
- Percentile 95 (percentile95)
- Standard deviation (biased) (stddev)
- Standard deviation (unbiased) (ustddev)
- Sum (sum)
- Sum Square (sum2)
- Variance (biased) (var)
- Variance (unbiased) (uvar)
-
Arithmetic group
- Absolute value (abs)
- Addition, sum, plus / Concatenation (add, +)
- Ceiling (ceil)
- Cube root (cbrt)
- Division (div, \)
- Division remainder (rem, %)
- Floor (floor)
- Modulo (mod, %%)
- Multiplication, product (mul, *)
- Power (pow)
- Real division (rdiv, /)
- Rounding (round)
- Sign (signum)
- Square root (sqrt)
- Subtraction, minus / Additive inverse (sub, -)
-
Conversion group
- Duration (duration)
- Format date (formatdate)
- From base16, b16, hex (from16)
- From base64, b64 (from64)
- From UTF8 (fromutf8)
- From Z85, base85 (fromz85)
- Human size (humanSize)
- Make byte array (mkboxar)
- Parse date (parsedate)
- Regular expression, regexp (re)
- Template (template)
- Timestamp (timestamp)
- To base16, b16, hex (to16)
- To base64, b64, hex (to64)
- To BigInt (bigint)
- To boolean (bool)
- To Float (float)
- To image (image)
- To Int (int)
- To IPv4 (ip4)
- To IPv4 net (net4)
- To IPv6 (ip6)
- To IPv6 compatible (compatible)
- To IPv6 mapped (mapped)
- To IPv6 net (net6)
- To IPv6 translated (translated)
- To MAC address (mac)
- To string (str)
- To string (stringify)
- To UTF8 (toutf8)
- To Z85, base85 (toz85)
- Cryptography group
- Date group
- Flow group
- General group
-
Geolocation group
- Coordinates distance (distance)
- Geocoord (geocoord)
- Geographic coordinate system (coordsystem)
- Geohash (geohash)
- Geohash string (geohashstr)
- Geolocated Accuracy Radius with MaxMind GeoIP2 (mm2accuracyradius)
- Geolocated ASN (mmasn)
- Geolocated ASN with MaxMind GeoIP2 (mm2asn)
- Geolocated AS Organization Name with MaxMind GeoIP2 (mm2asorg)
- Geolocated AS owner (mmasowner)
- Geolocated City (mmcity)
- Geolocated City with MaxMind GeoIP2 (mm2city)
- Geolocated Connection Speed (mmspeed)
- Geolocated connection type with MaxMind GeoIP2 (mm2con)
- Geolocated Coordinates (mmcoordinates)
- Geolocated coordinates with MaxMind GeoIP2 (mm2coordinates)
- Geolocated Country (mmcountry)
- Geolocated Country with MaxMind GeoIP2 (mm2country)
- Geolocated ISP (mmisp)
- Geolocated ISP name with MaxMind GeoIP2 (mm2isp)
- Geolocated Latitude (mmlatitude)
- Geolocated Latitude with MaxMind GeoIP2 (mm2latitude)
- Geolocated Level 1 Subdivision with MaxMind GeoIP2 (mm2subdivision1)
- Geolocated Level 2 Subdivision with MaxMind GeoIP2 (mm2subdivision2)
- Geolocated Longitude (mmlongitude)
- Geolocated Longitude with MaxMind GeoIP2 (mm2longitude)
- Geolocated Organization (mmorg)
- Geolocated organization name with MaxMind GeoIP2 (mm2org)
- Geolocated Postal Code (mmpostalcode)
- Geolocated Postal Code with MaxMind GeoIP2 (mm2postalcode)
- Geolocated Region (mmregion)
- Geolocated Region Name (mmregionname)
- ISO-3166-1 Continent Alpha-2 Code (continentalpha2)
- ISO-3166-1 Continent Name (continentname)
- ISO-3166-1 Country Alpha-2 Code (countryalpha2)
- ISO-3166-1 Country Alpha-2 Continent (countrycontinent)
- ISO-3166-1 Country Alpha-3 Code (countryalpha3)
- ISO-3166-1 Country Latitude (countrylatitude)
- ISO-3166-1 Country Longitude (countrylongitude)
- ISO-3166-1 Country Name (countryname)
- Latitude (latitude)
- Latitude and longitude coordinates (latlon)
- Longitude (longitude)
- Parse geocoord format (parsegeo)
- Represent geocoord format (reprgeo)
- Round coordinates (gridlatlon)
- JSON group
- Logic group
-
Mathematical group
- Arc cosine (acos)
- Arc sine (asin)
- Arc tangent (atan)
- Bitwise AND (band, &)
- Bitwise left shift (lshift, <<)
- Bitwise NOT (bnot, ~)
- Bitwise OR (bor, |)
- Bitwise right shift (rshift, >>)
- Bitwise unsigned right shift (urshift, >>>)
- Bitwise XOR (bxor, ^)
- Cosine (cos)
- e (mathematical constant) (e)
- Exponential: base e (exp)
- Hyperbolic cosine (cosh)
- Hyperbolic sine (sinh)
- Hyperbolic tangent (tanh)
- Logarithm: base 2 (log2)
- Logarithm: base 10 (log10)
- Logarithm: natural / arbitrary base (log)
- Pi (mathematical constant) (pi)
- Sine (sin)
- Tangent (tan)
- Meta Analysis group
- Name group
-
Network group
- HTTP Status Description (httpstatusdescription)
- HTTP Status Type (httpstatustype)
- IP Protocol (ipprotocol)
- IP Reputation Score (reputationscore)
- IP Reputation Tags (reputation)
- IPv4 legal use (purpose)
- IPv6 host number (host)
- IPv6 routing number (routing)
- Is IPv4 (ipip4)
- Is Private IPv4 (isprivate)
- Is Public IPv4 (ispublic)
- Squid Black Lists Flags (sbl)
- Order group
-
Packet group
- Ethernet destination MAC address (etherdst)
- Ethernet payload (etherpayload)
- Ethernet source MAC address (ethersrc)
- Ethernet status (etherstatus)
- Ethernet tag (ethertag)
- EtherType (ethertype)
- Has Ethernet frame (hasether)
- Has IPv4 datagram (hasip4)
- Has TCP segment (hastcp)
- Has UDP datagram (hasudp)
- IPv4 destination address (ip4dst)
- IPv4 differentiated services (ip4ds)
- IPv4 explicit congestion notification (ip4ecn)
- IPv4 flags (ip4flags)
- IPv4 fragment offset (ip4fragment)
- IPv4 header checksum (ip4cs)
- IPv4 header length (ip4hl)
- IPv4 identification (ip4ident)
- IPv4 payload (ip4payload)
- IPv4 protocol (ip4proto)
- IPv4 source address (ip4src)
- IPv4 status (ip4status)
- IPv4 time to live (ip4ttl)
- IPv4 total length (ip4len)
- IPv4 type of service (ip4tos)
- TCP ACK (tcpack)
- TCP checksum (tcpcs)
- TCP destination port (tcpdst)
- TCP flags (tcpflags)
- TCP header length (tcphl)
- TCP payload (tcppayload)
- TCP sequence number (tcpseq)
- TCP source port (tcpsrc)
- TCP status (tcpstatus)
- TCP urgent pointer (tcpurg)
- TCP window size (tcpwin)
- UDP checksum (udpcs)
- UDP destination port (udpdst)
- UDP length (udplen)
- UDP payload (udppayload)
- UDP source port (udpsrc)
- UDP status (udpstatus)
- Statistical group
-
String group
- Contains (has, ->)
- Contains - case insensitive (weakhas)
- Contains tokens (toktains)
- Contains tokens - case insensitive (weaktoktains)
- Edit distance: Damerau (damerau)
- Edit distance: Hamming (hamming)
- Edit distance: Levenshtein (levenshtein)
- Edit distance: OSA (osa)
- Ends with (endswith)
- Format number (formatnumber)
- Hostname public suffix (publicsuffix)
- Hostname root domain (rootdomain)
- Hostname root prefix (rootprefix)
- Hostname root suffix (rootsuffix)
- Hostname subdomains (subdomain)
- Hostname top level domain (topleveldomain)
- Is empty (isempty)
- Is in (`in`, <-)
- Is in - case insensitive (weakin)
- Length (length)
- Locate (locate)
- Lower case (lower)
- Matches (matches, ~)
- Peek (peek)
- Replace all (replaceall)
- Replace first (replace)
- Shannon entropy (shannonentropy)
- Split (split)
- Split regexp (splitre)
- Starts with (startswith)
- Substitute (subs)
- Substitute all (subsall)
- Substring (substring)
- Trim both sides (trim)
- Trim the left side (ltrim)
- Trim the right side (rtrim)
- Upper case (upper)
-
Web group
- Absolute URI (absoluteuri)
- Opaque URI (opaqueuri)
- URI authority (uriauthority)
- URI fragment (urifragment)
- URI host (urihost)
- URI path (uripath)
- URI port (uriport)
- URI query (uriquery)
- URI scheme (urischeme)
- URI ssp (urissp)
- URI user (uriuser)
- URL decode (urldecode)
- User Agent Company (uacompany)
- User Agent Company URL (uacompanyurl)
- User Agent Device Icon (uadeviceicon)
- User Agent Device Information URL (uadeviceinfourl)
- User Agent Device Type (uadevicetype)
- User Agent Family (uafamily)
- User Agent Icon (uaicon)
- User Agent Information URL (uainfourl)
- User Agent is Robot (uaisrobot)
- User Agent Name (uaname)
- User Agent OS Company (uaoscompany)
- User Agent OS Company URL (uaoscompanyurl)
- User Agent OS Family (uaosfamily)
- User Agent OS Icon (uaosicon)
- User Agent OS Name (uaosname)
- User Agent OS URL (uaosurl)
- User Agent Type (uatype)
- User Agent URL (uaurl)
- User Agent Version (uaversion)
-
Aggregation operations
-
Working in the search window
-
Generate charts
- Affinity chord diagram
- Availability timeline
- Bipartite chord diagram
- Bubble chart
- Chart aggregation
- Custom date chart aggregation
- Flame graph
- Flat world map by coordinates
- Flat world map by country
- Google animated heat map
- Google area map
- Google heat map
- Graph diagram
- Histogram
- Pew Pew map
- Pie chart
- Pie layered chart
- Punch card
- Robust Random Cut Forest chart
- Sankey diagram
- Scatter plot
- Time heatmap
- Triple exponential chart
- Voronoi treemap
- Data enrichment
- Setting up a data table
- Advanced data operations
- Use case: eCommerce behavior analysis
-
Generate charts
- Managing your queries
- Best practices for data search
- Monitoring tables
-
Parsers and collectors
- About Devo tags
- Special Devo tags and data tables
-
List of Devo parsers
- Business & Consumer
- Cloud technologies
- Databases
- Host and Operating Systems
-
Network and application security
- auth.secureauth
- auth.securenvoy
- av.mcafee
- av.sophos
- box.iptables
- edr.cylance
- edr.fireeye.alerts
- edr.minervalabs.events
- endpoint.symantec
- firewall.checkpoint
- firewall.cisco firepower and vpn.cisco
- firewall.fortinet
- firewall.huawei
- firewall.juniper
- firewall.paloalto
- firewall.pfsense
- firewall.sonicwall
- firewall.sophos
- firewall.sophos.xgfirewall
- firewall.stonegate
- firewall.windows
- mail.proofpoint
- nac.aruba
- network.meraki
- network.versa
- proxy.bluecoat
- proxy.forcepoint
- proxy.squid
- uba.varonis
- vuln.beyondtrust
- vpn.pulsesecure.sa
- Network connectivity
- Web servers
- Technologies supported in CEF syslog format
- Collectors
- Activeboards
-
Dashboards
- Create a new dashboard
-
Working with dashboard widgets
- Availability timeline widget
- Chord diagram widget
- Circle world map widget
- Color key value widget
- Color world map widget
- Column chart widget
- Comparative chart widget
- Funnel widget
- Gauge meter widget
- Google heatmap widget
- Heat calendar widget
- Line chart widget
- Monitoring widget
- Pie chart widget
- Punch card widget
- Sectored pie chart widget
- Table widget
- Time heatmap widget
- Tree diagram widget
- Voronoi tree widget
- Configuring and sharing dashboards
- Alerts and notifications
- Panels
- Applications
- Tools
- Social Intelligence
- API reference
- Release notes
Google Cloud Platform collector
Service description
This collector will retrieve the information stored in Google Cloud Platform (GCP), such as audit logs, networking, load balance, and more.
Data source description
The Google Cloud Platform (GCP) centralizes all the monitoring information from all services in the cloud catalog inside a service called Stackdriver.
There is certain information that is enabled by default and is free of charge. However, generating some other information will imply some costs and that's why it must be enabled manually. In both cases, the generated information (we will call them messages) will arrive at the Stackdriver service.
There exist many more GCP services than the ones shown in the diagram, these are only a small sample.
The Stackdriver service has different ways of exporting the information stored inside (structured in messages). In this case, it uses another GCP service called PubSub, which contains a "Topic" object that will receive a filtered set of messages from the Stackdriver service. Then, the GCP collector will retrieve all those messages from the "Topic" object using a subscription (concretely in "pull" mode):
In order to make the retrieving process easier, it is recommended to split the source messages using different "Topic" objects. This can be done by resource type, region, project id, and so on:
Service name definitions
The GCP collector that we provide processes the stored information and sends it to the Devo platform. Data will be categorized in different tables in your Devo domain, as you can check in the following table.
Services | Description | Devo data tables |
---|---|---|
Audit Resource | Get information related to Cloud Audit Logs, which maintains three audit logs for each Google Cloud project, folder, and organization: Admin Activity, Data Access, and System Event. Google Cloud services write audit log entries to these logs to help you answer the questions of "who did what, where, and when?" within your Google Cloud resources. | cloud.gcp.logging.audited_resource |
Virtual Machines | Get information about the Virtual Machine Instance, CPU, Memory, and more. | cloud.gcp.logging.gce_instance |
Redis | Get information on memory store for Redis instance's performance and allows you to check in on your instances to make sure they're behaving as desired. | cloud.gcp.logging.redis |
Networking | Get information related to how the memory store uses VPC peering to connect your VPC network with the internal Google services network. | cloud.gcp.logging.gce_subnetwork |
Load Balance | Get information related to how a load balancer distributes user traffic across multiple instances of your applications. | cloud.gcp.logging.load_balancer |
Kubernetes Engine | Get information about Google Kubernetes Engine (GKE) that provides a managed environment for deploying, managing, and scaling your containerized applications using Google infrastructure. | cloud.gcp.logging.k8s |
PubSub | Get information on the asynchronous messaging service that decouples services that produce events from services that process events. | cloud.gcp.logging.pubsub |
Setup
The collector will be executed inside of the Collector Server as well as a Standalone mode (docker container). Some configurations will be necessary for the proper functioning of the collector.
Source setup types
In order to have the GCP collector running in a proper way, the data source (in this case GCP) must have several service structures configured. Specifically, it is required to set subscription objects to pull mode.
The required GCP setup can be created either automatically or manually. Depending on a configuration file property called autoconfig, one mode or another will be used and the behavior will be the following:
Automatic | The autoconfig property must have the enabled option set to true. Also, the internal service definition should have a configuration class value set (this cannot be set by the customer). The autoconfiguration functionality will create different "sinks" using the predefined service definition or using some custom definitions from the configuration file. |
---|---|
Manual | The autoconfig property must have the enabled option set to false and any configuration class defined in the internal service definition will be ignored. Depending if the property subscription_name is set, two different behaviors will be applied:
|
Service types
The GCP collector must have at least one service entry in the services section. These services can be of two types:
Predefined services | These services will not require to set some properties in the configuration file since they are already internally defined. This is the list of predefined services:
|
---|---|
Custom services | These services will require some extra properties in the configuration file:
The syntax to be used in sink_filter_resource_* properties is detailed here. |
Credentials
Follow the next steps to create the Service Account that will be used to collect the alerts and enable the necessary API and scopes to use it.
- Go to your Google GCP console project and open the left menu. Click IAM & Admin → Service Accounts.
Click on + Create Service Account to create the credentials.
- Follow the 3 steps to create the credentials. Enter a name and click on Create. Steps 2 and 3 are optional, so you may just click Continue to go to step 3 and select Done.
- Now you have to add the Keys to the service account that was previously created and download it in JSON format. After clicking the Done button, you’ll be redirected to the services accounts of your project. Search the service account that you created and click it.
In the Service account details area, click ADD KEY → Create new key, select the JSON format, and click CREATE. Download the credentials file and move it to the
<any_directory>/devo-collectors/gcp/credentials/
directory.
Running collector
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.
Structure
The following directory structure will be required as part of the setup procedure (it can be created under any directory):
<any_directory>
└── devo-collectors/
└── gcp/
├── certs/
│ ├── chain.crt
│ ├── <your_domain>.key
│ └── <your_domain>.crt
├── credentials/
│ └── credentials-gcp.json
└── config/
└── config-gcp.yaml
Devo credentials
In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <any_directory>/devo-collectors/gcp/certs/
. Learn more about security credentials in Devo here.
Editing the config-gcp.yaml file
In the config-gcp.yaml file, replace the <short_unique_identifier>
,
<source_id>, <project_id>, <credentials_file>
and <credentials_file_content_base64>
values and enter the ones that you got in the previous steps. In the <short_unique_identifier>
placeholder, enter the value that you choose.
The GCP collector has many different configurations. See some usage config examples and custom services below.
Predefined services and autoconfig functionality enabled:
config-gcp.yaml
globals: debug: false id: not_used name: gcp persistence: type: filesystem # File system persistence ON config: directory_name: state # Directory where the persistence will be saved in case of using filesystem outputs: devo_1: # Cloud Devo config EU (for US use us.elb.relay.logtrust.net) type: devo_platform config: address: collector-eu.devo.io # URL to use for sending messages port: 443 # SSL for using certificates or TCP for sending without encryption type: SSL chain: chain.crt # \ cert: <your_domain>.crt # Credential files downloaded from a Devo domain key: <your_domain>.key # / inputs: gcp: id: <short_unique_identifier> # The value of this field will be used internally for having independent persistence areas enabled: true # If "false" value this "input" will not be used by collector requests_per_second: 5 # Setup how many request API por second use_time_slots_for_retrievals: false autoconfig: enabled: true # Determines if the "autoconfiguration" functionality will be executed refresh_interval_in_seconds: 3600 # Setting up the time for executing autoconfig interval. 3600 seconds is recommended. credentials: source_id: <source_id> # This value will be used for adding to message "tag" as fourth level project_id: <project_id> # This value will be used for adding to message "tag" as fifth level filename: <filename> # Credentials file generated from the "GCP administration console" file_content_base64: <file_content_base64> # Credentials file content generated from the "GCP administration console" in base64 format services: # Services available for this collector are virtual machines, networking, load balance, redis and PubSub. virtual_machines: request_period_in_seconds: 15 # Setting up time interval between API requests. 15sec is recommended. regions: # Regions for being used as filter of source data, use "all" value for no filter by region - all networking: request_period_in_seconds: 15 regions: - all load_balancing: request_period_in_seconds: 15 regions: - all pubsub: request_period_in_seconds: 15 regions: - all redis: request_period_in_seconds: 15 regions: - all custom_service_1: tag_subtype: "all1" subscription_name: "custom_service_1-subscription" request_period_in_seconds: 30 custom_service_2: tage: "my.app.gcp.test1" subscription_name: "custom_service_2-subscription" request_period_in_seconds: 30
The custom configuration section can be set with many different values. Find below several examples of the custom configuration section and their impact on the different services involved.
Predefined services and autoconfig functionality disabled:
config-gcp.yaml - custom configuration section, example 1
globals: debug: false id: not_used name: gcp persistence: type: filesystem # File system persistence ON config: directory_name: state # Directory where the persistence will be saved in case of using filesystem outputs: devo_1: # Cloud Devo config EU (for US use us.elb.relay.logtrust.net) type: devo_platform config: address: collector-eu.devo.io # URL to use for sending messages port: 443 # SSL for using certificates or TCP for sending without encryption type: SSL chain: chain.crt # \ cert: <your_domain>.crt # Credential files downloaded from a Devo domain key: <your_domain>.key # / inputs: gcp: id: <short_unique_identifier> # The value of this field will be used internally for having independent persistence areas enabled: true # If "false" value this "input" will not be used by collector requests_per_second: 5 # Setup how many request API por second use_time_slots_for_retrievals: false autoconfig: enabled: false # Determines if the "autoconfiguration" functionality will be executed refresh_interval_in_seconds: 3600 # Setting up the time for executing autoconfig interval. 3600 seconds is recommended. credentials: source_id: <source_id> # This value will be used for adding to message "tag" as fourth level project_id: <project_id> # This value will be used for adding to message "tag" as fifth level filename: <filename> # Credentials file generated from the "GCP administration console" file_content_base64: <file_content_base64> # Credentials file content generated from the "GCP administration console" in base64 format services: # Services available for this collector are virtual machines, networking, load balance, redis and PubSub. virtual_machines: request_period_in_seconds: 15 # Setting up time interval between API requests. 15sec is recommended. regions: - europe-west2 # Regions for being used as filters of source data - europe-west3 networking: request_period_in_seconds: 15 regions: - europe-west3 load_balancing: request_period_in_seconds: 15 regions: - all pubsub: request_period_in_seconds: 15 regions: - all redis: request_period_in_seconds: 15 regions: - all custom_service_1: tag_subtype: "all1" subscription_name: "custom_service_1-subscription" request_period_in_seconds: 30 custom_service_2: tag: "my.app.gcp.test1" subscription_name: "custom_service_2-subscription" request_period_in_seconds: 30
Custom services, autoconfig functionality enabled and using regions:
config-gcp.yaml - custom configuration section, example 2
globals: debug: false id: not_used name: gcp persistence: type: filesystem # File system persistence ON config: directory_name: state # Directory where the persistence will be saved in case of using filesystem outputs: devo_1: # Cloud Devo config EU (for US use us.elb.relay.logtrust.net) type: devo_platform config: address: collector-eu.devo.io # URL to use for sending messages port: 443 # SSL for using certificates or TCP for sending without encryption type: SSL chain: chain.crt # \ cert: <your_domain>.crt # Credential files downloaded from a Devo domain key: <your_domain>.key # / inputs: gcp: id: <short_unique_identifier> # The value of this field will be used internally for having independent persistence areas enabled: true # If "false" value this "input" will not be used by collector requests_per_second: 5 # Setup how many request API por second use_time_slots_for_retrievals: false autoconfig: enabled: true # Determines if the "autoconfiguration" functionality will be executed refresh_interval_in_seconds: 3600 # Setting up the time for executing autoconfig interval. 3600 seconds is recommended. credentials: source_id: <source_id> # This value will be used for adding to message "tag" as fourth level project_id: <project_id> # This value will be used for adding to message "tag" as fifth level filename: <filename> # Credentials file generated from the "GCP administration console" file_content_base64: <file_content_base64> # Credentials file content generated from the "GCP administration console" in base64 format services: all_resources_na: sink_filter_resource: "resource.type:\"\"" sink_filter_resource_region: "resource.type:\"\" AND resource.labels.zone:\"{region}\"" regions: - northamerica - us all_resources_europe: sink_filter_resource: "resource.type:\"\"" sink_filter_resource_region: "resource.type:\"\" AND resource.labels.zone:\"{region}\"" regions: - europe
The syntax to be used in sink_filter_resource_* properties is detailed here.
Custom services, autoconfig functionality disabled and using regions:
config-gcp.yaml - custom configuration section, example 3
globals: debug: false id: not_used name: gcp persistence: type: filesystem # File system persistence ON config: directory_name: state # Directory where the persistence will be saved in case of using filesystem outputs: devo_1: # Cloud Devo config EU (for US use us.elb.relay.logtrust.net) type: devo_platform config: address: collector-eu.devo.io # URL to use for sending messages port: 443 # SSL for using certificates or TCP for sending without encryption type: SSL chain: chain.crt # \ cert: <your_domain>.crt # Credential files downloaded from a Devo domain key: <your_domain>.key # / inputs: gcp: id: <short_unique_identifier> # The value of this field will be used internally for having independent persistence areas enabled: true # If "false" value this "input" will not be used by collector requests_per_second: 5 # Setup how many request API por second use_time_slots_for_retrievals: false autoconfig: enabled: false # Determines if the "autoconfiguration" functionality will be executed refresh_interval_in_seconds: 3600 # Setting up the time for executing autoconfig interval. 3600 seconds is recommended. credentials: source_id: <source_id> # This value will be used for adding to message "tag" as fourth level project_id: <project_id> # This value will be used for adding to message "tag" as fifth level filename: <filename> # Credentials file generated from the "GCP administration console" file_content_base64: <file_content_base64> # Credentials file content generated from the "GCP administration console" in base64 format services: all_resources_na: sink_filter_resource: "resource.type:\"\"" sink_filter_resource_region: "resource.type:\"\" AND resource.labels.zone:\"{region}\"" regions: - northamerica - us all_resources_europe: sink_filter_resource: "resource.type:\"\"" sink_filter_resource_region: "resource.type:\"\" AND resource.labels.zone:\"{region}\"" regions: - europe
The syntax to be used in sink_filter_resource_* properties is detailed here.
Custom services and specifying "subscription name". In order to do this, the autoconfig functionality must be disabled and the use of the regions section is not allowed in this case (it will be ignored if exists):
config-gcp.yaml - custom configuration section, example 4
globals: debug: false id: not_used name: gcp persistence: type: filesystem # File system persistence ON config: directory_name: state # Directory where the persistence will be saved in case of using filesystem outputs: devo_1: # Cloud Devo config EU (for US use us.elb.relay.logtrust.net) type: devo_platform config: address: collector-eu.devo.io # URL to use for sending messages port: 443 # SSL for using certificates or TCP for sending without encryption type: SSL chain: chain.crt # \ cert: <your_domain>.crt # Credential files downloaded from a Devo domain key: <your_domain>.key # / inputs: gcp: id: <short_unique_identifier> # The value of this field will be used internally for having independent persistence areas enabled: true # If "false" value this "input" will not be used by collector requests_per_second: 5 # Setup how many request API por second use_time_slots_for_retrievals: false autoconfig: enabled: false # Determines if the "autoconfiguration" functionality will be executed refresh_interval_in_seconds: 90 # Setting up the time for executing autoconfig interval. 90sec is recommended. credentials: source_id: <source_id> # This value will be used for adding to message "tag" as fourth level project_id: <project_id> # This value will be used for adding to message "tag" as fifth level filename: <filename> # Credentials file generated from the "GCP administration console" file_content_base64: <file_content_base64> # Credentials file content generated from the "GCP administration console" in base64 format services: all_resources_usa: subscription_name: "subscription-a" all_resources_europe: subscription_name: "subscription-b"
Download the Docker image
The collector should be deployed as a Docker container. Click here to download the Docker image of the collector as a .tgz file.
Use the following command to add the Docker image to the system:
gunzip -c collector-gcp-docker-image-<version>.tgz | docker load
Once the Docker image is imported, it will show the real name of the Docker image (including version info).
The Docker image can be deployed on the following services:
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/gcp/
docker run \
--name collector-gcp \
--volume $PWD/certs:/devo-collector/certs \
--volume $PWD/config:/devo-collector/config \
--volume $PWD/state:/devo-collector/state \
--env CONFIG_FILE=config-gcp.yaml \
--rm -it docker.devo.internal/collector/gcp:<version>
Replace <version>
with the proper version.
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/gcp/
directory.
docker-compose.yaml
version: '3'
services:
collector-gcp:
build:
context: .
dockerfile: Dockerfile
image: docker.devo.internal/collector/gcp:${IMAGE_VERSION:-latest}
container_name: collector-gcp
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
- ./state:/devo-collector/state
environment:
- CONFIG_FILE=${CONFIG_FILE:-config-gcp.yaml}
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/gcp/
directory:
IMAGE_VERSION=<version> docker-compose up -d
Activeboards
Click here to download a preconfigured Activeboard that makes use of this collector and try in your Devo domain.
To instantiate them, follow these instructions:
Create a new Activeboard in your domain. Learn how to do it here.
In Edit mode, click the ellipsis button and select Edit raw configuration.
Open the downloaded file, select all the text, and copy it into the clipboard.
Paste the contents of the file in the raw editor. Make sure you replace the existing configuration completely.
Click Save changes. The Activeboard should show up immediately.