Parsers and collectors / Collectors / Google Cloud Platform collector

Google Cloud Platform collector

The Google Cloud Platform (GCP) centralizes all the monitoring information from all services in the cloud catalog inside a service called Stackdriver.

There is certain information that is enabled by default and is free of charge. However, generating some other information will imply some costs and that's why it must be enabled manually. In both cases, the generated information (we will call them messages) will arrive at the Stackdriver service.

The Stackdriver service has different ways of exporting the information stored inside (structured in messages). In this case, it uses another GCP service called PubSub, which contains a "Topic" object that will receive a filtered set of messages from the Stackdriver service. Then, the GCP collector will retrieve all those messages from the "Topic" object using a subscription (concretely in "pull" mode):

In order to make the retrieving process easier, it is recommended to split the source messages using different "Topic" objects. This can be done by resource type, region, project id and so on:

There exist many more GCP services than the ones shown in the diagram, these are only a small sample.

The GCP collector that we provide processes the stored information and sends it to the Devo platform. Data will be categorized in different tables in your Devo domain, as you can check in the following table.

GCP services

Listed in the table below are some service names, details and how Devo platform treats the data.

Services	Devo technologies
Audit Resource	cloud.gcp.logging.audited_resource
Virtual Machines	cloud.gcp.logging.gce_instance
Redis	cloud.gcp.logging.redis
Networking	cloud.gcp.logging.gce_subnetwork
Load Balance	cloud.gcp.logging.load_balancer
Kubernetes Engine	cloud.gcp.logging.k8s
PubSub	cloud.gcp.logging.pubsub

Setup

Structure

The collector will be executed as a Docker container, and some files will be necessary for the proper functioning of the container. The following directory structure will be required as part of the setup procedure (it can be created under any directory):

<any_directory>
└── devo-collectors/
    └── gcp/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        ├── credentials/
        │   └── credentials-gcp.json
        └── config/
            └── config-gcp.yaml

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <any_directory>/devo-collectors/gcp/certs/. Learn more about security credentials in Devo here.

GCP credentials

Follow the next steps to create the service account that will be used to collect the alerts and to enable the necessary API and scopes to use it.

Create a project in the Google GCP console:
Go to the credentials section, then click on Manage Service Accounts.
Click Create service account, download the credentials file in JSON format and move it to <any_directory>/devo-collectors/gcp/credentials/ in the directory.

Configuration

Data source setup

In order to have the GCP collector running in a proper way, the data source (in this case GCP) must have several service structures configured. Specifically, it is required to set subscription objects to pull mode.

The required GCP setup can be created either automatically or manually. Depending on a configuration file property called autoconfig, one mode or another will be used and the behavior will be the following:

Automatic	The autoconfig property must have the enabled option set to true. Also, the internal service definition should have a configuration class value set (this cannot be set by the customer). The autoconfiguration functionality will create different "sinks" using the predefined service definition or using some custom definitions from the configuration file.
Manual	The autoconfig property must have the enabled option set to false and any configuration class defined in the internal service definition will be ignored. Depending if the property subscription_name is set, two different behaviors will be applied: If subscription_name is defined, the value will be used for connecting to the GCP subscription. If the subscription doesn't exist, it will cause an eternal loop checking every minute if that subscription exists (the console logs will show a warning). If subscription_name is not defined, a subscription with the same name as the one auto-calculated by the autoconfiguration functionality will be expected.

Automatic

The autoconfig property must have the enabled option set to true. Also, the internal service definition should have a configuration class value set (this cannot be set by the customer). The autoconfiguration functionality will create different "sinks" using the predefined service definition or using some custom definitions from the configuration file.

Manual

The autoconfig property must have the enabled option set to false and any configuration class defined in the internal service definition will be ignored. Depending if the property subscription_name is set, two different behaviors will be applied:

If subscription_name is defined, the value will be used for connecting to the GCP subscription. If the subscription doesn't exist, it will cause an eternal loop checking every minute if that subscription exists (the console logs will show a warning).
If subscription_name is not defined, a subscription with the same name as the one auto-calculated by the autoconfiguration functionality will be expected.

Service types

The GCP collector must have at least one service entry in the services section. These services can be of two types:

Predefined services	These services will not require to set some properties in the configuration file since they are already internally defined. This is the list of predefined services: virtual_machines → Resources from "Compute Engine" GCP service. networking → Resources from "VPC network" GCP service. load_balancing → Resources from "Network services > Load balancing" GCP service. pubsub → Resources from "PubSub" service. redis → Resources from "Redis" service.
Custom services	These services will require some extra properties in the configuration file: sink_filter_resource → This is the value of the filter that will be applied to the GCP Sink object when the all value is set in the regions section during the automatic configuration phase (if enabled). sink_filter_resource_region → This is the value of the filter that will be applied to the GCP Sink object when any value distinct from all is set in the regions section during the autoconfiguration phase (if enabled). The filter must contain a placeholder with value {region} as part of its value. The syntax to be used in sink_filter_resource_* properties is detailed here.

Predefined services

These services will not require to set some properties in the configuration file since they are already internally defined. This is the list of predefined services:

virtual_machines → Resources from "Compute Engine" GCP service.
networking → Resources from "VPC network" GCP service.
load_balancing → Resources from "Network services > Load balancing" GCP service.
pubsub → Resources from "PubSub" service.
redis → Resources from "Redis" service.

Custom services

These services will require some extra properties in the configuration file:

sink_filter_resource → This is the value of the filter that will be applied to the GCP Sink object when the all value is set in the regions section during the automatic configuration phase (if enabled).
sink_filter_resource_region → This is the value of the filter that will be applied to the GCP Sink object when any value distinct from all is set in the regions section during the autoconfiguration phase (if enabled). The filter must contain a placeholder with value {region} as part of its value.

The syntax to be used in sink_filter_resource_* properties is detailed here.

Config file examples

The configuration file config-gcp.yaml has two sections, one section which is inherited from the base library used for creating the collector (base configuration) and another section that is common to all collectors developed by Devo (custom configuration).

The base configuration section is as follows:

config-gcp.yaml - base configuration section

config:
  path: ./config                                       # <- Absolute path where configuration will be stored
  version: 1
devo_connection:
  address: collector-eu.devo.io                        # <- URL to use for sending messages
  port: 443
  type: SSL                                            # <- SSL for using certificates or TCP for sending without encryption
  chain: ./certs/chain.crt                             # \
  cert: ./certs/<devo_domain>.crt                      #  > Credential files downloaded from a Devo domain
  key: ./certs/<devo_domain>.key                       # /
log:
  backup_count: 5
  devo_out: 'true'
  devo_stats: 'true'
  format: '%(asctime)s %(levelname)s %(message)s'
  level: info
  max_size: 2097152
  path: ./log                                          # <- Directory to be used for storing the application internal logs
  stdout: 'false'
state:
  path: ./state                                        # <- Directory to be used for storing the "state" files

The custom configuration section can be set with many different values depending on the collector and their requirements. Find below some examples of the custom configuration section file:

Predefined services and Autoconfiguration functionality enabled:

config-gcp.yaml - custom configuration section, example 1

custom:
  inputs:
    gcp:
      id: 1                                        # <- "input_id", used for internal identifications
      enabled: true                                # <- If "false", this "input" will not be used by the collector
      autoconfig:
        enabled: true                              # <- Determines if the "autoconfiguration" functionality will be executed
      credentials:
        source_id: source-1                        # <- This value will be used for adding to message "tag" as fourth level
        project_id: project-a                      # <- This value will be used for adding to message "tag" as fifth level
        credentials_file: credentials-gcp.json     # <- Credentials file generated from the "GCP administration console"
      services:
        virtual_machines:
          regions:                                 # <- Regions to being used as filter of source data, use "all" to not filter by region
            - europe-west2
            - europe-west3
        networking:
          regions:
            - europe-west3
        load_balancing:
          regions:
            - all

Predefined services and Autoconfiguration functionality disabled:

config-gcp.yaml - custom configuration section, example 2

custom:
  inputs:
    gcp:
      id: 1                                        # <- "input_id", used for internal identifications
      enabled:true                                 # <- If "false", this "input" will not be used by the collector
      autoconfig:
        enabled: false                             # <- Determines if the "autoconfiguration" functionality will be executed
      credentials:
        source_id: source-1                        # <- This value will be used for adding to message "tag" as fourth level
        project_id: project-a                      # <- This value will be used for adding to message "tag" as fifth level
        credentials_file: credentials-gcp.json     # <- Credentials file generated from the "GCP administration console"
      services:
        virtual_machines:
          regions:                                 # <- Regions to be used as filter of source data, use "all" to not filter by region
            - europe-west2
            - europe-west3
        networking:
          regions:
            - europe-west3
        load_balancing:
          regions:
            - all

Custom services, autoconfiguration functionality enabled and using regions:

config-gcp.yaml - custom configuration section, example 3

custom:
  inputs:
    gcp:
      id: 1                                         # <- "input_id", used for internal identifications
      enabled: true                                 # <- If "false", this "input" will not be used by the collector
      autoconfig:
        enabled: true                               # <- Determines if the "autoconfiguration" functionality will be executed
      credentials:
        source_id: source-1                         # <- This value will be used for adding to message "tag" as fourth level
        project_id: project-a                       # <- This value will be used for adding to message "tag" as fifth level
        credentials_file: credentials-gcp.json
      services:
        all_resources_na:
          sink_filter_resource: "resource.type:\"\""
          sink_filter_resource_region: "resource.type:\"\" AND resource.labels.zone:\"{region}\""
          regions:
            - northamerica
            - us
        all_resources_europe:
          sink_filter_resource: "resource.type:\"\""
          sink_filter_resource_region: "resource.type:\"\" AND resource.labels.zone:\"{region}\""
          regions:
            - europe

The syntax to be used in sink_filter_resource_* properties is detailed here.

Custom services, autoconfiguration functionality disabled and using regions:

config-gcp.yaml - custom configuration section, example 4

custom: 
  inputs:
    gcp:
    id: 1                                        # <- "input_id", used for internal identifications
    enabled: true                                # <- If "false", this "input" will not be used by the collector
    autoconfig:
      enabled: false                             # <- Determines if the "autoconfiguration" functionality will be executed
    credentials:
      source_id: source-1                        # <- This value will be used for adding to message "tag" as fourth level
      project_id: project-a                      # <- This value will be used for adding to message "tag" as fifth level
      credentials_file: credentials-gcp.json
      services:
        all_resources_na:
          sink_filter_resource: "resource.type:\"\""
          sink_filter_resource_region: "resource.type:\"\" AND resource.labels.zone:\"{region}\""
          regions:
            - northamerica
            - us
        all_resources_europe:
          sink_filter_resource: "resource.type:\"\""
          sink_filter_resource_region: "resource.type:\"\" AND resource.labels.zone:\"{region}\""
          regions:
            - europe

The syntax to be used in sink_filter_resource_* properties is detailed here.

Custom services and specifying "subscription name". In order to do this, the autoconfiguration functionality must be disabled and the use of the regions section is not allowed in this case (it will be ignored if exists):

config-gcp.yaml - custom configuration section, example 5

custom:
  inputs:
    gcp:
      id: 1                                        # <- "input_id", used for internal identifications
      enabled: true                                # <- If "false", this "input" will not be used by the collector
      autoconfig:
        enabled: false                             # <- Determines if the "autoconfiguration" functionality will be executed
      credentials:
        source_id: source-1                        # <- This value will be used for adding to message "tag" as fourth level
        project_id: project-a                      # <- This value will be used for adding to message "tag" as fifth level
        credentials_file: credentials-gcp.json
      services:
        all_resources_usa:
          subscription_name: "subscription-a"
        all_resources_europe:
          subscription_name: "subscription-b"

Run the collector

The collector should be deployed as a Docker container. Click here to download the Docker image of the collector as a .tgz file.

Use the following command to add the Docker image to the system:

gunzip -c collector_cloud.tgz | docker load

Once the Docker image is imported, it will show the real name of the Docker image (including version info).

The Docker image can be deployed on the following services:

Docker
Docker Compose

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/gcp/

docker run \
--name collector-gcp \
--volume $PWD/certs:/devo-collector/certs \
--volume $PWD/config:/devo-collector/config \
--volume $PWD/credentials:/devo-collector/credentials \
--env CONFIG_FILE=config-gcp.yaml \
--rm -it docker.devo.internal/collector/cloud:<version>

Replace <version> with the proper version.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/gcp/ directory.

docker-compose.yaml

version: '3'
services:
  collector-gcp:
    image: docker.devo.com/collector/cloud:${IMAGE_VERSION:-latest}
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./credentials:/devo-collector/credentials
  environment:
      - CONFIG_FILE=${CONFIG_FILE:-config-gcp.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/gcp/ directory:

IMAGE_VERSION=<version> docker-compose up -d

Download as PDF

Did you find what you were looking for?

If not, please let us know what you need. Your feedback will help us to improve.