G Suite Reports collector

Service description

The G Suite Reports API is used to gain insights on content management with Google activity, audit administrator actions, and generate customer and user usage reports.

Data source description

The G Suite API generates account activities for these applications and sources. The collector process the Google API responses and send them to Devo platform that will categorize all information received on tables along rows and columns on your Devo domain.

G Suite Reports

Listed in the table below are some application names, details and how Devo platform treats the data.

Each report uses the basic endpoint request with report-specific parameters or event type. The maximum time period for each report is the last 180 days. For more references about G Suite Reports, visit the Google API Reference documentation.

Setup

G Suite Reports credentials

G Suite Reports requires a script to generate a token (token.pickle). Google uses a third-party service (OAuth) to authenticate this service. To create the required credentials, you must access the Google API Console and get the credentials-gsuite-reports.json file.

1. Create a project in the Google API console.
   
   
   
2. Go to Credentials → + Create Credentials and select OAuth Client ID:
   
   
   
3. Select Desktop App, enter the name you prefer and click the Create button.
   
   
   Click OK in the popup window that appears.
4. Download the credentials in JSON format by clicking the black arrow in the right corner.
   
   
   

5. Rename the file as credentials-gsuite-reports.json and move it to <any_directory>/devo-collectors/gsuite-reports/credentials directory.
   

Assigning roles to G Suite Reports 

Follow these steps to grant domain access to G Suite Reports:

1. Go to the G Suite Admin Console.

2. Click the Security option and select Advanced Settings.
   
   

3. From the Authentication section, click Manage API client access.

4. Grant the user who will delegate the permissions access to the G Suite Reports.

   1. Click Admin Roles and select Create a new role.

   2. Search for Reports API privileges list and drop-down to view the View Access checkbox. Check it and save the new role.

   3. Search the user and assign the new role.

Access token

To run these applications, you'll need:

• Python 3.6 or greater

• A G Suite domain with API access enabled.

• A Google account in that domain with administrator privileges.

Follow these steps to generate the required token:

1. Get the script (quickstart.py) to generate the G Suite Reports access token. Click here to download it.
2. Move the script to <any_directory>/devo-collectors/gsuite-reports/credentials.
3. Check if quickstart.py and credentials-gsuite-reports.json are available on the same directory (<any_directory>/devo-collectors/gsuite-reports/credentials).

4. Run the command below to install the Google Auth API library on <any_directory>/devo-collectors/gsuite-reports/credentials

   $ pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib

5. Run the command below on <any_directory>/devo-collectors/gsuite-reports/credentials to create the token.pickle file. A Google consent window will prompt asking for permissions. Follow the instructions and allow the application.

   $ python quickstart.py

Run the collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure will be required as part of the setup procedure (it can be created under any directory):

<any_directory>
└── devo-collectors/
    └── gsuite-reports/
          ├── certs/
          │   ├── chain.crt
          │   ├── <your_domain>.key
          │   └── <your_domain>.crt
          ├── credentials/
          │   └── credentials-gsuite-reports.json
          │   └── token.pickle
          └── config/
                  └── config-gsuite-reports.yaml

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in /devo-collectors/certs. Learn more about security credentials in Devo here. 

Editing the config-gsuite-reports.yaml file

In the config-gsuite-reports.yaml file, replace the <short_unique_identifier> placeholder by the required value.

globals:
  debug: false                                                     # <- Setup as True or False for debugging mode
  id: not_used
  name: gsuite
  persistence:                                                     # <- Persistence setup filesystem
    type: filesystem
    config:
      directory_name: state                                        # <- Persistence directory
outputs:
  devo_1:
    type: devo_platform
    config:
      address: eu.elb.relay.logtrust.net                           # <- Devo platform address EU/US
      port: 443
      type: SSL
      chain: chain.crt
      cert: <your_domain>.crt                                      # <- Please, replace with the certificate from your Devo domain (Administration>Credentials>x.509)
      key: <your_domain>.key                                       # <- Please, replace with the certificate from your Devo domain (Administration>Credentials>x.509)
inputs:
  gsuite_reports:
    id: <short_unique_identifier>                                  # The value of this field will be used internally for having independent persistence areas
    enabled: false                                                 # <- As the G Suite reports shares the same config as G Suite alerts, in order to use the config only for alerts. Reports is disabled.
    requests_per_second: 5                                         # <- Setting up requests per second. 5 recommended.
    autoconfig:
      enabled: false                                              
      refresh_interval_in_seconds: 600                             # <- Time wait in second between requests - 600s recommended.
    credentials:
      token_pickle_filename: token.pickle                          # <- token.pickle file name
    services:                                                      # <- List with the Alerts that you want to collect
      # Request frequency values are in seconds
      # Default values recommended. 
      access_transparency:
        request_period_in_seconds: 60                              # <- Controls waiting time for to the next request
        start_time: "9999-12-31T23:59:59.999Z"
      admin:
        request_period_in_seconds: 60
        start_time: "9999-12-31T23:59:59.999Z"                     # <- Setting a start time for data collection. Format sample supportted "YYYY-mm-ddTHH:MM:SS.sssZ"
      calendar:
        request_period_in_seconds: 3600
        start_time: "9999-12-31T23:59:59.999Z"
      login:
        request_period_in_seconds: 3600
        start_time: "9999-12-31T23:59:59.999Z"
      chat:
        request_period_in_seconds: 3600
        start_time: "9999-12-31T23:59:59.999Z"
      drive:
        request_period_in_seconds: 60
        start_time: "9999-12-31T23:59:59.999Z"
      gcp:
        request_period_in_seconds: 3600
        start_time: "9999-12-31T23:59:59.999Z"
      gplus:
        request_period_in_seconds: 3600
        start_time: "9999-12-31T23:59:59.999Z"
      groups:
        request_period_in_seconds: 3600
        start_time: "9999-12-31T23:59:59.999Z"
      groups_enterprise:
        request_period_in_seconds: 3600
        start_time: "9999-12-31T23:59:59.999Z"
      jamboard:
        request_period_in_seconds: 3600
        start_time: "9999-12-31T23:59:59.999Z"
      meet:
        request_period_in_seconds: 60
        start_time: "9999-12-31T23:59:59.999Z"
      mobile:
        request_period_in_seconds: 3600
        start_time: "9999-12-31T23:59:59.999Z"
      rules:
        request_period_in_seconds: 3600
        start_time: "9999-12-31T23:59:59.999Z"
      saml:
        request_period_in_seconds: 3600
        start_time: "9999-12-31T23:59:59.999Z"
      token:
        request_period_in_seconds: 3600
        start_time: "9999-12-31T23:59:59.999Z"
      user_accounts:
        request_period_in_seconds: 3600
        start_time: "9999-12-31T23:59:59.999Z"

Download the Docker image

The collector should be deployed as a Docker container. Click here to download the Docker image of the collector as a .tgz file.

Use the following command to add the Docker image to the system:

gunzip -c collector-gsuite-docker-image.tgz | docker load

The Docker image can be deployed on the following services:

• Docker

• Docker Compose

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/gsuite-reports/

docker run \
--name collector-gsuite-reports \
--volume $PWD/certs:/devo-collector/certs \
--volume $PWD/config:/devo-collector/config \
--volume $PWD/state:/devo-collector/state \
--env CONFIG_FILE=config-gsuite-reports.yaml \
--rm -it docker.devo.internal/collector/gsuite:<version>

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/gsuite-reports/ directory.

version: '3'
services:
  collector-gsuite-reports:
    build:
      context: .
      dockerfile: Dockerfile
    image: docker.devo.internal/collector/gsuite:${IMAGE_VERSION:-latest}
    container_name: collector-gsuite-reports
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config-gsuite-reports.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/gsuite-reports/ directory:

IMAGE_VERSION=<version> docker-compose up -d

G Suite Report lag times

The lag times in this table show how long it can take before data for specific Admin console reports and audit logs are available.

Retrieving report or audit log data for older dates or a wide time range might take so long that, by the time results are available, the most recent log data might no longer be fresh. For tools that require real-time monitoring, use a short time range. Many products listed above (such as Gmail and Google Drive) are relevant for G Suite only, and not for other Google services, such as Cloud Identity.

Learn more about limits and quotas and lag times.

Activeboards

A number of predefined Activeboards that make use of the configured collectors can be downloaded here. Click here to download a preconfigured Activeboard that you can try in your Devo domain.

To instantiate them, follow these instructions:

1. Create a new Activeboard in your domain. Learn how to do it here.

2. In Edit mode, click the ellipsis button and select Edit raw configuration.

3. Open the downloaded file, select all the text, and copy it into the clipboard.

4. Paste the contents of the file in the raw editor. Make sure you replace the existing configuration completely.

5. Click Save changes. The Activeboard should show up immediately.

Disclaimer

The API limits the number of requests for your APIs Console project. The API project's maximum number of requests per second (project QPS) is 5 QPS and the maximum number of requests per day (project QPD) is 150,000 QPD across the account. If these limits are exceeded, the server returns an HTTP 503 status code.

You might find that your Admin console reports and audit logs don’t show the latest data, because reports don’t reflect real-time data. The lag times in the table below show how long it can take before data for specific Admin console reports and audit logs is available. Some reports might take longer to display updated information.