--- title: "Google Cloud Platform collector" canonical: "https://docs.devo.com/space/latest/162496513/Google%20Cloud%20Platform%20collector" format: markdown --- > Macro (excerpt-include) > Macro (toc) ## Purpose An analyst wants to detect malicious behavior in cloud resources. Using the [Google Cloud Platform](https://cloud.google.com) collector to send logs to Devo, the analyst will find unauthorized API calls that could indicate compromise. As a result, the analyst will revoke compromised credentials, preventing attackers from escalating privileges in cloud resources. ## Example tables | **[Table](https://devodocs.atlassian.net/wiki/spaces/latest/pages/94661194)** | **Description** | | --- | --- | | cloud.gcp.*.* | Google Cloud Platform logs | | cloud.gcp.scc.findings | Security Command Center findings | | casb.netskope.transaction_events | Netskope web transaction events | ## Configuration requirements To run this collector, there are some configurations detailed below that you need to take into account. | **Configuration** | **Details** | | --- | --- | | GCP console access | - You should have credentials to access the console. | | Permissions | - Administrator permissions to access the GCP console. | | Logging services | The following features have been configured:
- GCP project - Service account - GCP Pub/Sub - Sink (optional) | | Enable SCC | - **SCC Audit logs**: you used the `logging` service to pull the data source. - **SSC Comand Center**: you used `scc_findigs` service to pull the data source. | | Credentials | - JSON credentials have been filled or deleted. | ## Authorize it ### Enable the Logging service GCP centralizes all the monitoring information from all services in the cloud catalog that is inside the service named after **Logging**. > ⚠️ You have to use the `logging` service to pull this data source. Some information is enabled by default and free of charge. Other information, that in case of activating its generation, will concur some costs, so it must be enabled manually. In both cases, the generated information (*messages*) will arrive at the **Logging** service. ![image](media://79d36cd0-2893-4db6-8f5f-2c8d4c13d6a9) The **Logging** service has different ways of exporting the information stored and structured in messages. In this case, it’s being used by another GCP service called **PubSub**, basically, this service will contain a topic object that will receive a filtered set of messages from the **Logging** service, then the GCP collector will retrieve all those messages from the topic object using a subscription (in the *pull* mode). ![image](media://709b3150-701b-46cb-aeca-209272a29b75) To facilitate the retrieve is recommended to split the source message using different topic objects, you can split it by resource type, region, project ID, and so on: ![image](media://9aa736a6-267e-4b37-8e96-b11a94e47812) 1. In the Google Cloud console, search for **"IAM & Admin" **and click on it. 2. Click **"Create a Project"** if you don't have one already. 3. Fill in the project details and click **"Create"**. 4. Copy the **Project ID** for use in the collector configuration. Navigate to **"IAM & Admin -> Service Accounts"**. 5. Click **"+ Create Service Account"**. Enter a name and description for the service account and click **"Create and continue"**. 6. Add the **"Pub/Sub Subscriber"** role. If enabling undelivered messages logging, also add **"Monitoring Viewer"** role. 7. Click **"Continue"** and then **"Done"**. 8. Find your new service account in the list, click on the three dots menu, and select **"Manage keys"**. 9. Click **"Add Key -> Create new key"**. 10. Choose "JSON format" and click **"Create"**. 11. Save the downloaded **JSON credentials** file securely. 12. Search for "Pub/Sub" in the console. 13. Click **"Create topic"**, enter a *Topic ID*, and check "Add a default subscription". 14. Click **"Create"**. For optional steps like [restricting a service account to subscription](https://cloud.google.com/pubsub/docs/access-control) or [setting up a Sink](https://cloud.google.com/logging/docs/export/configure_export_v2) review [Access control IAM documentation](https://cloud.google.com/pubsub/docs/access-control). After configured the **Logging service**, follow the steps below to enable any of the supported events types: ### Enable the Security Command Center (SCC) Audit logs You have to use the `logging` service to pull this data source. 1. Activate **Security Command Center **service to directly receive this type of events in the default sink. 2. [Set up a new topic](https://www.youtube.com/watch?v=Q-fzz2P6omQ&list=PLIivdWyY5sqKd-Cu1HZ7v5RiYE8gVsM7P&index=4) (optional). 3. Set up a a **Pub/Sub** (optional). 4. Set up a sink (optional). ### Enable the Security Command Center (SCC) Findings These events are obtained from the **Security Command Center** service and are injected directly into the Pub/Sub without going through the Logging service. You have to use the `scc_findigs` service to pull this data source. 1. Configure** Identity and Access Management **(IAM) roles. 2. Activate the **Security Command Center** API. 3. Set up a **Pub/Sub** topic. 4. Create a Notification configuration. ### Enable Netskope Web Transactions 1. **Configure IAM Roles** to the Service Account, ensuring the necessary permissions for Pub/Sub access. 2. Enable the **Netskope API** in the Netskope admin console. 3. In the GCP console, go to **Pub/Sub**, create a new **Topic**, and ensure “**Add a default subscription**” is checked. 4. Once the Topic is created, navigate to **Subscriptions**, edit the subscription properties as needed, and “**Save”**. 5. Add the correct **Subscription ID** and **Service Account credentials (Base64 encoded)** to the collector configuration. For additional details, refer to [Netskope's official documentation](https://docs.netskope.com/). ## Run it In the Cloud Collector App, [create](https://devodocs.atlassian.net/wiki/spaces/latest/pages/409305131) a **GCP** collector instance. Insert this parameters template, replacing the values enclosed in `< >`. The `id` must be a unique [five digit number](https://www.random.org/integers/?num=1&min=10000&max=99999&col=5&base=10&format=html&rnd=new). ``` { "inputs": { "gcp": { "id": "", "enabled": true, "services": { "": {} }, "credentials": { "source_id": "", "project_id": "" } } } } ``` ## Secure it Devo [Exchang](https://docs.devo.com/space/latest/159285257/Alert+packs+Exchange)e provides these resources: - [Account Manipulation (MITRE Att&ck Technique: T1098)](https://docs.devo.com/space/latest/325222844/Mitre+alert+packs+T1000-1099#Account-Manipulation) - [Cloud Infrastructure Discovery (MITRE Att&ck Technique: T1580)](https://docs.devo.com/space/latest/326500411/Mitre+alert+packs+T1500-1599#Cloud-Infrastructure-Discovery) - [Create Account (MITRE Att&ck Technique: T1136)](https://docs.devo.com/space/latest/326172673/Mitre+alert+packs+T1100-1199#Create-Account) - [Google Cloud Platform Log-Based Threat Detection Suite](https://docs.devo.com/space/latest/395739171/Platform+alert+pack:+GCP) - [Exploitation for Defense Evasion (MITRE Att&ck Technique: T1211)](https://docs.devo.com/space/latest/326696976/Mitre+alert+packs+T1200-1299#Exploitation-for-Defense-Evasion) - [Valid Accounts (MITRE Att&ck Technique: T1078)](https://docs.devo.com/space/latest/325222844/Mitre+alert+packs+T1000-1099#Valid-Accounts) - [Transfer Data to Cloud Account (MITRE Att&ck Technique: T1537)](https://docs.devo.com/space/latest/326500411/Mitre+alert+packs+T1500-1599#Transfer-Data-to-Cloud-Account) ## Monitor it Create an [inactivity alert](https://devodocs.atlassian.net/wiki/spaces/latest/pages/95126785) to detect interruptions of transfer of data from the source using the query ```xquery from cloud.gcp where toktains(hostchain,"collector-") select split(hostchain,"-",1) as collector_id ``` Set the inactivity alert to keep track of the `collector_id`. ## Parameters ### Example configuration for Logging and SCC) ``` { "global_overrides": { "debug": false }, "inputs": { "gcp": { "id": "", "enabled": true, "credentials": { "source_id": "", "project_id": "", "file_content_base64": "" }, "services": { "logging": { "subscription_name": "" }, "scc_findings": { "subscription_name": "" } } } } } ``` ### Example configuration for Netskope Web Transactions ``` { "global_overrides": { "debug": false }, "inputs": { "gcp": { "id": "", "enabled": true, "credentials": { "project_id": "", "file_content_base64": "" }, "services": { "netskope-web-transaction": { "subscription_name": "" } } } } } ``` ### Details | Parameter | Data Type | Necessity | Description | | --- | --- | --- | --- | | `id` | `string` | Mandatory | Five unique digits. | | `debug` | `string` | Discouraged | Do not enabled debug | | `enabled` | `boolean` | Mandatory | Indicates whether the collector is active. | | `source_id` | `string` | Mandatory | Custom identifier for infrastructure environment. | | `project_id` | `string` | Mandatory | The name of the GCP project. | | `file_content_base64` | `string` | Optional | Service Account key required for authentication. | | `subscription_name` | `string` | Mandatory | The ID of the Pub/Sub subscription. | | `` | `string` | Mandatory | Specifies the service being collected. `logging` and `scc_findings` can have multiple instances (e.g., `logging_0`, `logging_1`), while `netskope-web-transaction` must be configured separately. | ## Troubleshooting This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors. | **Error Type** | **Error ID** | **Error Message** | **Cause** | **Solution** | | --- | --- | --- | --- | --- | | InitVariablesError | 1 | `"file_content_base64" must be in a valid base64 format` | The `file_content_base64` parameter is not a valid base64 string. | Ensure that the content of `file_content_base64` is correctly encoded in base64. | | InitVariablesError | 2 | `"filename" property or file content base64 is mandatory in the configuration.` | Neither `filename` nor `file_content_base64` is provided. | Provide either `filename` or `file_content_base64` in the credentials section. | | InitVariablesError | 3 | `Credentials filename is empty or blank` | The `filename` parameter is missing or blank. | Ensure `filename` is correctly provided in the configuration. | | InitVariablesError | 11 | `Credentials file must be "service_account" type.` | The credentials file does not belong to a valid service account. | Use a valid service account credentials JSON file. | | InitVariablesError | 12 | `Credentials file does not exist: {ex}` | The specified credentials file is missing. | Verify the file path and ensure the credentials file exists. | | InitVariablesError | 13 | `Invalid credentials file format: {ex}` | The credentials file is not formatted correctly. | Ensure the credentials file is a valid JSON with the correct structure. | | InitVariablesError | 20 | `The property "start_time_utc" from configuration cannot be in the future.` | The provided `start_time_utc` is a future date. | Ensure `start_time_utc` is set to a past or present timestamp. | | SetupError | 101 | `Credentials info is missing from collector variables.` | The credentials information is not provided in the collector configuration. | Ensure that the `credentials_info` is properly defined in the collector configuration. | | ApiError | 401 | `Failed to load credentials: {str(ex)}` | The credentials provided are invalid or could not be loaded. | Verify the credentials JSON file and ensure it contains the correct authentication details. | | ApiError | 403 | `Forbidden (403): Access is forbidden, check permissions.` | The service account lacks the necessary permissions. | Ensure the service account has sufficient IAM permissions for Pub/Sub access. | | PullError | 301 | `An error occurred creating the subscription: {ex}` | The Pub/Sub subscription could not be created. | Verify that the subscription exists and that the service account has the required roles. | | PullError | 302 | `An error occurred during execution of the streaming pull: {ex}` | An issue occurred while retrieving messages from Pub/Sub. | Check network connectivity, credentials, and Pub/Sub subscription settings. | | PullError | 313 | `Error during _request_job_creation. {e.cause}` | A failure occurred while creating a job in the system. | Investigate the job creation error and refer to the API documentation for resolution. | ### Setting the target pods Usually this collector uses one pod, but multipod operation is supported. If events are entering the Pub/Sub faster than they are being collected, increase the number of pods. ### Lookups This collector does not create lookups. To convert data from the collector to a lookup, [create a lookup from a query](https://devodocs.atlassian.net/wiki/spaces/latest/pages/95204060). ## API Endpoints | Data Source | Endpoint | Service | Table | Available from release | | --- | --- | --- | --- | --- | | Logging (formerly StackDriver) | `pub/sub queue ` | `logging` | `cloud.gcp..`
`cloud.gcp..` | `v1.0.20 ` | | Security Command Center Findings | `pub/sub queue` | `scc_findings` | `cloud.gcp.scc.findings` | `v1.1.4 ` | | Netskope Web Transactions | `pub/sub lite ` | `netskope-web-transaction` | `casb.netskope.transaction_events ` | `v2.0.0 ` | ## Devo collector features | **Feature** | **Details** | | --- | --- | | Allow parallel downloading (`multipod`) | - `Allowed` | | Running environments | - `Collector server` | | Populated Devo events | - `Table` | ## Release Notes - 2.4.0 2025-06-26 - Updated DCSDK to 1.16.3. - Fixed single character ingestion issue. - 2.3.0 2025-05-05 - Updated DCSDK to 1.15.0. - 2.2.0 2025-03-05 - Added logging filter to handle ValueErrors related to closed RPC channels. - 2.1.0 2025-02-24 - Improved handling of string messages for valid JSON. - 2.0.0 2025-02-12 - Complete restructuring as a generic Pub/Sub data ingestion collector. - Architecture optimized for streaming data ingestion.