--- title: "Proofpoint on Demand collector" canonical: "https://docs.devo.com/space/latest/580845578/Proofpoint%20on%20Demand%20collector" format: markdown --- > Macro (excerpt-include) > Macro (toc) ## Purpose An analyst wants to detect malicious emails in [Proofpoint On Demand](https://www.proofpoint.com/us/products/email-protection). Using the Proofpoint POD collector to send email logs to Devo, the analyst will find suspicious attachment files and malicious URLs in messages. As a result, the analyst will quarantine affected emails, preventing sensitive data from being accessed by unauthorized users. ## Example tables | **Table** | **Description** | | --- | --- | | mail.proofpoint.pod.message | Email message data and threat information | | mail.proofpoint.pod.maillog | Detailed mail logs with delivery status and traffic patterns | ## Authorize it 1. Log in to your Proofpoint On Demand [admin console](https://admin.proofpoint.com/auth). ![30_Proofpoint On Demand collector.png](media://b4dc419a-a66a-4c78-b4f5-cdb7d24dfd9e) 2. Navigate to the **API settings** section in your admin menu. ![231ac9e1-205c-447d-9efa-e573623a923d.png](media://983ab0dc-cda8-443b-9e4d-5fba0a4ad10f) 3. Generate a **new API key** for the Log API. 4. Copy your **cluster ID** and the **API key**. ![ed53c09b-d50f-48db-a1b3-ec85574b1cae.png](media://7ed9fef6-8f89-4514-b2d0-010875511393) 5. Ensure that your API key has **read access permissions** to message and mail log data. ## Run it In the Cloud Collector App, [create](https://devodocs.atlassian.net/wiki/spaces/latest/pages/409305131) a **Proofpoint On Demand (PoD)** collector instance. Insert this parameters template, replacing the values enclosed in `< >`. The `id` must be a unique [five digit number](https://www.random.org/integers/?num=1&min=10000&max=99999&col=5&base=10&format=html&rnd=new). ``` { "inputs": { "proofpoint_on_demand": { "id": "", "enabled": true, "credentials": { "cluster_id": "", "api_key": "" }, "environment": "prod", "services": { "message": { "request_period_in_seconds": "", "start_time": "" }, "maillog": { "request_period_in_seconds": "", "start_time": "" } } } } } ``` ## Secure it ### Threat #### Detecting Suspicious Email Attachments Flags emails containing attachments with risky MIME types, helping you spot potential malware delivery attempts. This query identifies emails with potentially dangerous attachments (e.g., executable files) in the `mail.proofpoint.pod.message` table: ``` from mail.proofpoint.pod.message where lower(msgParts__labeledMime_str) like "%exe%" or lower(msgParts__labeledMime_str) like "%zip%" or lower(msgParts__labeledMime_str) like "%js%" select eventdate, connection__ip, filter__suborgs__sender, filter__suborgs__rcpts, msgParts__labeledMime_str ``` #### Detecting Bulk Email Attacks Alerts you to IP addresses sending unusually large volumes of email, which could be a sign of compromise or abuse. This query finds sender IPs that have sent a high volume of emails in a short period, which may indicate a spam or phishing campaign: ``` from mail.proofpoint.pod.maillog group by connection__ip having count() > 100 in 1h select connection__ip, count() as email_count ``` ## Monitor it Create an [inactivity alert](https://devodocs.atlassian.net/wiki/spaces/latest/pages/95126785) to detect interruptions of transfer of data from the source using the query ``` from mail.proofpoint.pod.message where toktains(hostchain,"collector-") select split(hostchain,"-",1) as collector_id ``` Set the inactivity alert to keep track of the `collector_id`. ## Parameters ### Example ``` { "global_overrides": { "debug": false }, "inputs": { "proofpoint_on_demand": { "id": "", "enabled": true, "credentials": { "cluster_id": "", "api_key": "" }, "environment": "prod", "services": { "message": { "request_period_in_seconds": "", "start_time": "", "override_tag_base": "", "override_url_base": "" }, "maillog": { "request_period_in_seconds": "", "start_time": "", "override_tag_base": "", "override_url_base": "" } } } } } ``` ### Details | **Parameter** | **Data type** | **Necessity** | **Details** | | --- | --- | --- | --- | | `input_id` | integer | Mandatory | Use this param to give an **unique** id to this input service. | | `enabled` | boolean | Mandatory | If the value is true, the input definition will be executed. If the value is false, the service will be ignored. | | `cluster_id` | string | Mandatory | Cluster id to get data from. | | `api_key` | string | Mandatory | API key to authenticate to the service. | | `request_period_in_seconds` | integer | Mandatory | It allows you to customize this behavior for each service. | | `start_time` | integer | Mandatory | Initial time period used when fetching data from the endpoint.
Due to the large amount of data produced by this service, using this parameter is discouraged except in special cases.
This parameter can be left blank, removed or commented.
Following [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339): `%Y-%m-%dT%H:%M:%S.%f%z`
Example: `2024-04-04T05:50:00.000-0500` | | `override_tag_base` | string | Optional | It allows to override the destination table in Devo.
Example: `mail.proofpoint.pod.maillog` | | `override_url_base` | string | Optional | It allows to change the connection URL to the websocket | ## Troubleshooting ### 1. Invalid or Missing Configuration **Symptom:** Collector fails to start or logs an error about configuration. **Possible Causes:** - Required parameters (such as `cluster_id`, `api_key`, or `start_time`) are missing or incorrectly formatted. - The `start_time` is not in the correct UTC format (e.g., `2024-07-08T00:00:00Z`). **Solution:** - Double-check your configuration file. All required fields must be present and correctly formatted. - Ensure all values are strings in quotes. - Example for `start_time`: `"start_time": "2024-07-08T00:00:00Z"` ### 2. Authentication Errors **Symptom:** Collector logs show authentication failures or messages such as "Potential issue with authentication. Please check credentials and detailed error message: {error_message}". **Possible Causes:** - The `api_key` or `cluster_id` is incorrect or expired. **Solution:** - Verify your Proofpoint credentials. - Generate a new API key if necessary and update your configuration. ### 3. Data Not Ingesting or Delays **Symptom:** No data appears in Devo tables, or there are significant delays. **Possible Causes:** - Network/firewall issues blocking outbound WSS connections to Proofpoint. - The collector is running an outdated version. - The collector is not able to keep up with the event volume. **Solution:** - Ensure your network allows outbound connections to the Proofpoint WebSocket endpoints. - Upgrade to the latest collector version. - Check system resources (CPU, memory) and consider scaling if needed. ### 4. Date/Time Parameter Errors **Symptom:** Error messages such as `Invalid type {type}. Must be epoch seconds, epoch millis, str, datetime, or DateTime`. **Possible Causes:** - The `start_time` or other date parameters are not in an accepted format. **Solution:** - Use one of the supported formats (e.g., ISO 8601 UTC string: `2024-07-08T00:00:00Z`). ### 5. Internal Collector Errors **Symptom:** Collector logs show errors related to internal client initialization or unexpected exceptions. **Possible Causes:** - Internal bug or misconfiguration. **Solution:** - Review the error message for details. - If the issue persists, contact [Devo Support](https://devo.my.site.com/support/login?ec=302&startURL=%2Fsupport%2Fs%2F) with the error log. ### 6. Upgrading/Downgrading Collector **Symptom:** Issues after upgrading or downgrading the collector. **Solution:** - Always stop the collector before upgrading or downgrading. - Review release notes for any changes in persistence or configuration ## API Endpoints The endpoint: ``` wss://logstream.proofpoint.com/v1/stream?cid={cluster_id}&type={type}&sinceTime={since_time}&toTime={to_time} ``` - The data type is controlled by the `type` parameter (e.g., `message` or `maillog`). - The `sinceTime` and `toTime` parameters allow you to specify a time window for the events you want to retrieve. - The `cid` parameter is your assigned Proofpoint cluster ID. | **Endpoint** | **Service** | **Table** | | --- | --- | --- | | `wss://.proofpoint.com/ws/message` | Message | mail.proofpoint.pod.message | | `wss://.proofpoint.com/ws/maillog` | Mail log | mail.proofpoint.pod.maillog | > ℹ️ Both endpoints use secure WebSocket (WSS) connections. ### API limits and duplicates #### Number of connections The number of connections that can be performed with one credential set is limited. The credentials cannot pull data from more sources than the defined ones. If the access token is being used by another session, the API will return a 409 error: *Exceeded maximum number of sessions per token.* The collector logs will show this error: ``` 2024-08-09T10:48:10.163 ERROR InputProcess::ProofpointOnDemandWSSPuller(proofpoint_on_demand,12345,message,predefined) -> Handshake status 409 Conflict -+-+- {'date': 'Fri, 09 Aug 2024 08:48:10 GMT', 'content-type': 'text/plain;charset=iso-8859-1', 'content-length': '47'} -+-+- b'Exceeded maximum number of sessions per token\r\n' - goodbye ``` #### Duplicated events It was observed that the API sometimes sends duplicate events. The collector can filter out duplicate events within an hour. ## Devo collector features | **Feature** | **Details** | | --- | --- | | Allow parallel downloading (`multipod`) | `not allowed` | | Running environments | `collector server` | ## Release Notes - **Version 1.2.2** (2024-10-07) - Upgraded DCSDK from v1.12.2 to v1.12.4, improved controlled stop when InputProcess is killed, changed internal queue management to protect against OOMK, improved controlled stop when both processes fail to instantiate, fixed various errors. - **Version 1.2.1 **(2024-08-08) - Downgraded DCSDK from v1.12.3 to v1.12.2. - **Version 1.2.0 **(2024-08-07) - Upgraded DCSDK from v1.12.2 to v1.12.3 - Reduced in-memory cached data to avoid memory issues. - **Version 1.1.0** (2024-08-06) - Added new parameters, reduced memory usage, optimized performance, fixed high CPU usage and improved duplicate filtering. - **Version 1.0.1 **(2024-04-08) - Added reset mechanism for stats counters to avoid growing them indefinitely. - **Version 1.0.0** (2024-04-05) - Initial version with message and maillog services.