--- title: "Proofpoint Tap collector" canonical: "https://docs.devo.com/space/latest/457637902/Proofpoint%20Tap%20collector" format: markdown --- > Macro (excerpt-include) > Macro (toc) ## Purpose An analyst wants to detect email-based threats targeting their organization. Using the [Proofpoint Targeted Attack Protection](https://www.proofpoint.com/us/products/advanced-email-security/targeted-attack-protection) collector to send email security logs to Devo, the analyst will find malicious URLs in emails. As a result, the analyst will block the malicious domains, preventing users from accessing phishing sites. ## Example tables | **Table** | **Description** | | --- | --- | | `mail.proofpoint.tapsiem_v2.clicksblocked` | Events for clicks to malicious URLs blocked | | `mail.proofpoint.tapsiem_v2.clickspermitted` | Events for clicks to malicious URLs permitted | | `mail.proofpoint.tapsiem_v2.messagesblocked` | Events for messages blocked that contained threats | | `mail.proofpoint.tapsiem_v2.messagesdelivered` | Events for messages delivered which contained threats | | `mail.proofpoint.tap_threats.events` | Detailed attributes about individual threats | | `mail.proofpoint.tap_campaigns.events` | Details about threat campaigns | | `mail.proofpoint.tap_people.top_clicks` | Users with a tendency to click on malicious URLs | | `mail.proofpoint.tap_people.vap` | Very Attacked People within the organization | | `tap.proofpoint.isolation.browser` | TAP isolation browser events | | `tap.proofpoint.isolation.browser_and_email` | TAP isolation browser and email events | | `tap.proofpoint.isolation.url` | TAP isolation URL events | | Union tables that uses these tags: `mail.all.events` | Union tables collect information for monitoring purposes. | ## Authorize it Ensure your TAP user has the **Unrestricted Devices** and **Visualizer role**. 1. Log in to the Proofpoint TAP dashboard. 2. Click the **“Settings”** tab. 3. Click **“Connected Applications”** on the left side of the screen. 4. In the Service credentials section, select **“Create New Credential” **in the Name section. 5. Type a name and click **“Generate”**. 6. In the Generated Service Credential pop-up, copy the **Service Principal** and **Secret values**. ## Run it In the Cloud Collector App, [create](https://devodocs.atlassian.net/wiki/spaces/latest/pages/409305131) a **Proofpoint TAP** collector instance. Insert this parameters template, replacing the values enclosed in `< >`. The `id` must be a unique [five digit number](https://www.random.org/integers/?num=1&min=10000&max=99999&col=5&base=10&format=html&rnd=new). ``` "inputs": { "proofpoint_tap": { "id": "", "enabled": true, "autoconfig": { "refresh_interval_in_seconds": "", "creation_timeout_in_second": "" }, "services": { "clicksBlocked": { "start_time_in_utc_format": "" }, "credentials": { "username": "", "password": "" } } } } ``` ## Secure it Devo [Exchange](https://devodocs.atlassian.net/wiki/spaces/latest/pages/158236705) provides these resources, which will help secure Proofpoint TAP: - *[Activeboard: Proofpoint email protection](https://devodocs.atlassian.net/wiki/spaces/latest/pages/704774145)* ## Monitor it Create an [inactivity alert](https://devodocs.atlassian.net/wiki/spaces/latest/pages/95126785) to detect interruptions of transfer of data from the source using the query ``` from mail.proofpoint.tapsiem_v2.clicksblocked where toktains(hostchain,"collector-") select split(hostchain,"-",1) as collector_id ``` Set the inactivity alert to keep track of the `collector_id`. ## Parameters ### Example ``` { "global_overrides": { "debug": false }, "inputs": { "proofpoint_tap": { "id": "", "enabled": true, "autoconfig": { "refresh_interval_in_seconds": "", "creation_timeout_in_second: "" }, "credentials": { "username": "", "password": "" }, "services": { "clicksBlocked": { "start_time_in_utc_format": "" }, "clicksPermitted": { "start_time_in_utc_format": "" }, "messagesBlocked": { "start_time_in_utc_format": "" }, "messagesDelivered": { "start_time_in_utc_format": "" }, "campaigns": { "start_time_in_utc_format": "", "forensics": "" }, "threats": { "start_time_in_utc_format": "", "forensics": "", "include_campaign_forensics": "" }, "people_topclicks": { "start_time_in_utc_format": "" }, "people_vap": { "start_time_in_utc_format": "" } } } } } ``` ### Details | **Parameter** | **Data type** | **Necessity** | **Details** | | --- | --- | --- | --- | | `id` | integer | Mandatory | Use this param to give a **unique** id to this input service. | | `inputs` | boolean | Mandatory | Use this param to enable or disable the given input logic when running the collector. If the value is `true`, the input will be run. If the value is `false`, it will be ignored. | | `refresh_interval_in_seconds` | integer | Mandatory | Use this param to set refresh interval for authentication, collector will check if auth is valid every time after mentioned seconds. Recommended value is 3600. | | `creation_timeout_in_second` | integer | Mandatory | Use this param to set puller creation timeout. Collector will throw error if Puller is not created in mentioned seconds. Recommended value is 3600. | | `username` | string | Mandatory | Provide the value of username used for authentication | | `password` | string | Mandatory | Provide the value of password used for authentication | | `start_time_in_utc_format` | string | Mandatory | This configuration allows you to set a custom date as the beginning of the period to download. It can’t be more than 7 days in the past. Format: `%Y-%m-%dT%H:%M:%SZ` | | `request_period_in_seconds` | integer | Optional | This configuration allows us to set the “seconds” after which next pull cycle to be made. Allowed limit of number of api requests is 1800 per day so we have kept 600 seconds as a default value. | | `override_tag` | string | Optional | This parameter allows to define a custom devo tag. | ## Troubleshooting This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors. | **Error type** | **Error ID** | **Error message** | **Cause** | **Solution** | | --- | --- | --- | --- | --- | | SetupError | 100 | `HTTP Error occurred while retrieving events from Proof point server` | username and password is not correct | Make sure that credentials are correct. | | 101 | `Error occurred while retrieving events from ProofPoint server` | start_time_in_utc is in future or not in proper format | Make sure the start time is not in future and not in proper format | | PullError | 300 | `HTTP Error occurred while retrieving events from Proof point server : {summery} , {details}` | This error happens when the collector tries to fetch the data from API. | In this error you will find the HTTP error code as well as the summary and details. | | | 301 | `Some Error occurred while retrieving events from ProofPoint server : {Exception}` | Some exception occurred while making the API request. | Reach out to the developer with the exact error message. | ## API Endpoints | **Endpoint** | **Service** | **Table** | | --- | --- | --- | | `/v2/siem/clicks/blocked` | `clicksBlocked` | `mail.proofpoint.tapsiem_v2.clicksblocked` | | `/v2/siem/clicks/permitted` | `clicksPermitted` | `mail.proofpoint.tapsiem_v2.clickspermitted` | | `/v2/siem/messages/blocked` | `messagesBlocked` | `mail.proofpoint.tapsiem_v2.messagesblocked` | | `/v2/siem/messages/delivered` | `messagesDelivered` | `mail.proofpoint.tapsiem_v2.messagesdelivered` | | `siem/all then -> v2/threat/summary/` | `threats` | `mail.proofpoint.tap_threats.events` | | `v2/threat/summary/ then -> v2/threat/summary/&includecampaignforensics=true` | `threats` | `mail.proofpoint.tap_threats.events` | | `v2/campaign/` | `campaigns` | `mail.proofpoint.tap_campaigns.events` | | `v2/forensics?campaignId=` | `campaigns` | `mail.proofpoint.tap_campaigns.events` | | `v2/people/top-clickers` | `people_topclicks` | `mail.proofpoint.tap_people.top_clicks` | | `v2/people/vap` | `people_vap` | `mail.proofpoint.tap_people.vap` | ### API Limits, issues #### for services `clicksBlocked`, `clicksPermitted`, `messageBlocked`, `messageDelivered` - `ClicksBlocked` (`/v2/siem/clicks/blocked`) ,` messageBlocked` (`/v2/siem/messages/blocked`), `messageDelivered `(`/v2/siem/messages/delivered`) can collectively make 1800 requests per day as per[ api documentation](https://help.proofpoint.com/Threat_Insight_Dashboard/API_Documentation/SIEM_API) - `clicksPermitted` (`/v2/siem/clicks/permitted`) can make 1800 reuqests per day as per [api documentation.](https://help.proofpoint.com/Threat_Insight_Dashboard/API_Documentation/SIEM_API) - We have put a **limit **on the api call,** **which will block the api requests after 400 requests for `ClicksBlocked`, `messageBlocked`and `messageDelivered` services. - For `clicksPermitted` service, we can make 1700 requests, to take extra precaution and avoid 429 error. - If the collector took, let’s say, 15 hours to make 400 requests, the collector will block the api call for 9 hours (9 + 15 = 24 hours) and it will resume making api requests after 9 hours. For these 9 hours no ingestion will occur for that service. - **Issues**: - Let’s say all the services make 1000 request in 15 hours and then due to some issue collector restarts. Now, after the restart collector is going to assume that service still make a 1800 request , ignoring the 1000 requests collector made before the restart. This will lead to surpass the 1800 requets per day api limit causing a 429 error. #### for services `threats`, `campaigns`, `people_topclicks`, `people_vap` - `threats` service has no api limit - `/v2/forensics?threatId=&includecampaignforensics=true` can make only 1800 request per day. Collector fetches from this endpoint when forensics paramter for is true for threat service. - `people_topclicks` (`v2/people/top-clickers`) and `people_vap` (`v2/people/vap`) services can make only 50 requests in 1 day as per the [api documentation](https://help.proofpoint.com/Threat_Insight_Dashboard/API_Documentation/People_API) - `campaigns` (`v2/campaign/ids`) service can only make 50 request in 1 day as per the [api documentation](https://help.proofpoint.com/Threat_Insight_Dashboard/API_Documentation/Campaign_API) - We have put a **limit **on the api call,** **which will block the api requests after 50 requests for a service - If the collector took , let’s say, 10 hours to make 50 request, the collector will block the api call for 14 hours (10 + 14 = 24 hours) and it will resume making api requests after 14 hours. For these 14 hours no ingestion will occur for that service - **Issues**: - Let’s say a service make 30 request in 3 hours and then due to some issue collector restarts. Now, after the restart collector is going to assume that service still make a 50 request , ignoring the 30 requests collector made before the restart. This will lead to surpass the 50 requets per day api limit causing a 429 error for that service. ### **Ideal **`request_period_in_seconds`** for each service to avoid 429 error** - `clicksBlocked` - 180 - `messageBlocked` - 180 - `messageDelivered` - 180 - `clicksPermitted`- 120 - `threats`- - `/v2/forensics?threatId=&includecampaignforensics=true` - 120 - `people_topclicks` - 3600 - `people_vap` - 3600 - `campaigns` - 3600 # Devo Collector Features | **Feature** | **Details** | | --- | --- | | Allow parallel downloading (`multipod`) | `not allowed` | | Running environments | `collector server` | | Populated Devo events | `table` | | Flattening preprocessing | `no` | ## Release Notes - **Version 3.3.0** (2025-04-16) - Improvements of the request limit for every service and optimized the pull and flatten logic of the threat service. - **Version 3.2.0 **(2025-03-21) - Refactor code and upgraded DCSDK to 1.15.0 - Differentiated error codes for SdkPersistenceServiceError. - Use of DOCKER_IMAGE environment variable to show docker_image property. - Now the property service_thread_execution_periods_in_seconds in collector_definition.yaml is optional. - Reduced Redis connections per collector to 2. - Implemented a mechanism to control if the certificates have expired. - Fixed CVE-2024-12797 vulnerability in cryptography library (updated from version 44.0.0 to 44.0.1). - Upgraded docker base image to 1.4.1 - Fixed the OOMK bug causing the collector to restart - **Version 3.1.1** (2024-10-29) - Fixed a bug related to request_time_in_seconds error for new endpoints - **Version 3.1.0** (2024-10-24) - Upgraded the DCSDK to v1.13.1 - Resolved the bug related to rate limit and repeated logs - **Version 3.0.0** (2024-08-26) - Upgraded the DCSDK from 1.11.0 to 1.12.4 - Updated the base Docker Image to 1.3.0 - Added new services (threats, campaigns, people_topclicks, people_vap) and new pullers for these services. - **Version 2.2.1 **(2024-02-23) - Upgraded the DCSDK from 1.10.3 to 1.11.0 - **Version 2.2.0** (2024-01-16) - Upgraded the DCSDK docker base image version to 1.1.0 and fixed collector unable to fetch data after a certain time period. - **Version 2.1.1 **(2023-12-21) - Released the first version of the Proofpoint TAP collector. ## Version migration > Macro (children)