---
title: "Rapid7 IntSights collector"
canonical: "https://docs.devo.com/space/latest/94657099/Rapid7%20IntSights%20collector"
format: markdown
---
> Macro (toc)

## Service description

The **Rapid7 IntSights** collector ingests threat indicators from the Insights Threat Intelligence Platform (TIP) as Devo lookup tables. This allows the use of the indicators as a correlation source when using the Devo platform to analyze security data from other systems as part of reactive alerting and proactive threat hunting.

IntSights (a Rapid 7 company) is a security company specialized in Endpoint Security and threat detection. IntSights provides cloud-native external threat detection to further extend Rapid7’s security operations platform, providing customers with end-to-end external and internal threat detection, automation, and remediation.

The information items that IntSights TIP provides are the Indicators of Compromise (IoC). Using the API, the collector extracts the IoC from IntSights and stores them in the Devo system as lookup tables, there are 5 types of IoC provided by IntSights: IP Address, DNS Domains, File Hashes, URLs, and Emails.

## Data source description

| **Data source** | **Lookup** | **Collector service** | **Remote endpoint** | **Description** |
| --- | --- | --- | --- | --- |
| IP address | *IntSights_IP_Address_IoC_List* | iocs_list_ips | `https://api.intsights.com:443/public/v3/iocs?type[0]=IpAddresses` | IoC related to IP Address, stored using the IP as the primary key of the lookup |
| Domains | *IntSights_Domain_IoC_List* | iocs_list_domains | `https://api.intsights.com:443/public/v3/iocs?type[0]=Domains` | IoC related to Domains, stored using the DNS domain as the primary key of the lookup |
| File hashes | *IntSights_Hash_IoC_List* | iocs_list_hashes | `https://api.intsights.com:443/public/v3/iocs?type[0]=Hashes` | IoC related to File Hashes, stored using the hash value as the primary key of the lookup |
| URLs | *IntSights_URL_IoC_List* | iocs_list_urls | `https://api.intsights.com:443/public/v3/iocs?type[0]=Urls` | IoC related to URLs, stored using the URL as the primary key of the lookup |
| Email address | *IntSights_Email_IoC_List* | iocs_list_emails | `https://api.intsights.com:443/public/v3/iocs?type[0]=Emails` | IoC related to Email Addresses, stored using the email as the primary key of the lookup |

## Vendor setup

In order to configure the connection to IntSights, you need to generate a `client_id` and an `api_key`. Here are the steps to generate the credentials:

| **Action** | **Steps** |
| --- | --- |
| Log in to the Rapid7 console: [IntSights - Authentication](https://dashboard.ti.insight.rapid7.com/) | 1. Log in to the Rapid7 IntSights console with your user credentials. |
| Generate API key | 1. Click the **Settings** wheel icon in the left menu.
2. Click the **Subscription** button.
3. Click **Generate API Key**. If it already exists and you don't remember the API Key and it is not being used anywhere else, you can revoke it and create a new one.
4. Copy the **Account ID** (parameter `client_id` in collector) and the **API Key** (`api_key` in collector). |
| Check permissions | 1. Click the **Settings** wheel icon in the left menu.
2. Click the **Users** button. You will see the administration page.
3. Check that the permissions are correct. |

## Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (<u>Cloud collector</u>), or deploy and host the collector in your own machine using a Docker image (<u>On-premise collector</u>).

> Macro (rw-ui-tabs-macro)
> 
> > Macro (rw-tab)
> 
> We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, [get in touch with us](https://devo.my.site.com/support/login?ec=302&startURL=%2Fsupport%2Fs%2F) and we will guide you through the configuration.
> 
> > Macro (rw-tab)
> 
> This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.
> 
> ### Structure
> 
> The following directory structure should be created for being used when running the IntSights collector:
> 
> ```
> <any_directory>
> └── devo-collectors/
>     └── devo-collector-intsights/
>           ├── certs/
>           │ ├── chain.crt
>           │ ├── <your_domain>.key
>           │ └── <your_domain>.crt
>           └── config/
>               └── config-intsight.yaml
> ```
> 
> ### Devo credentials
> 
> In Devo, go to **Administration → Credentials → X.509 Certificates**, download the **Certificate**, **Private key** and **Chain CA** and save them in `devo-collector-intsights/certs/`. Learn more about security credentials in Devo [here](https://devodocs.atlassian.net/wiki/spaces/latest/pages/94763701).
> 
> ![image](media://ea71368c-9c39-4a78-92f8-34c7c828c4f2)
> 
> ### Editing the config.yaml file
> 
> ```
> globals:
>   debug: <debug_value>
>   id: not_used
>   name: <name_collector>
>   persistence:
>     type: filesystem
>     config:
>       directory_name: state
> 
> outputs:
>   devo_1:
>     type: devo_platform
>     config:
>       address: <devo_address>
>       port: 443
>       type: SSL
>       chain: <chain_filename.crt>
>       cert: <cert_filename.crt>
>       key: <key_filename.key>
> 
> inputs:
>   intsights:
>     id: <input_id>
>     enabled: <input_status>
>     credentials:
>       client_id: <client_id>
>       api_key: <key_id>
>     base_url: <base_url>
>     services:
>       iocs_list_ips:
>         request_period_in_seconds: <request_period_in_seconds_ips> # Optional
>         days_to_look_back: <days_to_look_back_ips>
>       iocs_list_domains:
>         request_period_in_seconds: <request_period_in_seconds_domains> # Optional
>         days_to_look_back: <days_to_look_back_domains>
>       iocs_list_hashes:
>         request_period_in_seconds: <request_period_in_seconds_hashes> # Optional
>         days_to_look_back: <days_to_look_back_hashes>
>       iocs_list_urls:
>         request_period_in_seconds: <request_period_in_seconds_urls> # Optional
>         days_to_look_back: <days_to_look_back_urls>
>       iocs_list_emails:
>         request_period_in_seconds: <request_period_in_seconds_emails> # Optional
>         days_to_look_back: <days_to_look_back_emails>
> ```
> 
> Replace the placeholders with your corresponding values:
> 
> | **Parameter** | **Data type** | **Type** | **Value range** | **Details** |
> | --- | --- | --- | --- | --- |
> | `id_collector` | int | Mandatory | Minimum length: 1  
> Maximum length: 5 | Use this param to give a unique ID to this collector (not used) |
> | `name_collector` | str | Mandatory | Minimum length: 1  
> Maximum length: 10 | Use this param to give a valid name to this collector. |
> | `debug_value` | bool | Mandatory | false / true | Use this param to enable or disable the debug logging level. |
> | `devo_address` | str | Mandatory | - [collector-us.devo.io](http://collector-us.devo.io/)
> - [collector-eu.devo.io](http://collector-eu.devo.io/) | Use this param to identify the Devo Cloud where the events will be sent. |
> | `chain_filename` | str | Mandatory | Minimum length: 4  
> Maximum length: 20 | Use this param to identify the chain.cert  file downloaded from your Devo domain. Usually this file's name is: chain.crt |
> | `cert_filename` | str | Mandatory | Minimum length: 4  
> Maximum length: 20 | Use this param to identify the file.cert downloaded from your Devo domain. |
> | `key_filename` | str | Mandatory | Minimum length: 4  
> Maximum length: 20 | Use this param to identify the file.key downloaded from your Devo domain. |
> | `input_id` | int | Mandatory | Minimum length: 1  
> Maximum length: 5 | Use this param to give an unique id to this input service, (for instance 147 or 456) |
> | `input_status` | bool | Mandatory | false / true | If the value is true, the input definition will be executed. If the value is false, the service will be ignored. |
> | `client_id` | str | Mandatory | Minimum length: 1 | This is the client_id given by Rapid7 IntSights |
> | `api_key` | str | Mandatory | Minimum length: 1 | This is the api_key given by Rapid7 IntSights |
> | `base_url` | str | Mandatory | Regex: `^(https?:\/\/)?([\da-z\.-]+)\.([a-z\.])([\/\w \.-]*)*([a-z])$` | The URL of endpoint for IntSights API access. This value is given by Rapid7 console.<br>A common value for base_url is usually [https://api.intsights.com](https://api.intsights.com/) |
> | `request_period_in_seconds` | int | Mandatory | Minimum value: 1800 | Period in seconds used between each IoC pulling related data source.<br>Use the following periods as reference:<br>- Run every 30 min → 1800 sec
> - Run every 1 hour → 3600 sec
> - Run every 6 hours → 21600 sec
> - Run every 12 hours → 43200 sec
> - Run every 24 hours → 86400 sec<br>The IoC list changes very slowly, so it does not make sense to request it very frequently, in many installations it is enough to request it a few times a day. As the list is long, in any case a small request period (less than some minutes) is not advised. |
> | `days_to_look_back` | int | Mandatory | Minimum value: 1  
> Maximum value: 5 | The collector asks the API for all the IoCs from present moment to some days back, minimum in the last day (1 day back).<br>Usual values for services:<br>- emails → 5
> - urls → 5
> - hashes → 5
> - domains → 5
> - ips → 2 |
> 
> ### Download the Docker image
> 
> The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
> 
> | **Collector Docker image** | **SHA-256 hash** |
> | --- | --- |
> | [collector-rapid7_intsights_ioc_collector_if-docker-image-2.2.0](https://drive.google.com/file/d/1zuF8cp-myw1fZvIVjVihcLRTV5NqabXk/view?usp=drive_link) | `1c96a75b0ed6b2002241c4fb68bc211d95e050f03c59c6e2ca47e37d19968f51` |
> 
> Use the following command to add the Docker image to the system:
> 
> ```
> gunzip -c collector-rapid7-intsights-docker-image-<version>.tgz | docker load
> ```
> 
> > ℹ️ Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace "`<version>`" with a proper value.
> 
> The Docker image can be deployed on the following services:
> 
> #### Docker
> 
> Execute the following command on the root directory `<any_directory>/devo-collectors/devo-collector-intsights/`
> 
> ```
> docker run \
> --name collector-rapid7-intsights\
> --volume $PWD/certs:/devo-collector/certs \
> --volume $PWD/config:/devo-collector/config \
> --volume $PWD/state:/devo-collector/state \
> --env CONFIG_FILE=config.yaml \
> --rm -it docker.devo.internal/collectors/rapid7-intsights:<version>
> ```
> 
> > ⚠️ Replace `<version>` with the required value.
> 
> #### Docker Compose
> 
> The following Docker Compose file can be used to execute the Docker container. It must be created in the `<any_directory>/devo-collectors/devo-collector-intsights/` directory.
> 
> ```
> version: '3'
> services:
>   intsights:
>     build:
>       context: .
>       dockerfile: Dockerfile
>     image: docker.devo.internal/collectors/rapid7-intsights:${IMAGE_VERSION:-latest}
>     container_name: collector-rapid7-intsights
>     volumes:
>       - ./certs:/devo-collector/certs
>       - ./config:/devo-collector/config
>       - ./state:/devo-collector/state
>     environment:
>       - CONFIG_FILE=${CONFIG_FILE:-config.yaml}
> ```
> 
> To run the container using docker-compose, execute the following command from the `<any_directory>/devo-collectors/devo-collector-intsights/` directory:
> 
> ```
> IMAGE_VERSION=<version> docker-compose up -d
> ```
> 
> > ⚠️ Replace `<version>` with the required value.

## Change log

| **Release** | **Released on** | **Release type** | **Details** | **Recommendations** |
| --- | --- | --- | --- | --- |
| `v2.2.0` | 10/13/2023 | [IMPROVEMENTS] | **Improvements:**<br>- Added `status` field to lookups
- Parameter `request_period_in_seconds` made optional in config, default to 21600 | `Recommended Version` |
| `v2.1.0` | 10/2/2023 | [BUG FIXING] [IMPROVEMENTS] | #### Improvements<br>- Upgraded DCSDK from 1.8.0 to 1.9.2
  - Upgrade internal dependencies
  - Store lookup instances into DevoSender to avoid creation of new instances for the same lookup
  - Ensure service_config is a dict into templates
  - Ensure special characters are properly sent to the platform
  - Changed log level to some messages from info to debug
  - Changed some wrong log messages
  - Upgraded some internal dependencies
  - Changed queue passed to setup instance constructor<br>#### Bug Fixing<br>- Fixed format for input JSON data in case that they are a dict or list | `Recommended Version` |
| `v2.0.0` | 7/11/2023 | [BUG FIXING] [IMPROVEMENTS] | #### Improvements<br>- Upgraded DCSDK from 1.1.4 to 1.8.0
  - Added log traces for knowing the execution environment status (debug mode)
  - Fixes in the current puller template version
  - Improved log trace details when runtime exceptions happen
  - Refactored source code structure
  - New "templates" functionality
  - Functionality for detecting some system signals for starting the controlled stopping
  - Input objects sends again the internal messages to `devo.collectors.out` table
  - Upgraded DevoSDK to version 3.6.4 to fix a bug related to a connection loss with Devo
  - Refactored source code structure
  - Changed way of executing the controlled stopping
  - Minimized probabilities of suffering a DevoSDK bug related to "sender" to be null
  - Ability to validate collector setup and exit without pulling any data
  - Ability to store in the persistence the messages that couldn't be sent after the collector stopped
  - Ability to send messages from the persistence when the collector starts and before the puller begins working
  - Ensure special characters are properly sent to the platform
  - Added a lock to enhance sender object
  - Added new class attrs to the `__setstate__` and `__getstate__` queue methods
  - Fix sending attribute value to the `__setstate__` and `__getstate__` queue methods
  - Added log traces when queues are full and have to wait
  - Added log traces of queues time waiting every minute in debug mode
  - Added method to calculate queue size in bytes
  - Block incoming events in queues when there are no space left
  - Send telemetry events to Devo platform
  - Upgraded internal Python dependency Redis to v4.5.4
  - Upgraded internal Python dependency DevoSDK to v5.1.3
  - Fixed obfuscation not working when messages are sent from templates
  - New method to figure out if a puller thread is stopping
  - Upgraded internal Python dependency DevoSDK to v5.0.6
  - Improved logging on messages/bytes sent to Devo platform
  - Fixed wrong bytes size calculation for queues
  - New functionality to count bytes sent to Devo Platform (shown in console log)
  - Upgraded internal Python dependency DevoSDK to v5.0.4
  - Fixed bug in persistence management process, related to persistence reset
  - Aligned source code typing to be aligned with Python 3.9.x
  - Inject `environment` property from user config
  - Obfuscation service can be now configured from user config and module definiton
  - Obfuscation service can now obfuscate items inside arrays<br>#### Bug Fixing<br>- Endpoints have been changed from v2 to v3. v2 endpoints are no longer available in the API, therefore this collector can no longer be downgraded. | `Recommended version`<br>Highly recommended to update, the API no longer supports the endpoints used in previous versions. |