--- title: "Snyk collector" canonical: "https://docs.devo.com/space/latest/691667119/Snyk%20collector" format: markdown --- > Macro (toc) ## Overview Snyk is a developer-first platform for securing code, dependencies, containers, IaC and cloud deployments, from code to cloud and back to code. ## Devo collector features | **Feature** | **Details** | | --- | --- | | Allow parallel downloading (`multipod`) | `not allowed` | | Running environments | - `collector server` - `on-premise` | | Populated Devo events | `table` | | Flattening preprocessing | `no` | | Allowed source events obfuscation | `yes` | ## Data sources | **Data source** | **Description** | **API endpoint** | **Collector service name** | **Devo table** | **Available from release** | | --- | --- | --- | --- | --- | --- | | Audit Events | Fetch all the Audit Events. | [Snyk API · Apiary](https://snyk.docs.apiary.io/#reference/audit-logs) | `organization_audit_logs` | `sast.snyk.organization.audit` | v1.0 | For more information on how the events are parsed, [visit our page](https://devodocs.atlassian.net/wiki/spaces/latest/pages/356122684). ## Vendor setup In order to retrieve the data, the following details will be required from your Thinkst API. - To use this API, you must get your token from Snyk. It can be seen on `https://snyk.io/account/` after you register with Snyk and login.The token should be supplied in an Authorization header with the token, preceded by token. - Each request to Snyk has to be done in the context of a Snyk organization. If no organization is specified, the user's default organization (user is identified according to the `API_KEY`) will be used. The organization determines the access rights, licenses policy and is the unit of billing for private projects. An organization should be given as a query parameter named org, with the public identifier given to this org. The list of organizations and their corresponding public ids can be found with the organization resource. ## Minimum configuration required for basic pulling Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below. > ℹ️ This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details. | **Setting** | **Details** | | --- | --- | | `Organization ID` | Organization ID you set up | | `Authentication Token` | Auth Token provided from set up | > ℹ️ See the **Accepted authentication methods** section to verify what settings are required based on the desired authentication method. ## Accepted authentication methods | **Authentication method** | **API** | **Auth token** | | --- | --- | --- | | Basic authentication | [REQUIRED] | [REQUIRED] | ## Run the collector Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector). > Macro (rw-ui-tabs-macro) > > > Macro (rw-tab) > > We use a piece of software called Collector Server to host and manage all our available collectors. > > To enable the collector for a customer: > > 1. In the **Collector Server** **GUI**, access the **domain** in which you want this instance to be created > 2. Click **Add Collector** and find the one you wish to add. > 3. In the **Version** field, select the latest value. > 4. In the **Collector Name** field, set the value you prefer (this name must be unique inside the same Collector Server domain). > 5. In the sending method select **Direct Send. Direct Send** configuration is optional for collectors that create `Table` events, but mandatory for those that create `Lookups`. > 6. In the **Parameters** section, establish the **Collector Parameters** as follows below: > > ``` > { > "global_overrides": { > "debug": true > }, > "inputs": { > "snyk": { > "id": 1, > "enabled": true, > "credentials": { > "admin_token": "" > }, > "services": { > "organization_audit_logs": { > "request_period_in_seconds": 86400, > "override_devo_tag": "", > "org_id": "", > "from_date": "YYYY-MM-DD", > "filters": { > "excludeEvent": [ > "api.access", > "org.project.test" > ] > } > } > } > } > } > } > ``` > > > ℹ️ All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object. > > Please replace the placeholders with real world values following the description table below: > > | | | | | | > | --- | --- | --- | --- | --- | > | **Parameter** | **Data type** | **Type** | **Value range / Format** | **Details** | > | `debug_status` | `bool` | `Mandatory` | `false` / `true` | If the value is `true`, the debug logging traces will be enabled when running the collector. If the value is `false`, only the `info`, `warning` and `error` logging levels will be printed. | > | `collector_id` | `int` | `Mandatory` | Minimum length: 1 > Maximum length: 5 | Use this param to give an unique id to this collector. | > | `enabled` | `bool` | `Mandatory` | `false` / `true` | Use this param to enable or disable the given input logic when running the collector. If the value is `true`, the input will be run. If the value is `false`, it will be ignored. | > | `admin_token` | `str` | `Mandatory` | | | > | `org_id` | `str` | `Mandatory` | | | > | `overide_tag` | `str` | `Optional` | regex
`^(?=[a-z])[a-z\\d_-]+(?:\\.[a-z][a-z\\d_-]*){2,}$` | Change to tag | > | `from_date` | `date` | `Mandatory` | `2023-11-27` | Up to 90 days prior | > | `excludeEvent` | `object` | `Optional` | | This anything in the “event” in Devo can be excluded. | > > > Macro (rw-tab) > > This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running. > > ### Structure > > The following directory structure should be created for being used when running the collector: > > ``` > > └── devo-collectors/ > └── / > ├── certs/ > │ ├── chain.crt > │ ├── .key > │ └── .crt > ├── state/ > └── config/ > └── config.yaml > ``` > > > ⚠️ Replace `` with the proper value. > > ### Devo credentials > > In Devo, go to **Administration → Credentials → X.509 Certificates**, download the **Certificate**, **Private key** and **Chain CA** and save them in `/certs/`. Learn more about security credentials in Devo [here](#). > > ![image-20240624-103235.png](media://5a8e2a9f-323e-4394-9f5f-04b505fb8d2a) > > > ⚠️ Replace `` with the proper value. > > ### Editing the config.yaml file > > ``` > globals: > debug: true > id: not_used > name: snyk > persistence: > type: filesystem > config: > directory_name: state > multiprocessing: false > queue_max_size_in_mb: 1024 > queue_max_size_in_messages: 1000 > queue_max_elapsed_time_in_sec: 60 > queue_wrap_max_size_in_messages: 100 > # type: redis > # config: > # host: host > # port: port > # password: password > # db: db > outputs: > # devo_us_1: > # type: devo_platform > # config: > # address: collector-us.devo.io > # port: 443 > # type: SSL > # chain: chain.crt > # cert: .crt > # key: .key > # devo_eu_1: > # type: devo_platform > # config: > # address: collector-eu.devo.io > # port: 443 > # type: SSL > # chain: chain.crt > # cert: .crt > # key: .key > # relay_1: > # type: syslog > # config: > # address: 127.0.0.1 > # port: 13000 > # period_sender_stats_in_seconds: 60 > # concurrent_connections: 2 > console_1: > type: console > > inputs: > snyk: > id: 1001 > enabled: true > credentials: > admin_token: admin_token_uuid > services: > organization_audit_logs: > type: predefined > request_period_in_seconds: 86400 > override_devo_tag: > org_id: org_id_uuid > from_date: YYYY-MM-DD # up to 90 days ago > filters: > excludeEvent: > - api.access > - org.project.test > ``` > > > ℹ️ All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object. > > Replace the placeholders with your required values following the description table below: > > | | | | | | > | --- | --- | --- | --- | --- | > | **Parameter** | **Data type** | **Type** | **Value range** | **Details** | > | `debug_status` | `bool` | `Mandatory` | `false` / `true` | If the value is `true`, the debug logging traces will be enabled when running the collector. If the value is `false`, only the `info`, `warning` and `error` logging levels will be printed. | > | `collector_id` | `int` | `Mandatory` | Minimum length: 1 > Maximum length: 5 | Use this param to give an unique id to this collector. | > | `collector_name` | `str` | `Mandatory` | Minimum length: 1 > Maximum length: 10 | Use this param to give a valid name to this collector. | > | `devo_address` | `str` | `Mandatory` | `collector-us.devo.io` > `collector-eu.devo.io` | Use this param to identify the Devo Cloud where the events will be sent. | > | `chain_filename` | `str` | `Mandatory` | Minimum length: 4 > Maximum length: 20 | Use this param to identify the chain.cert  file downloaded from your Devo domain. Usually this file's name is: `chain.crt` | > | `cert_filename` | `str` | `Mandatory` | Minimum length: 4 > Maximum length: 20 | Use this param to identify the `file.cert` downloaded from your Devo domain. | > | `key_filename` | `str` | `Mandatory` | Minimum length: 4 > Maximum length: 20 | Use this param to identify the `file.key` downloaded from your Devo domain. | > | `short_unique_id` | `int` | `Mandatory` | Minimum length: 1 > Maximum length: 5 | Use this param to give an unique id to this input service.
> ⚠️ This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision. | > | `enabled` | `bool` | `Mandatory` | `false` / `true` | Use this param to enable or disable the given input logic when running the collector. If the value is `true`, the input will be run. If the value is `false`, it will be ignored. | > | `admin_token` | `str` | `Mandatory` | | | > | `org_id` | `str` | `Mandatory` | | | > | `overide_tag` | `str` | `Optional` | regex
`^(?=[a-z])[a-z\\d_-]+(?:\\.[a-z][a-z\\d_-]*){2,}$` | Change to tag | > | `from_date` | `date` | `Mandatory` | `2023-11-27` | Up to 90 days prior | > | `excludeEvent` | `object` | `Optional` | | This anything in the “event” in Devo can be excluded. | > | `enabled` | `bool` | `Mandatory` | `false` / `true` | Use this param to enable or disable the given input logic when running the collector. If the value is `true`, the input will be run. If the value is `false`, it will be ignored. | > | `admin_token` | `str` | `Mandatory` | | | > | `org_id` | `str` | `Mandatory` | | | > > ### Download the Docker image > > The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table: > > | **Collector Docker image** | **SHA-256 hash** | > | --- | --- | > | [collector-snyk_collector_if-docker-image-1.1.1](https://drive.google.com/file/d/1VZmbkUk8e13tJZj2Ofy8Hd1AhuaXi7Sq/view?usp=drive_link) | `bb524d5d5210e4e22ab7bedd510576d18c55155533adc3e9a8f0465eae1ddeb7` | > > Use the following command to add the Docker image to the system: > > ``` > gunzip -c -.tgz | docker load > ``` > > > ⚠️ Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace `` and `` with a proper value. > > The Docker image can be deployed on the following services: > > #### Docker > > Execute the following command on the root directory `/devo-collectors//` > > ``` > docker run > --name collector- > --volume $PWD/certs:/devo-collector/certs > --volume $PWD/config:/devo-collector/config > --volume $PWD/state:/devo-collector/state > --env CONFIG_FILE=config.yaml > --rm > --interactive > --tty > : > ``` > > > ⚠️ Replace ``, `` and `` with the proper values. > > #### Docker Compose > > The following Docker Compose file can be used to execute the Docker container. It must be created in the `/devo-collectors//` directory. > > ``` > version: '3' > services: > collector-: > image: :${IMAGE_VERSION:-latest} > container_name: collector- > volumes: > - ./certs:/devo-collector/certs > - ./config:/devo-collector/config > - ./credentials:/devo-collector/credentials > - ./state:/devo-collector/state > environment: > - CONFIG_FILE=${CONFIG_FILE:-config.yaml} > ``` > > To run the container using docker-compose, execute the following command from the `/devo-collectors//` directory: > > ``` > IMAGE_VERSION= docker-compose up -d > ``` > > > ⚠️ Replace ``, `` and `` with the proper values. ## Collector operations This section is intended to explain how to proceed with specific operations of this collector.
Verify collector operations ### Initialization The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration. A successful run has the following output messages for the initializer module: ``` 2024-02-02T11:21:08.075 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.045", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Validating service input config\", \"time\": \"2024-02-02T16:21:08.045018Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"birds\", \"module_name\": \"ThinkstCanaryBirdsPuller\"}"} AttributeError: 'CollectorMultiprocessingQueue' object has no attribute 'max_size_in_messages' 2024-02-02T11:21:08.075 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.046", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Running overriding rules\", \"time\": \"2024-02-02T16:21:08.046346Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"birds\", \"module_name\": \"ThinkstCanaryBirdsPuller\"}"} 2024-02-02T11:21:08.076 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.046", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Validating the rate limiter config given by the user\", \"time\": \"2024-02-02T16:21:08.046601Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"birds\", \"module_name\": \"ThinkstCanaryBirdsPuller\"}"} 2024-02-02T11:21:08.076 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.046", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \" setting has not been defined. The generic settings will be used instead.\", \"time\": \"2024-02-02T16:21:08.046783Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"birds\", \"module_name\": \"ThinkstCanaryBirdsPuller\"}"} 2024-02-02T11:21:08.076 INFO InputProcess::MainThread -> Validating service input config 2024-02-02T11:21:08.076 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.046", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Adding raw config to the collector store\", \"time\": \"2024-02-02T16:21:08.046946Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"birds\", \"module_name\": \"ThinkstCanaryBirdsPuller\"}"} 2024-02-02T11:21:08.076 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.047", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Running custom validation rules\", \"time\": \"2024-02-02T16:21:08.047106Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"birds\", \"module_name\": \"ThinkstCanaryBirdsPuller\"}"} 2024-02-02T11:21:08.077 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.047", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"ThinkstCanaryBirdsPuller(thinkst,12345,birds,predefined) Finalizing the execution of init_variables()\", \"time\": \"2024-02-02T16:21:08.047273Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"birds\", \"module_name\": \"ThinkstCanaryBirdsPuller\"}"} 2024-02-02T11:21:08.077 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.051", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"ThinkstCanaryAuditPuller(thinkst,12345,audit_trail_log,predefined) Starting the execution of init_variables()\", \"time\": \"2024-02-02T16:21:08.051362Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"audit_trail_log\", \"module_name\": \"ThinkstCanaryAuditPuller\"}"} 2024-02-02T11:21:08.077 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.051", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Validating service metadata\", \"time\": \"2024-02-02T16:21:08.051570Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"audit_trail_log\", \"module_name\": \"ThinkstCanaryAuditPuller\"}"} 2024-02-02T11:21:08.078 INFO InputProcess::MainThread -> Running overriding rules 2024-02-02T11:21:08.078 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.053", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Validating defined module definition\", \"time\": \"2024-02-02T16:21:08.053176Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"audit_trail_log\", \"module_name\": \"ThinkstCanaryAuditPuller\"}"} 2024-02-02T11:21:08.078 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.057", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Validating common input config\", \"time\": \"2024-02-02T16:21:08.057634Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"audit_trail_log\", \"module_name\": \"ThinkstCanaryAuditPuller\"}"} 2024-02-02T11:21:08.078 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user 2024-02-02T11:21:08.078 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.058", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Validating service input config\", \"time\": \"2024-02-02T16:21:08.058971Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"audit_trail_log\", \"module_name\": \"ThinkstCanaryAuditPuller\"}"} 2024-02-02T11:21:08.079 INFO InputProcess::MainThread -> setting has not been defined. The generic settings will be used instead. 2024-02-02T11:21:08.079 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.060", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Running overriding rules\", \"time\": \"2024-02-02T16:21:08.060588Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"audit_trail_log\", \"module_name\": \"ThinkstCanaryAuditPuller\"}"} 2024-02-02T11:21:08.079 INFO InputProcess::MainThread -> Adding raw config to the collector store 2024-02-02T11:21:08.079 INFO InputProcess::MainThread -> Running custom validation rules 2024-02-02T11:21:08.079 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.060", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Validating the rate limiter config given by the user\", \"time\": \"2024-02-02T16:21:08.060780Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"audit_trail_log\", \"module_name\": \"ThinkstCanaryAuditPuller\"}"} 2024-02-02T11:21:08.080 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.060", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \" setting has not been defined. The generic settings will be used instead.\", \"time\": \"2024-02-02T16:21:08.060948Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"audit_trail_log\", \"module_name\": \"ThinkstCanaryAuditPuller\"}"} 2024-02-02T11:21:08.080 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.061", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Adding raw config to the collector store\", \"time\": \"2024-02-02T16:21:08.061109Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"audit_trail_log\", \"module_name\": \"ThinkstCanaryAuditPuller\"}"} 2024-02-02T11:21:08.080 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.061", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Running custom validation rules\", \"time\": \"2024-02-02T16:21:08.061269Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"audit_trail_log\", \"module_name\": \"ThinkstCanaryAuditPuller\"}"} 2024-02-02T11:21:08.080 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.063", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"ThinkstCanaryAuditPuller(thinkst,12345,audit_trail_log,predefined) Finalizing the execution of init_variables()\", \"time\": \"2024-02-02T16:21:08.063397Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"audit_trail_log\", \"module_name\": \"ThinkstCanaryAuditPuller\"}"} 2024-02-02T11:21:08.080 INFO InputProcess::MainThread -> ThinkstCanaryIncidentPuller(thinkst,12345,incidents,predefined) Finalizing the execution of init_variables() 2024-02-02T11:21:08.081 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.067", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"ThinkstCanaryIncidentPuller(thinkst,12345,incidents,predefined) Starting the execution of init_variables()\", \"time\": \"2024-02-02T16:21:08.067826Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"incidents\", \"module_name\": \"ThinkstCanaryIncidentPuller\"}"} 2024-02-02T11:21:08.081 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.068", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Validating service metadata\", \"time\": \"2024-02-02T16:21:08.068059Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"incidents\", \"module_name\": \"ThinkstCanaryIncidentPuller\"}"} 2024-02-02T11:21:08.081 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.069", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Validating defined module definition\", \"time\": \"2024-02-02T16:21:08.069740Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"incidents\", \"module_name\": \"ThinkstCanaryIncidentPuller\"}"} 2024-02-02T11:21:08.082 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.074", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Validating common input config\", \"time\": \"2024-02-02T16:21:08.074830Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"incidents\", \"module_name\": \"ThinkstCanaryIncidentPuller\"}"} 2024-02-02T11:21:08.082 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.076", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Validating service input config\", \"time\": \"2024-02-02T16:21:08.076506Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"incidents\", \"module_name\": \"ThinkstCanaryIncidentPuller\"}"} 2024-02-02T11:21:08.083 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.078", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Running overriding rules\", \"time\": \"2024-02-02T16:21:08.078607Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"incidents\", \"module_name\": \"ThinkstCanaryIncidentPuller\"}"} 2024-02-02T11:21:08.083 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.078", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Validating the rate limiter config given by the user\", \"time\": \"2024-02-02T16:21:08.078856Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"incidents\", \"module_name\": \"ThinkstCanaryIncidentPuller\"}"} 2024-02-02T11:21:08.083 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.079", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \" setting has not been defined. The generic settings will be used instead.\", \"time\": \"2024-02-02T16:21:08.079078Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"incidents\", \"module_name\": \"ThinkstCanaryIncidentPuller\"}"} 2024-02-02T11:21:08.083 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.079", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Adding raw config to the collector store\", \"time\": \"2024-02-02T16:21:08.079288Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"incidents\", \"module_name\": \"ThinkstCanaryIncidentPuller\"}"} 2024-02-02T11:21:08.083 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.079", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Running custom validation rules\", \"time\": \"2024-02-02T16:21:08.079498Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"incidents\", \"module_name\": \"ThinkstCanaryIncidentPuller\"}"} 2024-02-02T11:21:08.084 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.080", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"ThinkstCanaryIncidentPuller(thinkst,12345,incidents,predefined) Finalizing the execution of init_variables()\", \"time\": \"2024-02-02T16:21:08.080907Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"incidents\", \"module_name\": \"ThinkstCanaryIncidentPuller\"}"} 2024-02-02T11:21:08.085 WARNING InputProcess::MainThread -> A previous rate limiter with same "period_in_seconds" and "number_of requests" was already existing: "86400/1000" 2024-02-02T11:21:08.086 INFO InputProcess::MainThread -> ThinkstCanaryCanaryTokenPuller(thinkst,12345,tokens,predefined) Starting the execution of init_variables() 2024-02-02T11:21:08.086 INFO InputProcess::MainThread -> Validating service metadata 2024-02-02T11:21:08.088 INFO InputProcess::MainThread -> Validating defined module definition 2024-02-02T11:21:08.092 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.086", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"ThinkstCanaryCanaryTokenPuller(thinkst,12345,tokens,predefined) Starting the execution of init_variables()\", \"time\": \"2024-02-02T16:21:08.086092Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"tokens\", \"module_name\": \"ThinkstCanaryCanaryTokenPuller\"}"} 2024-02-02T11:21:08.092 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.086", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Validating service metadata\", \"time\": \"2024-02-02T16:21:08.086338Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"tokens\", \"module_name\": \"ThinkstCanaryCanaryTokenPuller\"}"} 2024-02-02T11:21:08.092 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.088", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Validating defined module definition\", \"time\": \"2024-02-02T16:21:08.088386Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"tokens\", \"module_name\": \"ThinkstCanaryCanaryTokenPuller\"}"} 2024-02-02T11:21:08.093 INFO InputProcess::MainThread -> Validating common input config 2024-02-02T11:21:08.094 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.093", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Validating common input config\", \"time\": \"2024-02-02T16:21:08.093600Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"tokens\", \"module_name\": \"ThinkstCanaryCanaryTokenPuller\"}"} 2024-02-02T11:21:08.095 INFO InputProcess::MainThread -> Validating service input config 2024-02-02T11:21:08.095 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-02-02 16:21:08.095", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Validating service input config\", \"time\": \"2024-02-02T16:21:08.095132Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"thinkst\", \"service_name\": \"tokens\", \"module_name\": \"ThinkstCanaryCanaryTokenPuller\"}"} 2024-02-02T11:21:08.096 INFO InputProcess::MainThread -> Running overriding rules 2024-02-02T11:21:08.097 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user 2024-02-02T11:21:08.097 INFO InputProcess::MainThread -> setting has not been defined. The generic settings will be used instead. 2024-02-02T11:21:08.097 INFO InputProcess::MainThread -> Adding raw config to the collector store 2024-02-02T11:21:08.097 INFO InputProcess::MainThread -> Running custom validation rules 2024-02-02T11:21:08.097 INFO InputProcess::MainThread -> ThinkstCanaryCanaryTokenPuller(thinkst,12345,tokens,predefined) Finalizing the execution of init_variables() ``` ### Events delivery and Devo ingestion The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method. A successful run has the following output messages for the initializer module: ``` INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0 INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {} INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Sender: SyslogSender(standard_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True} INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Standard - Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 44 (elapsed 0.007 seconds) INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0 INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {} INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Sender: SyslogSender(internal_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True} INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Internal - Total number of messages sent: 1, messages sent since "2022-06-28 10:39:22.516313+00:00": 1 (elapsed 0.019 seconds)By default, these information traces will be displayed every 10 minutes. ``` #### Sender services The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (`internal`, `standard`, and `lookup`). This collector uses the following Sender Services: | **Sender services** | **Description** | | --- | --- | | `internal_senders` | In charge of delivering internal metrics to Devo such as logging traces or metrics. | | `standard_senders` | In charge of delivering pulled events to Devo. | #### Sender statistics Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type: | **Logging trace** | **Description** | | --- | --- | | `Number of available senders: 1` | Displays the number of concurrent senders available for the given Sender Service. | | `sender manager internal queue size: 0` | Displays the items available in the internal sender queue.
> ℹ️ This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders. | | `Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds)` | Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:
- 44 events were sent to Devo since the collector started. - The last checkpoint timestamp was `2022-06-28 10:39:22.511671+00:00`. - 21 events where sent to Devo between the last UTC checkpoint and now. - Those 21 events required `0.007 seconds` to be delivered.
> ℹ️ By default these traces will be shown every 10 minutes. |
Check memory usage To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process. - The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector. - The global pressure of the available memory is displayed in the `global` value. - All metrics (Global, RSS, VMS) include the value before freeing and after `previous -> after freeing memory` ``` INFO InputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(34.50MiB -> 34.08MiB), VMS(410.52MiB -> 410.02MiB) INFO OutputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(28.41MiB -> 28.41MiB), VMS(705.28MiB -> 705.28MiB) ``` > ℹ️ Differences between `RSS` and `VMS` memory usage: > ℹ️ > ℹ️ - `RSS` is the Resident Set Size, which is the actual physical memory the process is using > ℹ️ - `VMS` is the Virtual Memory Size which is the virtual memory that process is using
Enable/disable the logging debug mode Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes. - To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector. - To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector. For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode.
## Change log | **Release** | **Released on** | **Release type** | **Details** | **Recommendations** | | --- | --- | --- | --- | --- | | `v1.1.1` | 9/4/2024 | [IMPROVEMENTS] | **Improvements**
- Updated API endpoints. Added new service group audit logs. | `Recommended version` | | `v1.0.0` | 2/5/2024 | [NEW RELEASE] | **New features**
- New `audit` service. | `-` |