--- title: "ServiceNow collector" canonical: "https://docs.devo.com/space/latest/168198207/ServiceNow%20collector" format: markdown --- > Macro (toc) ## Overview **ServiceNow Incident Management **(previously ITSM) helps you to monitor, track, and analyze service incidents to support business operations. This collector enables you to integrate ServiceNow with the Devo Platform making it easy to query and analyze the ServiceNow Incident Management data, view it, and pre-configured Activeboards, or customize them to suit your needs. Devo’s Service Now Collector interfaces with **ServiceNow** to help streamline security-related service management and IT operations. The data in ServiceNow tickets can be mirrored to Devo so that you can track the status and information in the task. ## Devo collector features | **Feature** | **Details** | | --- | --- | | Allow parallel downloading (`multipod`) | - `Not allowed` | | Running environments | - `Collector server` - `On-premise` | | Populated Devo events | - `Table` - `Lookups` | | Flattening preprocessing | - `Yes` | ## Data sources | | | | | | | | --- | --- | --- | --- | --- | --- | | **Data Source** | **Description** | **API Endpoint** | **Collector service name** | **Devo Table** | **Available from release** | | **Incident data** | Incident is an unplanned interruption to a service or reduction in the quality of service. Failure of a configuration item that has not yet affected service is also an incident – for example, failure of one disk from a mirror set. | `api/now/table/incident` | `incident` | `itsm.servicenow.tables.incident` | `v1.0.0` | | **Event data** | Events are an indication in ServiceNow that something notable has occurred. Events can be generated by server-side scripts, workflows, ServiceNow processes, or by user actions such as: Impersonating a user. Logging in. | `api/now/table/ecc_event` | `event` | `itsm.servicenow.tables.event` | `v1.0.0` | | **Change data** | A change request records the detailed information about the change, such as the reason for the change, the priority, the risk, the type of change, and the change category. | `api/now/table/chg_model` | `change` | `itsm.servicenow.tables.change` | `v1.0.0` | | **User data** | The individuals who use the ServiceNow within an organization. The system administrator has the authority to create a new user, browse through the list of existing users, delete a user and impersonate a user | `api/now/table/sys_user` | `user` | `itsm.servicenow.tables.user` | `v1.0.0` | | **Location data** | If your organization supports more than one distinct location, you can configure these locations in the platform to help further identify users, assets, and incidents. | `api/now/table/cmn_location ` | `location` | `itsm.servicenow.tables.location` | `v1.0.0` | | **Application Server** | A base table for logical CIs, which indicate the primary function of a physical or virtual server such as a Tomcat server or a WebSphere server. | `api/now/table/cmdb_ci_app_server` | `cmdb_ci_app_server` | `itsm.servicenow.cmdb.cmdbciappserver` | `v1.0.0` | | **Database Instance** | Software and memory used to manipulate data in a database. | `api/now/table/cmdb_ci_db_instance ` | `cmdb_ci_db_instance` | `itsm.servicenow.cmdb.cmdbcidbinstance` | `v1.0.0` | | **Infrastructure Service** | IT services which support providing computing infrastructure. | `api/now/table/cmdb_ci_infra_service` | `cmdb_ci_infra_service` | `itsm.servicenow.cmdb.cmdbciinfraservice` | `v1.0.0` | | **Configuration Item** | Base configuration item table. | `api/now/table/cmdb_ci` | `cmdb_ci(list)` | `itsm.servicenow.cmdb.cmdbci` | `v1.0.0` | | **Server** | Base class for all types of servers. | `api/now/table/cmdb_ci_server ` | `cmdb_ci_server` | `itsm.servicenow.cmdb.cmdbciserver` | `v1.0.0` | | **Service** | IT Service that directly supports a Business Process (ITIL). | `api/now/table/cmdb_ci_service` | `cmdb_ci_service` | `itsm.servicenow.cmdb.cmdbciservice` | `v1.0.0` | | **Virtual Machine HyperVisor** | Virtual Machine HyperVisor | `api/now/table/cmdb_ci_vm` | `cmdb_ci_vm` | `itsm.servicenow.cmdb.cmdbcivm` | `v1.0.0` | | **CI Relationship** | Store relationship between your CIs. | `lapi/now/table/cmdb_rel_ci ` | `cmdb_rel_ci` | `itsm.servicenow.cmdb.cmdbrelci` | `v1.0.0` | | **cmn location** | Store locations in common table | `api/now/table/cmn_location` | `cmn_location` | `itsm.servicenow.cmdb.cmnlocation` | `v1.0.0` | | **Any Extended table** | Extending an existing ServiceNow table means the new table inherits the parent table's columns as well as its business logic | `api/now/table/` | | `custom_tag` | `v1.0.0` | | **Any Database view** | A Database View is essentially a pseudo-table representing a join between several contributing tables. | `api/now/table/` | | `custom_tag` | `v1.0.0` | For more information on how the events are parsed, [visit our page](https://docs.devo.com/space/latest/94664125/itsm.servicenow). ## Vendor setup There are some requirements to enable this collector: 1. Valid **ServiceNow** credentials. 2. Set the **time zone** of the ServiceNow instance to GMT. In order to retrieve the data, you need to create an instance to obtain **credentials**. In the following table are the steps on how to create an instance, get the credentials, and put the **time zone** of the instance in GMT. | **Action** | **Steps** | | --- | --- | | Go to **ServiceNow's** development website. | 1. Go to ServiceNow developers website [ServiceNow Developers](https://developer.servicenow.com/dev.do#!/home). 2. Click on **Sign In **in the top-right corner of the page. | | Enter your login **credentials**. | 1. Enter your **email **address in the text box and click on **Next**. 2. Enter your **password** in the text box and click on **Sign In**. | | Request an **instance**. | 1. After logging in, click on **Request Instance** in the top menu. On the right-hand side. 2. Choose a release, for example, Rome and click on the **Request** button. 3. Wait for the instance to be generated, it may take a few minutes.
> ⚠️ The instance has to be awake for the collector to work. It goes into hibernation mode after a few hours if it is not used.
> ⚠️ Instances with no activity for over 10 days will be claimed back in to the pool. | | Copy **credentials**. | 1. When the instance is ready, a pop-up window will appear with the **instance URL**, **username** and **password**. 2. Save **instance URL**, **username** and **password**. 3. (Optional) If you want to access the new instance in the ServiceNow console click on **Open Instance**.
> ℹ️ It is important to save the **instance URL**, **username** and **password** to enter in the Collector configuration. | | Set instance **time zone** to GMT. | 1. Click on the photo of the instance administrator, on the right side of the upper panel of the screen. Then click on **Profile**. 2. Click the dropdown menu for the Timezone field and then select *GMT*. 3. Finally click on **Update**. | | (Optional) **Explore** the instance in the ServiceNow console. | (Optional) If you open the instance, you will be redirected to the ServiceNow console where you can create tables, filters and database views. | ## Minimum configuration required for basic pulling Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below. > ℹ️ This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details. | | | | --- | --- | | **Setting** | **Details** | | `api_base_url` | This parameter is the URL of the ServiceNow instance. | | `username` | Username to authenticate to the service. | | `password` | User password to authenticate to the service. | | `client id` | Client Id to authenticate to the service. | | `client secret` | Client Secret to authenticate to the service. | | `standard_table_service_name` | This parameter allows you to define a standard Service Now service based on known sources of Tables. | > ℹ️ See the **Accepted authentication methods** section to verify what settings are required based on the desired authentication method. ## Accepted authentication methods Depending on how did you obtain your credentials, you will have to either fill or delete the following properties on the JSON `credentials` configuration block. | | | | | | | --- | --- | --- | --- | --- | | **Authentication Method** | **Username** | **Password** | **client id** | **client secret** | | Username/Password
ClientId/ClientSecret | [REQUIRED] | [REQUIRED] | [REQUIRED] | [REQUIRED] | ## Run the collector Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector). > Macro (rw-ui-tabs-macro) > > > Macro (rw-tab) > > We use a piece of software called Collector Server to host and manage all our available collectors. > > To enable the collector for a customer: > > 1. In the **Collector Server** **GUI**, access the **domain** in which you want this instance to be created > 2. Click **Add Collector** and find the one you wish to add. > 3. In the **Version** field, select the latest value. > 4. In the **Collector Name** field, set the value you prefer (this name must be unique inside the same Collector Server domain). > 5. In the sending method select **Direct Send. Direct Send** configuration is optional for collectors that create `Table` events, but mandatory for those that create `Lookups`. > 6. In the **Parameters** section, establish the **Collector Parameters** as follows below: > > ### Editing the JSON configuration > > ``` > { > "global_overrides": { > "debug": false > }, > "inputs": { > "servicenow": { > "id": "", > "enabled": "", > "requests_per_second": "", > "api_base_url": "", > "credentials": { > "username": "", > "password": "", > "client_id": "", > "client_secret": "" > }, > "services": { > "": { > "historic_date": "", > "request_period_in_seconds": "", > "override_devo_tag": "" > }, > "": { > "historic_date": "", > "request_period_in_seconds": "", > "override_devo_tag": "", > "sysparm_display_value": "" > }, > "": { > "types": ["extended_table"], > "historic_date": "", > "request_period_in_seconds": "", > "table_name": "", > "override_devo_tag": "", > "sysparm_display_value": "" > }, > "": { > "types": ["extended_table"], > "historic_date": "", > "request_period_in_seconds": "", > "table_name": "", > "override_devo_tag": "", > "sysparm_display_value": "" > }, > "": { > "types": ["database_view"], > "historic_date": "", > "full_poll_every_sec": "", > "request_period_in_seconds": "", > "column_name": "", > "time_column": "", > "unique_key": "", > "column_name_types": "", > "table_name": "", > "override_devo_tag": "" > } > } > } > } > } > ``` > > > ℹ️ All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object. > > Please replace the placeholders with real world values following the description table below: > > | | | | | | > | --- | --- | --- | --- | --- | > | **Parameter** | **Data Type** | **Type** | **Value Range / Format** | **Details** | > | `short_unique_id` | `int` | `Mandatory` | Minimum length: 1 > Maximum length: 5 | Use this param to give an unique id to this input service.
> ⚠️ This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision. | > | `input_status` | `bool` | `Mandatory` | `false` / `true` | Use this param to enable or disable the given input logic when running the collector. If the value is `true`, the input will be run. If the value is `false`, it will be ignored. | > | `requests_per_second` | `int` | `Optional` | Minimum value: 1 | Customize the maximum number of API requests per second. If not used, the default setting will be used: `100000` requests/sec.
> 📝 This parameter should be removed if it is not used. | > | `api_base_url` | `str` | `Mandatory` | Valid URL following this regex: > `(^http|^https)(:\/\/)[-a-zA-Z0-9@:%._\\+~#=]{1,256}(\.)(service[-]?now)(\.com?$)` | This parameter is the URL of the ServiceNow instance. | > | `username` | `str` | `Mandatory` | Any | This parameter is the username to authenticate to the service. | > | `password` | `str` | `Mandatory` | Any | This parameter is the user password to authenticate to the service. | > | `client_id` | `str` | `Mandatory` | Any | This parameter is the client id to authenticate to the service. | > | `client_secret` | `str` | `Mandatory` | Any | This parameter is the client secret to authenticate to the service. | > | `standard_table_service_name` | `str` | `Mandatory` | Minimum length: 1 | This parameter allows you to define a standard Service Now service based on ready-to-use sources of Tables.
> ℹ️ Define just one of the following accepted list
For Service Now Tables:
- `incident` > - `event` > - `change` > - `user` > - `location` | > | `standard_cmbd_service_name` | `str` | `Mandatory` | Minimum length: 1 | This parameter allows you to define a standard Service Now service based on ready-to-use sources of CMBD Tables.
> ℹ️ Define just one of the following accepted list
For Service Now CMBD Tables:
- `cmdb_ci_app_server` > - `cmdb_ci_db_instance` > - `cmdb_ci_infra_service` > - `cmdb_ci` > - `cmdb_ci_server` > - `cmdb_ci_service` > - `cmdb_ci_vm` > - `cmdb_rel_ci` > - `cmn_location` | > | `custom_table_service` | `str` | `Mandatory` | Minimum length: 1 | This parameter allows you to define a new custom service to pull data from non-standard tables.
> ⚠️ The given name to this service must be unique. Having other services with the same name will produce a failure when reading the file.
> 📝 This parameter should be removed if it is not used. | > | `custom_cmbd_service` | `str` | `Mandatory` | Minimum length: 1 | This parameter allows you to define a new custom service to pull data from non-standard CMDB tables.
> ⚠️ The given name to this service must be unique. Having other services with the same name will produce a failure when reading the file.
> 📝 This parameter should be removed if it is not used. | > | `custom_db_view_service` | `str` | `Mandatory` | Minimum length: 1 | This parameter allows you to define a new custom service to pull data from non-standard DB Views entities.
> ⚠️ The given name to this service must be unique. Having other services with the same name will produce a failure when reading the file.
> 📝 This parameter should be removed if it is not used. | > | `table_name` | `str` | `Mandatory` | Minimum length: 1 | This parameter is the name of the ServiceNow table from which to ingest data. | > | `db_view_service_name` | `str` | `Mandatory` | Minimum length: 1 | This parameter is the name of the DB View service. For example: `db_view_1`. | > | `full_poll_every_sec` | `int` | `Mandatory` | Minimum value: 1 | This parameter indicates the number of seconds that pass until the next FULL request | > | `column_names_list` | `list` | `Mandatory` | At least one element in the list | This parameter defines the columns to be pulled from the given table. The expected content is an array of column names following the format below:
``` > "column_name": [ > "", > "", > "" > ] > ```
For example:
``` > "column_name": [ > "inc_number", > "inc_sys_created_on", > "inc_data_1" > ] > ``` | > | `time_column_name` | `str` | `Mandatory` | Minimum length: 1 | This parameter is the name of the column by which the data will be sorted.
For example: > `inc_sys_created_on` | > | `unique_key` | `str` | `Mandatory` | Minimum length: 1 | This parameter is the name of the field that contains the key -> 'id'.
For example: > `inc_number.` | > | `column_name_types_list` | `list` | `Mandatory` | It has to be the same length as `column_names_list` | This parameter defines the list of types of column names. The most common value is `str`. It has the following form:
``` > "column_name_types": [ > "", > "", > "" > ] > ```
For example:
``` > column_name_types: [ > "str", > "str", > "str" > ] > ``` | > | `historic_date` | `str` | `Optional` | With format: `YYYY-mm-dd HH:MM:SS.sssZ` | This configuration allows you to set a custom date as the beginning of the period to download. This allows downloading historical data (1 month back for example) before downloading new events.
> 📝 This parameter can be removed or commented. | > | `request_period_in_seconds` | `int` | `Optional` | Minimum value: 1 | Period in seconds used between each data pulling, this value will overwrite the default value (300 seconds).
> 📝 This parameter can be removed or commented. | > | `override_devo_tag` | `str` | `Optional` | A Devo Tag. For more information see [Devo Tags](https://docs.devo.com/space/latest/95126204/About%20Devo%20tags). | This parameter allows to define a custom devo tag.
> 📝 This parameter can be removed or commented. | > | `sysparm_display_value` | `bool` | `Optional` | `true` or `false` | Exports the display value (**true**) or raw value (**false**) for the CSV, Excel, XLSX, and XML file formats.
> 📝 This parameter can be removed or commented. | > | `date_field_value` | `str` | `Optional` | `sys_created_on` or `sys_updated_on` | Allows you to choose between the update date or the creation date to retrieve events. Using the update date, event status changes are shown. > The default value of this parameter is: `sys_created_on`
> 📝 This parameter can be removed or commented. | > > ![10_servicenow.png](media://2c8e9952-0f2c-4f75-9bd9-69ab49cbbb4a) > > > Macro (rw-tab) > > This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running. > > ### Structure > > The following directory structure should be created for being used when running the collector: > > ``` > > └── devo-collectors/ > └── / > ├── certs/ > │ ├── chain.crt > │ ├── .key > │ └── .crt > ├── state/ > └── config/ > └── config.yaml > ``` > > > ⚠️ Replace `` with the proper value. > > ### Devo credentials > > In Devo, go to **Administration → Credentials → X.509 Certificates**, download the **Certificate**, **Private key** and **Chain CA** and save them in `/certs/`. Learn more about security credentials in Devo [here](#). > > ![image](media://80294528-c042-4dd4-9a86-8f2a5a2cbca4) > > > ⚠️ Replace `` with the proper value. > > ### Editing the config.yaml file > > ``` > globals: > debug: false > id: > name: > persistence: > type: filesystem > config: > directory_name: state > > outputs: > devo_1: > type: devo_platform > config: > address: > port: 443 > type: SSL > chain: > cert: > key: > > inputs: > servicenow: > id: > enabled: > requests_per_second: > api_base_url: > > credentials: > username: > password: > client_id: > client_secret: > > services: > : > historic_date: > request_period_in_seconds: > override_devo_tag: > sysparm_display_value: > override_date_field: > > : > historic_date: > request_period_in_seconds: > override_devo_tag: > sysparm_display_value: > override_date_field: > > : > types: > - extended_table > historic_date: > request_period_in_seconds: > table_name: > override_devo_tag: > sysparm_display_value: > override_date_field: > > : > types: > - extended_table > historic_date: > request_period_in_seconds: > table_name: > override_devo_tag: > sysparm_display_value: > override_date_field: > > : > types: > - database_view > historic_date: > full_poll_every_sec: > request_period_in_seconds: > column_name: > time_column: > unique_key: > column_name_types: > table_name: > override_devo_tag: > ``` > > > ℹ️ All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object. > > Replace the placeholders with your required values following the description table below: > > | | | | | | > | --- | --- | --- | --- | --- | > | **Parameter** | **Data Type** | **Type** | **Value Range** | **Details** | > | `collector_id` | `int` | `Mandatory` | Minimum length: 1 > Maximum length: 5 | Use this param to give an unique ID to this collector. | > | `collector_name` | `str` | `Mandatory` | Minimum length: 1 > Maximum length: 10 | Use this param to give a valid name to this collector. | > | `devo_address` | `str` | `Mandatory` | `collector-us.devo.io` > `collector-eu.devo.io` | Use this param to identify the Devo Cloud where the events will be sent. | > | `chain_filename` | `str` | `Mandatory` | Minimum length: 4 > Maximum length: 20 | Use this param to identify the chain.cert  file downloaded from your Devo domain. Usually this file's name is: `chain.crt` | > | `cert_filename` | `str` | `Mandatory` | Minimum length: 4 > Maximum length: 20 | Use this param to identify the `file.cert` downloaded from your Devo domain. | > | `key_filename` | `str` | `Mandatory` | Minimum length: 4 > Maximum length: 20 | Use this param to identify the `file.key` downloaded from your Devo domain. | > | `short_unique_id` | `int` | `Mandatory` | Minimum length: 1 > Maximum length: 5 | Use this param to give an unique id to this input service.
> ℹ️ This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision. | > | `input_status` | `bool` | `Mandatory` | `false` / `true` | Use this param to enable or disable the given input logic when running the collector. If the value is `true`, the input will be run. If the value is `false`, it will be ignored. | > | `requests_per_second` | `int` | `Optional` | Minimum value: 1 | Customize the maximum number of API requests per second. If not used, the default setting will be used: `100000` requests/sec.
> ℹ️ This parameter should be removed if it is not used. | > | `api_base_url` | `str` | `Mandatory` | Valid URL following this regex: > `(^http|^https)(:\/\/)[-a-zA-Z0-9@:%._\\+~#=]{1,256}(\.)(service[-]?now)(\.com?$)` | This parameter is the URL of the ServiceNow instance. | > | `username` | `str` | `Mandatory` | Any | This parameter is the username to authenticate to the service. | > | `password` | `str` | `Mandatory` | Any | This parameter is the user password to authenticate to the service. | > | `client_id` | `str` | `Mandatory` | Any | This parameter is the client id to authenticate to the service. | > | `client_secret` | `str` | `Mandatory` | Any | This parameter is the client secret to authenticate to the service. | > | `standard_table_service_name` | `str` | `Mandatory` | Minimum length: 1 | This parameter allows you to define a standard Service Now service based on known sources of Tables.
> ℹ️ Define just one of the following accepted list
For Service Now Tables:
- `incident` > - `event` > - `change` > - `user` > - `location` | > | `standard_cmbd_service_name` | `str` | `Mandatory` | Minimum length: 1 | This parameter allows you to define a standard Service Now service based on known sources of CMBD Tables.
> ℹ️ Define just one of the following accepted list
For Service Now CMBD Tables:
- `cmdb_ci_app_server` > - `cmdb_ci_db_instance` > - `cmdb_ci_infra_service` > - `cmdb_ci` > - `cmdb_ci_server` > - `cmdb_ci_service` > - `cmdb_ci_vm` > - `cmdb_rel_ci` > - `cmn_location` | > | `custom_table_service` | `str` | `Mandatory` | Minimum length: 1 | This parameter allows you to define a new custom service to pull data from non-standard tables.
> ⚠️ The given name to this service must be unique. Having other services with the same name will produce a failure when reading the file.
> ℹ️ This parameter should be removed if it is not used. | > | `custom_cmbd_service` | `str` | `Mandatory` | Minimum length: 1 | This parameter allows you to define a new custom service to pull data from non-standard CMDB tables.
> ⚠️ The given name to this service must be unique. Having other services with the same name will produce a failure when reading the file.
> ℹ️ This parameter should be removed if it is not used. | > | `custom_db_view_service` | `str` | `Mandatory` | Minimum length: 1 | This parameter allows you to define a new custom service to pull data from non-standard DB Views entities.
> ⚠️ The given name to this service must be unique. Having other services with the same name will produce a failure when reading the file.
> ℹ️ This parameter should be removed if it is not used. | > | `table_name` | `str` | `Mandatory` | Minimum length: 1 | This parameter is the name of the ServiceNow table from which to ingest data. | > | `db_view_service_name` | `str` | `Mandatory` | Minimum length: 1 | This parameter is the name of the DB View service. For example: `db_view_1`. | > | `full_poll_every_sec` | `int` | `Mandatory` | Minimum value: 1 | This parameter indicates the number of seconds that pass until the next FULL request | > | `column_names_list` | `list` | `Mandatory` | At least one element in the list | This parameter defines the columns to be pulled from the given table. The expected content is an array of column names following the format below:
``` > column_name: > - > - > - > ```
For example:
``` > column_name: > - inc_number > - inc_sys_created_on > - inc_data_1 > ``` | > | `time_column_name` | `str` | `Mandatory` | Minimum length: 1 | This parameter is the name of the column by which the data will be sorted. For example:
`inc_sys_created_on` | > | `unique_key` | `str` | `Mandatory` | Minimum length: 1 | This parameter is the name of the field that contains the key -> 'id'. For example: `inc_number.` | > | `column_name_types_list` | `list` | `Mandatory` | It has to be the same length as `column_names_list` | This parameter defines the list of types of column names. The most common value is `str`. It has the following form:
``` > column_name_types: > - > - > - > ```
For example:
``` > column_name_types: > - str > - str > - str > ``` | > | `historic_date` | `str` | `Optional` | With format: `YYYY-mm-dd HH:MM:SS.sssZ` | This configuration allows you to set a custom date as the beginning of the period to download. This allows downloading historical data (1 month back for example) before downloading new events.
> ℹ️ This parameter can be removed or commented. | > | `request_period_in_seconds` | `int` | `Optional` | Minimum value: 1 | Period in seconds used between each data pulling, this value will overwrite the default value (300 seconds).
> ℹ️ This parameter can be removed or commented. | > | `override_devo_tag` | `str` | `Optional` | A Devo Tag. For more information see [Devo Tags](https://docs.devo.com/space/latest/95126204/About%20Devo%20tags). | This parameter allows to define a custom devo tag.
> ℹ️ This parameter can be removed or commented. | > | `sysparm_display_value` | `bool` | `Optional` | `true` or `false` | Exports the display value (**true**) or raw value (**false**) for the CSV, Excel, XLSX, and XML file formats.
> 📝 This parameter can be removed or commented. | > | `date_field_value` | `str` | `Optional` | `sys_created_on` or `sys_updated_on` | Allows you to choose between the update date or the creation date to retrieve events. Using the update date, event status changes are shown. > The default value of this parameter is: `sys_created_on`
> 📝 This parameter can be removed or commented. | > > ### Download the Docker image > > The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table: > > | **Collector Docker image** | **SHA-256 hash** | > | --- | --- | > | [collector-servicenow_if-docker-image-1.6.0.tgz](https://drive.google.com/file/d/12EXZXzG4XizCTzVVLDJNkJagpXEXL2bd/view?usp=drive_link) | `1555b10034770db6a4e7be2a2954c0acd72ff60b375be414a3c982a62a3b732b` | > > Use the following command to add the Docker image to the system: > > ``` > gunzip -c -.tgz | docker load > ``` > > > ⚠️ Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace `` and `` with a proper value. > > The Docker image can be deployed on the following services: > > #### Docker > > Execute the following command on the root directory `/devo-collectors//` > > ``` > docker run > --name collector- > --volume $PWD/certs:/devo-collector/certs > --volume $PWD/config:/devo-collector/config > --volume $PWD/state:/devo-collector/state > --env CONFIG_FILE=config.yaml > --rm > --interactive > --tty > : > ``` > > > ⚠️ Replace ``, `` and `` with the proper values. > > #### Docker Compose > > The following Docker Compose file can be used to execute the Docker container. It must be created in the `/devo-collectors//` directory. > > ``` > version: '3' > services: > collector-: > image: :${IMAGE_VERSION:-latest} > container_name: collector- > volumes: > - ./certs:/devo-collector/certs > - ./config:/devo-collector/config > - ./credentials:/devo-collector/credentials > - ./state:/devo-collector/state > environment: > - CONFIG_FILE=${CONFIG_FILE:-config.yaml} > ``` > > To run the container using docker-compose, execute the following command from the `/devo-collectors//` directory: > > ``` > IMAGE_VERSION= docker-compose up -d > ``` > > > ⚠️ Replace ``, `` and `` with the proper values. ## Collector services detail This section is intended to explain how to proceed with specific actions for services.
Predefined Services This section describes the predefined services of the ServiceNow collector. The predefined services can be classified into: - **ServiceNow Standard Table** - **ServiceNow Standard CMDB** ### Devo categorization and destination for Standard Table The following table shows the Devo Tag to which the events are ingested based on each ServiceNow table: | | | | --- | --- | | **ServiceNow Standard Table** | **Devo Tag** | | `incident` | `itsm.servicenow.tables.incident` | | `event` | `itsm.servicenow.tables.event` | | `change` | `itsm.servicenow.tables.change` | | `user` | `itsm.servicenow.tables.user` | | `location` | `itsm.servicenow.tables.location` | ### Devo categorization and destination for Standard CMDB The following table shows the Devo Tag to which the events are ingested based on each ServiceNow table: | | | | --- | --- | | **ServiceNow Standard CMDB** | **Devo Tag** | | `cmdb_ci_app_server` | `itsm.servicenow.cmdb.cmdbciappserver` | | `cmdb_ci_db_instance` | `itsm.servicenow.cmdb.cmdbcidbinstance` | | `cmdb_ci_infra_service` | `itsm.servicenow.cmdb.cmdbciinfraservice` | | `cmdb_ci(list)` | `itsm.servicenow.cmdb.cmdbci` | | `cmdb_ci_server` | `itsm.servicenow.cmdb.cmdbciserver` | | `cmdb_ci_service` | `itsm.servicenow.cmdb.cmdbciservice` | | `cmdb_ci_vm` | `itsm.servicenow.cmdb.cmdbcivm` | | `cmdb_rel_ci` | `itsm.servicenow.cmdb.cmdbrelci` | | `cmn_location` | `itsm.servicenow.cmdb.cmnlocation` |
Verify data collection Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console. This service has the following components: | | | | --- | --- | | **Component** | **Description** | | Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. | | Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. | #### Setup output A successful run has the following output messages for the setup module: ``` INFO ServiceNowDataPullerSetup(example_collector,servicenow_data_puller#111,incident#predefined) -> Puller Setup Started INFO ServiceNowDataPullerSetup(example_collector,servicenow_data_puller#111,incident#predefined) -> Checking if the user has enough permissions to fetch data from ServiceNOW INFO ServiceNowDataPullerSetup(example_collector,servicenow_data_puller#111,incident#predefined) -> Provided username and password have the required permissions to fetch the data INFO ServiceNowDataPullerSetup(example_collector,servicenow_data_puller#111,incident#predefined) -> Puller Setup Terminated INFO ServiceNowDataPullerSetup(example_collector,servicenow_data_puller#111,incident#predefined) -> Setup for module "ServiceNowDataPullerTables" has been successfully executed ``` #### Puller output A successful initial run has the following output messages for the puller module: > ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action. ``` 2022-08-26T13:13:31.485 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> PrePull Started. 2022-08-26T13:13:31.485 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Found Historic Date as: 2008-05-10 05:35:13 2022-08-26T13:13:31.486 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> No saved state found, initializing.... 2022-08-26T13:13:31.486 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Initializing the rate limiter with 100000 calls per second 2022-08-26T13:13:31.486 WARNING InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Saved state loaded: {'last_polled_time': 1210397713.0, 'historic_date': 1210397713.0, 'last_full_refresh': None, 'ids_with_same_timestamp': []} 2022-08-26T13:13:31.486 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> PrePull terminated. 2022-08-26T13:13:31.486 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Starting data collection every 60 seconds 2022-08-26T13:13:31.487 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Pull Started. Retrieving timestamp: 2022-08-26 11:13:31.485261+00:00 2022-08-26T13:13:31.487 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Starting to pull ServiceNow table data is pulled from table incident and send to Devo 2022-08-26T13:13:31.487 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Pulling data from incident table 2022-08-26T13:13:31.487 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Fetching rows from ServiceNow table sorted by 'sys_created_on' time. 2022-08-26T13:13:31.487 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> This is the first run of the collector. Collector will fetch all the data available in the ServiceNow in the first poll 2022-08-26T13:13:33.372 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Successfully retried incident table data from server 2022-08-26T13:13:33.375 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Received 78 events from ServiceNow 2022-08-26T13:13:33.375 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Removing the duplicate events if present 2022-08-26T13:13:33.379 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Sent 78 events to Devo. 2022-08-26T13:13:33.379 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> State last_polled_time is updated with last fetched issue creation time 2022-08-26T13:13:33.379 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Saved state: {'last_polled_time': 1661339716.0, 'historic_date': 1210397713.0, 'last_full_refresh': 1661512411.485261, 'ids_with_same_timestamp': ['2dfb24c907e15110c009f4be7c1ed053']} 2022-08-26T13:13:33.380 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1661508811.485261) so far: Number of requests made: 1; Number of events received: 78; Number of duplicated events filtered out: 0; Number of events generated and sent: 78 2022-08-26T13:13:34.599 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Successfully retried incident table data from server 2022-08-26T13:13:34.600 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Received 0 events from ServiceNow 2022-08-26T13:13:34.600 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Fetching results from ServiceNow is completed. 2022-08-26T13:13:34.600 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1661508811.485261): Number of requests made: 2; Number of events received: 78; Number of duplicated events filtered out: 0; Number of events generated and sent: 78; Average of events per second: 25.04 2022-08-26T13:13:34.600 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Pull Terminated ``` After a successful collector’s execution (that is, no error logs found), you will see the following log message: ``` 2022-08-26T13:13:34.600 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,incident,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1661508811.485261): Number of requests made: 2; Number of events received: 78; Number of duplicated events filtered out: 0; Number of events generated and sent: 78; Average of events per second: 25.04 ``` > ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window. > ℹ️ Note that a `Partial` Statistics Report will be displayed after download a page when the pagination is required to pull all available events. Look for the report without the `Partial` reference.`(Partial) Statistics for this pull cycle (@devo_pulling_id=1661508811.485261) so far: Number of requests made: 1; Number of events received: 78; Number of duplicated events filtered out: 0; Number of events generated and sent: 78(Partial) Statistics for this pull cycle (@devo_pulling_id=1661508438.289114) so far: Number of requests made: 1; Number of events received: 78; Number of duplicated events filtered out: 0; Number of events generated and sent: 78(Partial) Statistics for this pull cycle (@devo_pulling_id=1661507818.035402) so far: Number of requests made: 1; Number of events received: 78; Number of duplicated events filtered out: 0; Number of events generated and sent: 78`
Restart the persistence This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps: 1. Edit the configuration file. 2. Change the value of the `historic_date` parameter to a different one. 3. Save the changes. 4. Restart the collector. The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided. > ⚠️ Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
Troubleshooting This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors. | | | | | | | --- | --- | --- | --- | --- | | **Error Type** | **Error Id** | **Error Message** | **Cause** | **Solution** | | InitVariablesError | 0 | Required setting. historic_date_timeformat is not of expected type: str | This error is raised when historic_date_timeformat is defined in collector_definitions.yaml but the format is not str. | This is an internal issue. Contact with Devo Support team. | | InitVariablesError | 1 | Invalid value is provided for the historic_date. Provide the date in %Y-%m-%d %H:%M:%S format | This error is raised when optional value historic_date does not match the required regex. | Make the value of historic_date in config.json match the indicated format. Or leave it empty so the collector starts pulling data N days ago at the current time. | | InitVariablesError | 2 | Time format for historic date must be %Y-%m-%d %H:%M:%S. e.g. 2015-03-15 05:35:13 | This error is raised when optional value historic_date does not match the required regex. | Make the value of historic_date in config.json match the indicated format. Or leave it empty so the collector starts pulling data N days ago at the current time. | | InitVariablesError | 3 | Required setting, sysparm_display_value not of expected type: bool | This error is raised when sysparm_display_value is defined in collector_definitions.yaml but the format is not bool. | This is an internal issue. Contact with Devo Support team. | | InitVariablesError | 4 | 'table_name' is the required field. Specify it in config.yaml | This error is raised when the required property table_name is not found in config.json. | Add table_name property in config.json. | | InitVariablesError | 5 | Required setting. 'table_name' is not of expected type: str | This error is raised when table_name is defined in config.json but the format is not str. | Edit the value of table_name in config.json so it is of type str. | | InitVariablesError | 6 | 'override_devo_tag' is the required field. Specify it in config.yaml | This error is raised when override_devo_tag is not found in config.json | Add override_devo_tag property in config.json. | | InitVariablesError | 7 | Required setting. 'devo_tag' is not of expected type: str | This error is raised when devo_tag is defined in collector_definitions.yaml but the format is not str. | This is an internal issue. Contact with Devo Support team. | | ApiError | 400 | Error occurred while generating token for serviceNow. | Wrong credentials or something else from API side. | Check the credentials and ensure that the collector has the necessary permissions to access the SerivceNow API. | | ApiError | 401 | The credentials provided in the config.yaml are incorrect. Please provide the correct credentials. Status code: 401 Error message: | This error is raised when there is a problem with the credentials. | Add the correct username and password fields in config.json, at credentials property. | | ApiError | 402 | The user is not authorized to {table} table data from the ServiceNOW. Status code: 403 Error message: | This error is raised when the credentials provided in config.json are valid, but the credentials used to access the API have no permissions to create a token. (403 Forbidden error) | Add some credentials in config.json with enough privileges to create a token from the API. | | ApiError | 403 | The requested URL {response.url} is not found. The URL may have been depreciated. Status code: 404 Error message: | Wrong credentials or something else from API side. | Check the credentials and ensure that the collector has the necessary permissions to access the Netskope API. | | ApiError | 404 | Unexpected error occurred while checking for permissions of user credentials. | This error is raised when there is a problem with the credentials. | Add the correct username and password fields in config.json, at credentials property. | | ApiError | 405 | After {max_number_of_retries} retries still getting the too many requests error. | This error is raised when API requests fail after a number of attempts. | This is an internal issue. Contact with Devo Support team. | | ApiError | 406 | Connection error occurred. Collector is not able to reach the Internet. Error message: | This error is raised when when the connection takes a long time. | Check that you have an internet connection | | ApiError | 407 | API request timed out. Error message: | This error is raised when an unknown problem occurs with the request. | This is an internal issue. Contact with Devo Support team. | | ApiError | 408 | Error occurred while fetching {table} table data from serviceNow. Error message: | This error is raised when an unknown problem occurs with the request. | This is an internal issue. Contact with Devo Support team. | | SetupError | 100 | Failed to check if the provided credentials have valid permissions. | This error is raised when an unexpected error occurred while checking credentials | If there are previous errors, fix them and if the error continues to appear contact with Devo Support team. | | SetupError | 101 | The credentials provided in the config.yaml are incorrect. Please provide the correct credentials. Status code: 401 Error message: | This error is raised when the credentials provided in config.json are not valid. (401 Unauthorized error) | Add the correct username and password fields in config.json, at credentials property. | | SetupError | 102 | The user is not authorized to fetch data from the ServiceNOW. Status code: 403 Error message: | This error is raised when the credentials provided in config.json are valid, but the credentials used to access the API have no permissions to create a token. (403 Forbidden error) | Add some credentials in config.json with enough privileges to create a token from the API. | | SetupError | 103 | The requested URL {response.url} is not found. The URL may have been depreciated" | This error is raised when the credentials provided in config.json are valid, but the authentication endpoint that is being requested to get a token is not found. | This is an internal issue. Contact with Devo Support team. | | SetupError | 104 | Bad request. Check the message from the server. Error Message: | This error is raised when the api returns a 400 code that indicates that the server cannot or will not process the request due to something that is perceived to be a client error. | Check if exists malformed request syntax, invalid request message framing, or deceptive request routing) | | SetupError | 105 | Servicenow instance in Hibernation mode. Sign in to ServiceNow and wake up the instance. | This error is raised when ServiceNow instance went into hibernation due to inactivity | Go to the ServiceNow developers website and wake up the instance. | | SetupError | 106 | Unexpected error occurred while checking for permissions of user credentials | This error is raised when there is a problem with the credentials. | Add the correct username and password fields in config.json, at credentials property. | | SetupError | 107 | Servicenow API is not returning a JSON response, check the API url and endpoint. | The error is raised when the API response type is not JSON, usually being HTML. | Check the api_base_url in the user config and make sure it is correct. | | PrePullError | 200 | Kindly enter valid date which is less than the current date | This error is raised when historic_date is later than the current date. | Go to the config.yaml and enter a historic_date less than the current date |
## Custom services tables and CMDB This section describes the custom services of the ServiceNow collector.
Predefined services The predefined services are classified into: - **ServiceNow Custom Table** - **Servicenow Custom CMDB** ### Devo categorization and destination for Custom Table This service is a custom service so the user is in charge of choosing a ServiceNow CMDB (`custom_servicenow_cmdb`) and also the Devo destination tag (`custom_devo_tag`). The following table shows the Devo table to which the events are ingested based on each ServiceNow table: | **ServiceNow Standard Table** | **Devo Tag** | | --- | --- | | `custom_servicenow_table` | `custom_devo_tag` | ### Devo categorization and destination for Custom CMDB This service is a custom service so the user is in charge of choosing a ServiceNow CMDB (`custom_servicenow_cmdb`) and also the Devo destination tag (`custom_devo_tag`). The following table shows the Devo table to which the events are ingested based on each ServiceNow table: | | | | --- | --- | | **ServiceNow Standard Table** | **Devo Tag** | | `custom_servicenow_cmdb` | `custom_devo_tag` |
Verify data collection Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console. This service has the following components: | | | | --- | --- | | **Component** | **Description** | | Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. | | Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. | #### Setup output A successful run has the following output messages for the setup module: ``` INFO InputProcess::ServiceNowDataPullerSetup(example_collector,servicenow#111,custom_tables#custom) -> Puller Setup Started INFO InputProcess::ServiceNowDataPullerSetup(example_collector,servicenow#111,custom_tables#custom) -> Checking if the user has enough permissions to fetch data from ServiceNOW INFO InputProcess::ServiceNowDataPullerSetup(example_collector,servicenow#111,custom_tables#custom) -> Provided username and password have the required permissions to fetch the data INFO InputProcess::ServiceNowDataPullerSetup(example_collector,servicenow#111,custom_tables#custom) -> Puller Setup Terminated INFO InputProcess::ServiceNowDataPullerSetup(example_collector,servicenow#111,custom_tables#custom) -> Setup for module has been successfully executed ``` #### Puller output A successful initial run has the following output messages for the puller module: > ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action. ``` 2022-08-26T13:16:46.933 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> PrePull Started. 2022-08-26T13:16:46.933 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Found Historic Date as: 2008-05-10 05:35:13 2022-08-26T13:16:46.933 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> No saved state found, initializing.... 2022-08-26T13:16:46.934 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Initializing the rate limiter with 100000 calls per second 2022-08-26T13:16:46.934 WARNING InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Saved state loaded: {'last_polled_time': 1210397713.0, 'historic_date': 1210397713.0, 'last_full_refresh': None, 'ids_with_same_timestamp': []} 2022-08-26T13:16:46.934 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> PrePull terminated. 2022-08-26T13:16:46.934 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Starting data collection every 600 seconds 2022-08-26T13:16:46.934 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Pull Started. Retrieving timestamp: 2022-08-26 11:16:46.933235+00:00 2022-08-26T13:16:46.935 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Starting to pull ServiceNow table data is pulled from table incident and send to Devo 2022-08-26T13:16:46.935 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Pulling data from incident table 2022-08-26T13:16:46.935 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Fetching rows from ServiceNow table sorted by 'sys_created_on' time. 2022-08-26T13:16:46.935 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> This is the first run of the collector. Collector will fetch all the data available in the ServiceNow in the first poll 2022-08-26T13:16:48.976 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Successfully retried incident table data from server 2022-08-26T13:16:48.978 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Received 78 events from ServiceNow 2022-08-26T13:16:48.978 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Removing the duplicate events if present 2022-08-26T13:16:48.980 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Sent 78 events to Devo. 2022-08-26T13:16:48.981 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> State last_polled_time is updated with last fetched issue creation time 2022-08-26T13:16:48.981 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Saved state: {'last_polled_time': 1661339716.0, 'historic_date': 1210397713.0, 'last_full_refresh': 1661512606.933235, 'ids_with_same_timestamp': ['2dfb24c907e15110c009f4be7c1ed053']} 2022-08-26T13:16:48.981 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1661509006.933235) so far: Number of requests made: 1; Number of events received: 78; Number of duplicated events filtered out: 0; Number of events generated and sent: 78 2022-08-26T13:16:50.305 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Successfully retried incident table data from server 2022-08-26T13:16:50.306 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Received 0 events from ServiceNow 2022-08-26T13:16:50.306 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Fetching results from ServiceNow is completed. 2022-08-26T13:16:50.306 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Statistics for this pull cycle (@devo_pulling_id=1661509006.933235): Number of requests made: 2; Number of events received: 78; Number of duplicated events filtered out: 0; Number of events generated and sent: 78; Average of events per second: 23.12 2022-08-26T13:16:50.306 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Pull Terminated ``` After a successful collector’s execution (that is, no error logs found), you will see the following log message: ``` 2022-08-26T13:16:50.306 INFO InputProcess::ServiceNowDataPullerTables(servicenow,111,custom_tables,custom) -> Statistics for this pull cycle (@devo_pulling_id=1661509006.933235): Number of requests made: 2; Number of events received: 78; Number of duplicated events filtered out: 0; Number of events generated and sent: 78; Average of events per second: 23.12 ``` > ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window. > ℹ️ Note that a `Partial` Statistics Report will be displayed after download a page when the pagination is required to pull all available events. Look for the report without the `Partial` reference. > ℹ️ > ℹ️ `(Partial) Statistics for this pull cycle (@devo_pulling_id=1661509006.933235) so far: Number of requests made: 1; Number of events received: 78; Number of duplicated events filtered out: 0; Number of events generated and sent: 78`
Restart the persistence This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps: 1. Edit the configuration file. 2. Change the value of the `historic_date` parameter to a different one. 3. Save the changes. 4. Restart the collector. The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided. > ⚠️ Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
Troubleshooting This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors. | | | | | | | --- | --- | --- | --- | --- | | **ErrorType** | **Error Id** | **Error Message** | **Cause** | **Solution** | | `InitVariablesError` | `29` | `'table_name' is the required field. Specify it in collector_definitions.yaml` | This error is raised when `table_name` property is not found in `collector_definitions.yaml`. | This is an internal issue. Contact with Devo Support team. | | | `30` | `Required setting. 'table_name' is not of expected type: str` | This error is raised when `table_name` is defined in `collector_definitions.yaml` but the format is not `str`. | This is an internal issue. Contact with Devo Support team. | | | `31` | `'devo_tag' is the required field. Specify it in collector_definitions.yaml` | This error is raised when `devo_tag` property is not found in `collector_definitions.yaml`. | This is an internal issue. Contact with Devo Support team. | | | `32` | `Required setting. 'devo_tag' is not of expected type: str` | This error is raised when `devo_tag` is defined in `collector_definitions.yaml` but the format is not `str`. | This is an internal issue. Contact with Devo Support team. |
## Custom Lookup services database view This section describes the custom services of the **ServiceNow** collector.
Predefined service ### Devo categorization and destination This service is a custom service so the user is in charge of choosing a ServiceNow DatabaseView (`dbview_servicenow`). Devo will send the events to a `custom_tag` The following table shows the Devo table to which the events are ingested based on each ServiceNow table: | **ServiceNow Standard Table** | **Devo Tag** | | --- | --- | | `dbview_servicenow` | `custom_tag` |
Verify data collection Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console. This service has the following components: | | | | --- | --- | | **Component** | **Description** | | Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. | | Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. | #### Setup output A successful run has the following output messages for the setup module: ``` INFO InputProcess::ServiceNowDataPullerSetup(example_collector,servicenow#111,db_view_1#custom) -> Puller Setup Started INFO InputProcess::ServiceNowDataPullerSetup(example_collector,servicenow#111,db_view_1#custom) -> Checking if the user has enough permissions to fetch data from ServiceNOW INFO InputProcess::ServiceNowDataPullerSetup(example_collector,servicenow#111,db_view_1#custom) -> Provided username and password have the required permissions to fetch the data INFO InputProcess::ServiceNowDataPullerSetup(example_collector,servicenow#111,db_view_1#custom) -> Puller Setup Terminated INFO InputProcess::ServiceNowDataPullerSetup(example_collector,servicenow#111,db_view_1#custom) -> Setup for module has been successfully executed ``` #### Puller output A successful initial run has the following output messages for the puller module: > ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action. ``` 2022-08-26T13:19:57.656 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> PrePull Started. 2022-08-26T13:19:57.657 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> Found Historic Date as: 2008-05-10 05:35:13 2022-08-26T13:19:57.657 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> No saved state found, initializing.... 2022-08-26T13:19:57.657 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> Initializing the rate limiter with 100000 calls per second 2022-08-26T13:19:57.657 WARNING InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> Saved state loaded: {'last_polled_time': 1210397713.0, 'historic_date': 1210397713.0, 'last_full_refresh': None, 'ids_with_same_timestamp': []} 2022-08-26T13:19:57.658 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> PrePull terminated. 2022-08-26T13:19:57.658 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> Starting data collection every 60 seconds 2022-08-26T13:19:57.658 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> Pull Started. Retrieving timestamp: 2022-08-26 11:19:57.656489+00:00 2022-08-26T13:19:57.658 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> Requesting data to remote 2022-08-26T13:19:57.659 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> Fetching rows from ServiceNow table sorted by inc_sys_created_on time. 2022-08-26T13:19:57.659 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> This is the first run of the collector or 172800 seconds of time is elapsed since the last 'FULL poll. Fetching all the data from ServiceNow in this poll. 2022-08-26T13:19:59.063 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> Successfully retried incident_metric table data from server 2022-08-26T13:19:59.064 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> Received 83 events from ServiceNow 2022-08-26T13:20:00.480 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> Successfully retried incident_metric table data from server 2022-08-26T13:20:00.481 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> Received 0 events from ServiceNow 2022-08-26T13:20:00.481 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> Total events received: 83 2022-08-26T13:20:00.481 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> Processing and cleaning received data 2022-08-26T13:20:00.481 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> 83 rows retrieved and cleaned 2022-08-26T13:20:00.487 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> 83 lookups sent 2022-08-26T13:20:00.488 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> State last_polled_time is updated with 2022-08-26 11:19:57.656489+00:00 time 2022-08-26T13:20:00.488 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> Saved state: {'last_polled_time': 1661512797.656489, 'historic_date': 1210397713.0, 'last_full_refresh': 1661512797.656489, 'ids_with_same_timestamp': []} 2022-08-26T13:20:00.488 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> Statistics for this pull cycle (@devo_pulling_id=1661509197.656489): Number of requests made: 2; Number of events received: 83; Number of duplicated events filtered out: 0; Number of events generated and sent: 83; Average of events per second: 29.31 2022-08-26T13:20:00.488 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> Pull Terminated ``` After a successful collector’s execution (that is, no error logs found), you will see the following log message: ``` 2022-08-26T13:20:00.488 INFO InputProcess::ServiceNowDataPullerDatabaseView(servicenow,111,db_view_1,custom) -> Statistics for this pull cycle (@devo_pulling_id=1661509197.656489): Number of requests made: 2; Number of events received: 83; Number of duplicated events filtered out: 0; Number of events generated and sent: 83; Average of events per second: 29.31 ``` > ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.
Restart the persistence This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps: 1. Edit the configuration file. 2. Change the value of the `historic_date` parameter to a different one. 3. Save the changes. 4. Restart the collector. The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided. > ⚠️ Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
## Collector operations This section is intended to explain how to proceed with specific operations of this collector.
Verify collector operations ### Initialization The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration. A successful run has the following output messages for the initializer module: ``` INFO MainThread -> (CollectorMultithreadingQueue) standard_queue_multithreading -> max_size_in_messages: 10000, max_size_in_mb: 1024, max_wrap_size_in_items: 100 WARNING MainThread -> [OUTPUT] OutputLookupSenders -> setting has been modified from 1.1 to 1.0 due to this configuration increases the Lookup sender performance. INFO MainThread -> [OUTPUT] OutputMultithreadingController(example_collector) -> Starting thread INFO MainThread -> [OUTPUT] DevoSender(standard_senders,devo_sender_0) -> Starting thread INFO MainThread -> [OUTPUT] DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 600 seconds) INFO MainThread -> [OUTPUT] DevoSenderManager(standard_senders,manager,devo_1)(devo_1) -> Starting thread INFO MainThread -> [OUTPUT] DevoSender(lookup_senders,devo_sender_0) -> Starting thread INFO MainThread -> [OUTPUT] DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 600 seconds) INFO MainThread -> [OUTPUT] DevoSenderManager(lookup_senders,manager,devo_1)(devo_1) -> Starting thread INFO MainThread -> 'InitVariables Started' INFO MainThread -> 'InitVariables Terminated' INFO MainThread -> [INPUT] InputMultithreadingController(example_collector) - Starting thread INFO MainThread -> [INPUT] InputThread(example_collector,servicenow_data_puller#111) - Starting thread INFO MainThread -> [INPUT] ServiceThread(example_collector,servicenow_data_puller#111,incident#predefined) - Starting thread INFO MainThread -> [SETUP] ServiceNowDataPullerSetup(example_collector,servicenow_data_puller#111,incident#predefined) - Starting thread INFO MainThread -> [INPUT] ServiceNowDataPullerTables(example_collector,servicenow_data_puller#111,incident#predefined) - Starting thread INFO ServiceNowDataPullerSetup(example_collector,servicenow_data_puller#111,incident#predefined) -> Puller Setup Started WARNING ServiceNowDataPullerTables(example_collector,servicenow_data_puller#111,incident#predefined) -> Waiting until setup will be executed INFO ServiceNowDataPullerSetup(example_collector,servicenow_data_puller#111,incident#predefined) -> Checking if the user has enough permissions to fetch data from ServiceNOW INFO MainThread -> {"collector_name": "example_collector", "collector_version": "5.1.3", "collector_owner": "", "started_at": ""} INFO DevoSender(standard_senders,devo_sender_0) -> DevoSender(standard_senders,devo_sender_0) -> Created a sender: {"group_name": "standard_senders", "instance_name": "devo_sender_0", "url": "collector-example...", ...} ``` ### Events delivery and Devo ingestion The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method. A successful run has the following output messages for the initializer module: ``` INFO MainThread -> [OUTPUT] DevoSender(standard_senders,devo_sender_0) -> Starting thread INFO MainThread -> [OUTPUT] DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 600 seconds) INFO MainThread -> [OUTPUT] DevoSenderManager(standard_senders,manager,devo_1)(devo_1) -> Starting thread INFO MainThread -> [OUTPUT] DevoSender(lookup_senders,devo_sender_0) -> Starting thread INFO MainThread -> [OUTPUT] DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 600 seconds) INFO MainThread -> [OUTPUT] DevoSenderManager(lookup_senders,manager,devo_1)(devo_1) -> Starting thread INFO DevoSender(standard_senders,devo_sender_0) -> DevoSender(standard_senders,devo_sender_0) -> Created a sender: {"group_name": "standard_senders", "instance_name": "devo_sender_0", "url": "...} ``` > ℹ️ By default, these information traces will be displayed every 10 minutes. #### Sender services The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (`internal`, `standard`, and `lookup`). This collector uses the following Sender Services: | | | | --- | --- | | **Sender Services** | **Description** | | `internal_senders` | In charge of delivering internal metrics to Devo such as logging traces or metrics. | | `standard_senders` | In charge of delivering pulled events to Devo. | | `lookup_senders` | In charge of delivering lookups to devo | #### Sender statistics Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type: | | | | --- | --- | | **Logging trace** | **Description** | | `Number of available senders: 1` | Displays the number of concurrent senders available for the given Sender Service. | | `sender manager internal queue size: 0` | Displays the items available in the internal sender queue.
> ℹ️ This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders. | | `Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds)` | Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:
- 44 events were sent to Devo since the collector started. - The last checkpoint timestamp was `2022-06-28 10:39:22.511671+00:00`. - 21 events where sent to Devo between the last UTC checkpoint and now. - Those 21 events required `0.007 seconds` to be delivered.
> ℹ️ By default these traces will be shown every 10 minutes. |
Check memory usage To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process. - The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector. - The global pressure of the available memory is displayed in the `global` value. - All metrics (Global, RSS, VMS) include the value before freeing and after `previous -> after freeing memory` ``` INFO InputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(34.50MiB -> 34.08MiB), VMS(410.52MiB -> 410.02MiB) INFO OutputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(28.41MiB -> 28.41MiB), VMS(705.28MiB -> 705.28MiB) ``` > ℹ️ Differences between `RSS` and `VMS` memory usage: > ℹ️ > ℹ️ - `RSS` is the Resident Set Size, which is the actual physical memory the process is using > ℹ️ - `VMS` is the Virtual Memory Size which is the virtual memory that process is using
Enable/disable the logging debug mode Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes. - To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector. - To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector. For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode.
Troubleshooting This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors. #### Common errors for all services | **Error type** | **Error ID** | **Error message** | **Cause** | **Solution** | | --- | --- | --- | --- | --- | | `InitVariablesError` | `1` | `api_base_url not of expected type: str` | This error is raised when `api_base_url` is not `str`. | Edit the value of `api_base_url` in `config.json` so it is of type `str`. | | `2` | `api_base_url must match regex: (^http|^https)(:\/\/)[-a-zA-Z0-9@:%._\\+~#=]{1,256}(\.)(service[-]?now)(\.com?$). (e.g.https://dev70590.service-now.com )` | This error is raised when optional value `api_base_url` does not match the required regex. | Make the value of `api_base_url` in `config.json` match the indicated format. | | `3` | `api_url_regex is the required field for validating the base url. Specify it in collector definitions` | This error is raised when `api_url_regex` property is not found in `collector_definitions.yaml`. | This is an internal issue. Contact with Devo Support team. | | `4` | `Required setting. 'api_url_regex' is not of expected type: str` | This error is raised when `api_url_regex` is defined in `collector_definitions.yaml` but the format is not `str`. | This is an internal issue. Contact with Devo Support team. | | `5` | `historic_date_timeformat is the required field for validating datetime format. Specify it in collector definitions` | This error is raised when `historic_date_timeformat` property is not found in `collector_definitions.yaml`. | This is an internal issue. Contact with Devo Support team. | | `6` | `Required setting. historic_date_timeformat is not of expected type: str` | This error is raised when `historic_date_timeformat` is defined in `collector_definitions.yaml` but the format is not `str`. | This is an internal issue. Contact with Devo Support team. | | `7` | `Required setting, historic_date not of expected type: str` | This error is raised when optional value `historic_date` added in `config.json` is not of type `str`. | Edit the value of `historic_date` in `config.json` so it is of type `str`. Or leave it empty so the collector starts pulling data N days ago at the current time. | | `8` | `Invalid value is provided for the historic_date. Provide the date in %Y-%m-%d %H:%M:%S format` | This error is raised when optional value `historic_date` does not match the required regex. | Make the value of `historic_date` in `config.json` match the indicated format. Or leave it empty so the collector starts pulling data N days ago at the current time. | | `9` | `Time format for historic date must be %Y-%m-%d %H:%M:%S. e.g.` `2015-03-15 05:35:13` | This error is raised when optional value `historic_date` does not match the required regex. | Make the value of `historic_date` in `config.json` match the indicated format. Or leave it empty so the collector starts pulling data N days ago at the current time. | | `10` | `Required setting, credentials not found in user configuration` | This error is raised when the required property `credentials` is not found in `config.json`. | Add `credentials` dictionary in `config.json`, including `client_id` and `client_secret` fields. | | `11` | `Required setting, credentials not of expected type: dict` | This error is raised when `credentials` is defined in `config.json` but the format is not `dict`. | Edit the value of `credentials` in `config.json` so it is of type `dict`. | | `12` | `Required setting, username not found in user configuration` | This error is raised when the required property `username` is not found in `config.json`, into `credentials` dictionary. | Add `username` property in `config.json`, into `credentials` dictionary. | | `13` | `Required setting, username not of expected type: str` | This error is raised when `username` is defined in `config.json` but the format is not `str`. | Edit the value of `username` in `config.json`, into `credentials` dictionary, so it is of type `str`. | | `14` | `Required setting 'password' not found in user configuration` | This error is raised when the required property `password` is not found in `config.json`, into `credentials` dictionary. | Add `password` property in `config.json`, into `credentials` dictionary. | | `15` | `Required setting 'password' not of expected type: str` | This error is raised when `password` is defined in `config.json` but the format is not `str`. | Edit the value of `password` in `config.json`, into `credentials` dictionary, so it is of type `str`. | | `16` | `'event_fetch_limit_in_items' is the required setting. Please specify it in collector definitions` | This error is raised when `event_fetch_limit_in_items` property is not found in `collector_definitions.yaml`. | This is an internal issue. Contact with Devo Support team. | | `17` | `'event_fetch_limit_in_items' is not of expected type: int` | This error is raised when `event_fetch_limit_in_items` is defined in `collector_definitions.yaml` but the format is not `int`. | This is an internal issue. Contact with Devo Support team. | | `18` | `'event_fetch_limit_in_items' should be in the range 1 to 10000` | This error is raised when `event_fetch_limit_in_items` is defined in `collector_definitions.yaml` but the format is not in range 1 to 10000. | This is an internal issue. Contact with Devo Support team. | | `19` | `'user_agent' is the required setting. Please specify it in collector definitions` | This error is raised when `user_agent` property is not found in `collector_definitions.yaml`. | This is an internal issue. Contact with Devo Support team. | | `20` | `'user_agent' is not of expected type: str` | This error is raised when `user_agent` is defined in `collector_definitions.yaml` but the format is not `str`. | This is an internal issue. Contact with Devo Support team. | | `21` | `default_historical_poll_days is the required field in case historic_date_utc is not specified. Specify it in collector definitions` | This error is raised when `default_historical_poll_days` property is not found in `collector_definitions.yaml`. | This is an internal issue. Contact with Devo Support team. | | `22` | `Required setting. default_historical_poll_days is not of expected type: int` | This error is raised when `default_historical_poll_days` is defined in `collector_definitions.yaml` but the format is not `int`. | This is an internal issue. Contact with Devo Support team. | | `23` | `'retry_after_sec' is the required setting. Please specify it in collector definitions` | This error is raised when `retry_after_sec` property is not found in `collector_definitions.yaml`. | This is an internal issue. Contact with Devo Support team. | | `24` | `'retry_after_sec' is not of expected type: int` | This error is raised when `retry_after_sec` is defined in `collector_definitions.yaml` but the format is not `int`. | This is an internal issue. Contact with Devo Support team. | | `33` | `max_number_of_retries is the required field. Specify it in collector definitions` | This error is raised when `max_number_of_retries` property is not found in `collector_definitions.yaml`. | This is an internal issue. Contact with Devo Support team. | | `34` | `Required setting. max_number_of_retries is not of expected type: int` | This error is raised when `max_number_of_retries` is defined in `collector_definitions.yaml` but the format is not `int`. | This is an internal issue. Contact with Devo Support team. | | `35` | `api_timeout_in_seconds is the required field. Specify it in collector definitions` | This error is raised when `api_timeout_in_seconds` property is not found in `collector_definitions.yaml`. | This is an internal issue. Contact with Devo Support team. | | `36` | `Required setting. api_timeout_in_seconds is not of expected type: int` | This error is raised when `api_timeout_in_seconds` is defined in `collector_definitions.yaml` but the format is not `int`. | This is an internal issue. Contact with Devo Support team. | | `SetupError` | `100` | `Failed to check if the provided credentials have valid permissions.` | This error is raised when an unexpected error occurred while checking credentials | If there are previous errors, fix them and if the error continues to appear contact with Devo Support team. | | `101` | `The credentials provided in the config.yaml are incorrect. Please provide the correct credentials.`
`Status code: 401`
`Error message: ` | This error is raised when the credentials provided in `config.json` are not valid. (401 Unauthorized error) | Add the correct `username` and `password` fields in `config.json`, at `credentials` property. | | `102` | `The user is not authorized to fetch data from the ServiceNOW.`
`Status code: 403`
`Error message: ` | This error is raised when the credentials provided in `config.json` are valid, but the credentials used to access the API have no permissions to create a token. (403 Forbidden error) | Add some credentials in `config.json` with enough privileges to create a token from the API. | | `103` | `The requested URL {response.url} is not found. The URL may have been depreciated" ` `Status code: 404`
`Error message: ` | This error is raised when the credentials provided in `config.json` are valid, but the authentication endpoint that is being requested to get a token is not found. | This is an internal issue. Contact with Devo Support team. | | `104` | `Bad request. Check the message from the server.`
`Error Message: ` | This error is raised when the api returns a 400 code that indicates that the server cannot or will not process the request due to something that is perceived to be a client error. | Check if exists malformed request syntax, invalid request message framing, or deceptive request routing) | | `105` | `Servicenow instance in Hibernation mode. Sign in to ServiceNow and wake up the instance.` | This error is raised when ServiceNow instance went into hibernation due to inactivity | Go to the ServiceNow developers website and wake up the instance. | | `106` | `Unexpected error occurred while checking for permissions of user credentials` | This error is raised when there is a problem with the credentials. | Add the correct `username` and `password` fields in `config.json`, at `credentials` property. | | | `107` | `Servicenow API is not returning a JSON response, check the API url and endpoint.` | The error is raised when the API response type is not JSON, usually being HTML. | Check the `api_base_url` in the user config and make sure it is correct. | | `PrePullError` | `200` | `Kindly enter valid date which is less than the current date` | This error is raised when `historic_date` is later than the current date. | Go to the `config.yaml `and enter a `historic_date` less than the current date | | `ApiError` | `400` | `Error occurred while fetching {table} table data from serviceNow." ` `Error message: ` | This error is raised when an unknown problem occurs with the request. | This is an internal issue. Contact with Devo Support team. | | `401` | `The credentials provided in the config.yaml are incorrect. Please provide the correct credentials.`
`Status code: 401`
`Error message: ` | This error is raised when there is a problem with the credentials. | Add the correct `username` and `password` fields in `config.json`, at `credentials` property. | | `402` | `The user is not authorized to {table} table data from the ServiceNOW.`
`Status code: 403`
`Error message: ` | This error is raised when the credentials provided in `config.json` are valid, but the credentials used to access the API have no permissions to create a token. (403 Forbidden error) | Add some credentials in `config.json` with enough privileges to create a token from the API. | | `403` | `The requested URL {response.url} is not found. The URL may have been depreciated`.
`Status code: 404`
`Error message: ` | This error is raised when the credentials provided in `config.json` are valid, but the authentication endpoint that is being requested to get a token is not found. | This is an internal issue. Contact with Devo Support team. | | `404` | `Unexpected error occurred while checking for permissions of user credentials` | This error is raised when there is a problem with the credentials. | Add the correct `username` and `password` fields in `config.json`, at `credentials` property. | | `405` | `After {max_number_of_retries} retries still getting the too many requests error.` | This error is raised when API requests fail after a number of attempts. | This is an internal issue. Contact with Devo Support team. | | `406` | `Connection error occurred. Collector is not able to reach the Internet.` `Error message: ` | This error is raised when when the connection takes a long time. | Check that you have an internet connection. | | `407` | `Error occurred while fetching {table} table data from serviceNow. ` `Error message: ` | This error is raised when an unknown problem occurs with the request. | This is an internal issue. Contact with Devo Support team. | #### Custom errors for Lookup Database View Service | **Error type** | **Error ID** | **Error message** | **Cause** | **Solution** | | --- | --- | --- | --- | --- | | `InitVariablesError` | `37` | `'full_poll_every_sec' is the required field. Specify it in config.yaml` | This error is raised when the required property `full_poll_every_sec` is not found in `config.json`. | Add `full_poll_every_sec` property in `config.json`. | | `38` | `Required setting. 'full_poll_every_sec' is not of expected type: int` | This error is raised when `full_poll_every_sec` is defined in `config.json` but the format is not `int`. | Edit the value of `full_poll_every_sec` in `config.json` so it is of type `int`. | | `39` | `'column_name' is the required field. Specify it in config.yaml` | This error is raised when the required property `column_name` is not found in `config.json`. | Add `column_name` property in `config.json`. | | `40` | `Required setting. column names should be of type: str` | This error is raised when `column_name` is defined in `config.json` but the format is not `str`. | Edit the value of `column_name` in `config.json` so it is of type `str`. | | `41` | `'unique_key' is the required field. Specify it in config.yaml` | This error is raised when the required property `unique_key` is not found in `config.json`. | Add `unique_key` property in `config.json`. | | `42` | `Required setting. 'unique_key' is not of expected type: str` | This error is raised when `unique_key` is defined in `config.json` but the format is not `str`. | Edit the value of `unique_key` in `config.json` so it is of type `str`. | | `43` | `'column_name_types' is the required field. Specify it in config.yaml` | This error is raised when the required property `column_name_types` is not found in `config.json`. | Add `column_name_types` property in `config.json`. | | `44` | `Required setting. column name types should be of type: str` | This error is raised when `column_name_types` is defined in `config.json` but the format is not `str`. | Edit the value of `column_name_types` in `config.json` so it is of type `str`. | | `45` | `'table_name' is the required field. Specify it in config.yaml` | This error is raised when the required property `table_name` is not found in `config.json`. | Add `table_name` property in `config.json`. | | `46` | `Required setting. 'table_name' is not of expected type: str` | This error is raised when `table_name` is defined in `config.json` but the format is not `str`. | Edit the value of `table_name` in `config.json` so it is of type `str`. | | `47` | `'time_column' is the required field. Specify it in config.yaml` | This error is raised when the required property `time_column` is not found in `config.json`. | Edit the value of `time_column` in `config.json` so it is of type `str`. | | `48` | `Required setting. 'time_column' is not of expected type: str` | This error is raised when `time_column` is defined in `config.json` but the format is not `str`. | Edit the value of `time_column` in `config.json` so it is of type `str`. | ### Custom errors for standard / custom tables and CMDB tables service | **Error type** | **Error ID** | **Error message** | **Cause** | **Solution** | | --- | --- | --- | --- | --- | | `InitVariablesError` | `29` | `'table_name' is the required field. Specify it in collector_definitions.yaml` | This error is raised when `table_name` property is not found in `collector_definitions.yaml`. | This is an internal issue. Contact with Devo Support team. | | | `30` | `Required setting. 'table_name' is not of expected type: str` | This error is raised when `table_name` is defined in `collector_definitions.yaml` but the format is not `str`. | This is an internal issue. Contact with Devo Support team. | | | `31` | `'devo_tag' is the required field. Specify it in collector_definitions.yaml` | This error is raised when `devo_tag` property is not found in `collector_definitions.yaml`. | This is an internal issue. Contact with Devo Support team. | | | `32` | `Required setting. 'devo_tag' is not of expected type: str` | This error is raised when `devo_tag` is defined in `collector_definitions.yaml` but the format is not `str`. | This is an internal issue. Contact with Devo Support team. |
## Change log | | | | | | | --- | --- | --- | --- | --- | | **Release** | **Released on** | **Release type** | **Details** | **Recommendations** | | `v2.0.0` | 2/11/2025 | [changed] | **Changed**
- Migrated API to v2 version and Implemented OAUTH. | `Recommended version` | | `v1.6.0` | 12/2/2024 | [IMPROVEMENTS] [SECURITY] | **Changed**
- Upgrade DCSDK to `v1.13.1` - Upgrade Docker image base to version `v1.3.1` in Dockerfile
**Security**
- Remove vulnerabilities in `libexpat1`, `expat` | `Update` | | `v1.5.1` | 8/9/2024 | [IMPROVEMENTS][BUG FIXES] | **Bug fixes**
- Made `date_field` param optional in the config.
**Improvements**
- User can fetch logs based on updated time. - Handled the exception of API returning HTML response instead of JSON. | `Update` | | `v1.5.0` | 2/2/2024 | [IMPROVEMENTS] | **Improvements**
- Upgraded DCSDK from 1.11.1 to 1.12.2: - Updated DevoSDK to v5.1.9 - Fixed some bug related to development on MacOS - Added an extra validation and fix when the DCSDK receives a wrong timestamp format - Added an optional config property for use the Syslog timestamp format in a strict way - Updated DevoSDK to v5.1.10 - Fix for SyslogSender related to UTF-8 - Enhace of troubleshooting. Trace Standardization, Some traces has been introduced. - Introduced a mechanism to detect "Out of Memory killer" situation - Fixed high vulnerability in Docker Image - Upgrade DevoSDK dependency to version v5.4.0 - Fixed error in persistence system - Applied changes to make DCSDK compatible with MacOS - Added new sender for relay in house + TLS - Added persistence functionality for gzip sending buffer - Added Automatic activation of gzip sending - Improved behaviour when persistence fails - Upgraded DevoSDK dependency - Fixed console log encoding - Restructured python classes - Improved behaviour with non-utf8 characters - Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB) - New persistence format/structure (compression in some cases) - Removed dmesg execution (It was invalid for docker execution) - Updated dcsdk-docker-base-image to version 1.3.0 | `Update` | | `v1.4.0` | 2/2/2024 | [FEATURE][IMPROVEMENTS] | **New features**
- Added new config parameter `date_field`: Allows events to be retrieved from their update date instead of from the creation date, thus showing event status changes.
**Improvements**
- **Upgraded DCSDK from 1.9.2 to 1.10.2:** - Fixed error in pyproject.toml related to project scripts endpoint - Introduced pyproject.toml - Added requirements.dev.txt - Added input metrics - Modified ouutput metrics - Updated DevoSDK to version 5.1.6 - Standardized exception messages for traceability - Added more detail in queue statistics | `Update` | | `v1.3.0` | 9/13/2023 | [FEATURE]
[IMPROVEMENTS] | **New Features**
- Added new config parameter `sysparm_display_value`: Exports the display value (true) or raw value (false) for the CSV, Excel, XLSX, and XML file formats.
**Improvements**
- Upgraded DCSDK from 1.9.1 to 1.9.2: - Upgrade internal dependencies | `Update` | | `v1.1.0` | 9/2/2022 | [FEATURE] | **New Features:**
- All services are now compatible with on-premise Service Now instances. | `Update` | | `v1.0.0` | 8/26/2022 | [FEATURE] | **New Features:**
- The following ServiceNow `Standard Tables` are ingested natively: - `incident` - `event` - `change` - `user` - `location` - The following ServiceNow `Standard CMDB Tables` are ingested natively: - `cmdb_ci_app_server` - `cmdb_ci_db_instance` - `cmdb_ci_infra_service` - `cmdb_ci(list)` - `cmdb_ci_server` - `cmdb_ci_service` - `cmdb_ci_vm` - `cmdb_rel_ci` - `cmn_location` - Any custom ServiceNow `Standard Table`, `CMDB Table` and `DatabaseView` and can be ingested through a new two dynamic services. | - |