---
title: "GitHub collector"
canonical: "https://docs.devo.com/space/latest/94656202/GitHub%20collector"
format: markdown
---
> Macro (toc)

## Overview

> ⚠️ Take into account that **version 2.0.0 contains breaking changes**.
> ⚠️ 
> ⚠️ The configuration in this article is only valid for versions v2.x. If you are using a v1.x, please upgrade to v2.x and carefully analyze this configuration file to successfully deploy the collector.

[GitHub](https://github.com/) is a version control platform that allows you to track changes to your codebase, flag bugs and issues for follow-up, and manage your product's build process. It simplifies the process of working with other people and makes it easy to collaborate on projects. Team members can work on files and easily merge their changes with the master branch of the project. GitHub API provides data about a hosted code repository, ranging from commits to pull requests and comments.

The Devo Github collector enables customers to retrieve data from **GitHub API** into Devo to query, correlate, analyze, and visualize it, enabling Enterprise IT and Cybersecurity teams to take the most impactful decisions at the petabyte scale.

## API limitations

> ⚠️ The main API limitation is the Rate Limit established by Github.

Like several other APIs, the GitHub API has a rate limit. Applications such as collectors are allowed to make a limited number of API requests per hour. Hitting the rate limit causes the collector to stop the time determined by the GitHub API until the next period (for instance, the collector has to stop for 1000 or 2000 seconds). All the API shares this limit, so the collector stops receiving and sending events to Devo for all the services. The result is a delay in the event reception, some events don’t arrive at Devo until several minutes later.

The rate limit depends on the section of the API used. From the collector's point of view, there are two kinds of Github API services, by **Organization** and by **Repositories**.

- For **Organization** services, the rate limit is 1750 requests per hour
- For **Repository** services, the default limit is 5000, but Enterprise Cloud Accounts may have a higher limit, up to 15000 requests per hour.

This number can seem high, and it is enough for a small corporation or individual, but unfortunately, it is extremely limited to extract all the data available from some corporations on Github. The reason is the way that data on Github should be extracted.

- Services by **Organization** (as audit) are simple, one call can return all the event items for the entire organization. Usually, the information is divided into several different pages (maybe 100 or more), and each page takes a request for the count.  Only if we want to extract data from several months or years, that can be tens of thousands of events, we are at risk of reaching the rate limit.
- For by **Repository** services (commits, events, forks), it is not possible to gather all in one call. First, we ask for a list of repositories (there are 148), and for each repository, we ask for the new items. We need to make this for each service, even if there is no data for that repository. If there is data, we need to download all the pages, so more requests are needed.

Let's put an example of this. Imagine that we have 200 code repositories in GitHub (some corporations have thousands) and we activate 10 services in the collector. Then we’ll need 2000 requests (as a minimum) for each data pull, even if there is no new data. This is a minimum value, additional requests can be needed for loading all data pages for each repository.

If we do a data pull every minute, the limit can be depleted in a few minutes, even when the limit is 15000.

When the limit is reached, Github API returns a header telling you how long the collector should stop to recover. The collector writes in the log a warning “API Rate Limit Exceeded, waiting for X seconds“ and stops for the time given.

So it is important to configure the collector carefully. **There are two ways to avoid the rate limit:**

- Optimize the number of services in execution, and select carefully those services that require to be monitored.
- Decrease the pull frequency, instead of trying to get data from all the services each minute, (the default setting), use the parameter `request_period_in_seconds` with values of 600, 1200, or more seconds. Prioritize some services related to alerts as Audit and call other services every 10 or more minutes. Some services could be called 1 or 2 times a day.

There is a trick that can be done to somehow increase the number of requests. Github allows the creation of one Personal Access Token per account. So, for this purpose, it would be possible to create multiple accounts and then use one account per service to be monitored. Those accounts must belong to the Organization from which the data will be pulled. Github only allows having one free account, so those accounts would be **paid** accounts.

## Migration from 2.x.x to 3.0.0

If you are using the `audit` service, the migration to version v3.0.0 requires a small human intervention. There is a change in the service name and parameters. In version 3.0.0:

- The `audit` service disappeared.
- A new service `organization_audit` has been added, including some improvements.

The old `audit` service uses a pagination method that doesn't allow to choose a initial date for event gathering. The new `organization_audit` service uses a different method, so we can now use `since` as other services. Unfortunately, the new method cannot use the pagination token from the old method, and a initial date has to be configured by the user.

The procedure for migration is:

1. Stop the old 2.x.x service and take note of the time.
2. Edit the config file. Delete the `audit` service and add a new `organization_audit` service with a `since` value that is the time when the old collector was stopped.

For instance, if we have this configuration for 2.x.x:

```
        "audit": {
          "request_period_in_seconds": "60",
          "persistence_reset_date": "2002-03-11"
        },
```

change it by:

```
        "organization_audit": {
          "request_period_in_seconds": "60",
          "since": "2024-05-22T23:00:00Z"
        }
```

where `2024-05-22T23:00:00Z` is the time when we stopped the old collector.

3. Change the image to new version 3.0.0 and restart the collector.

The new `organization_audit` service uses the same endpoint, events use the same format and they are stored in `vcs.github.organization.audit` as before.

## Configuration requirements

To run this collector, there are some configurations detailed below that you need to take into account.

| **Configuration** | **Details** |
| --- | --- |
| Token | You’ll need to create an access token to authenticate the collector on the **GitHub** server. |

> ℹ️ **Configuration**
> ℹ️ 
> ℹ️ Refer to the [Vendor setup ](#Vendor-setup)section to know more about these configurations.

## Devo collector features

| **Feature** | **Details** |
| --- | --- |
| Allow parallel downloading (`multipod`) | `not allowed` |
| Running environments | - `collector server`
- `on-premise` |
| Populated Devo events | `standard` |

## Data sources

|  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| **Data Source** | **Description** | **GitHub API endpoint** | **Collector service name** | **Type** | **Devo table** | **Available from release** |
| Collaborators | Information about collaborators. | `/repos/{owner}/{repo}/collaborators`<br>[Repositories - GitHub Docs](https://docs.github.com/en/rest/reference/repos#list-repository-collaborators)<br>- `metadata:read` | `collaborators` | `repository` | `vcs.github.repository.collaborators` | `v1.0.0` |
| Commits | Commits made in the repository | `/repos/{owner}/{repo}/commits`<br>[Repositories - GitHub Docs](https://docs.github.com/en/rest/repos#list-commits)<br>- `contents:read` | `commits` | `repository` | `vcs.github.repository.commits` | `v1.0.0` |
| Forks | Forks created in the repository | `/repos/{owner}/{repo}/forks`<br>[Repositories - GitHub Docs](https://docs.github.com/en/rest/repos#list-forks)<br>- `metadata:read` | `forks` | `repository` | `vcs.github.repository.forks` | `v1.0.0` |
| Events | Information about the different events such as resource creations or deletions. | `/repos/{owner}/{repo}/events`<br>[Eventos - GitHub Docs](https://docs.github.com/es/rest/activity/events#list-repository-events)<br>- `metadata:read` | `events` | `repository` | `vcs.github.repository.events` | `v1.0.0` |
| Issue comments | Comments made in every issue. | `/repos/{owner}/{repo}/comments`<br>[Issue comments - GitHub Docs](https://docs.github.com/en/rest/issues/comments#list-issue-comments-for-a-repository)<br>- `issues:read or `
- `pull_requests:read` | `issue_comments` | `repository` | `vcs.github.repository.issue_comments` | `v1.0.0` |
| Subscribers | Information about the different users subscribed to one repository. | `/repos/{owner}/{repo}/subscribers`<br>[Watching - GitHub Docs](https://docs.github.com/en/rest/activity/watching#list-watchers)<br>- `metadata:read` | `subscribers` | `repository` | `vcs.github.repository.subscribers` | `v1.0.0` |
| Pull requests | Pull requests made in the repository. | `/repos/{owner}/{repo}/pulls`<br>`/repos/{owner}/{repo}/pulls/{pull_number}/commits`<br>[Pulls - GitHub Docs](https://docs.github.com/en/rest/pulls#list-pull-requests)<br>- `pull_requests:read` | `pull_requests` | `repository` | `vcs.github.repository.pull_requests`<br>`vcs.github.repository.pull_request_commits` | `v1.0.0` |
| Subscriptions | Repositories you are subscribed. | `/repos/{owner}/{repo}/subscription`<br>[Activity - GitHub Docs](https://docs.github.com/en/rest/reference/activity#get-a-repository-subscription)<br>- `metadata:read` | `subscriptions` | `repository` | `vcs.github.repository.subscriptions` | `v1.0.0` |
| Releases | Information about releases made in the repository. | `/repos/{owner}/{repo}/releases`<br>[Repositories - GitHub Docs](https://docs.github.com/en/rest/repos#list-releases)<br>- `contents:read` | `releases` | `repository` | `vcs.github.repository.releases` | `v1.0.0` |
| Stargazers | Information about users who start repositories making them favorites | `/repos/{owner}/{repo}/stargazers`<br>[Starring - GitHub Docs](https://docs.github.com/en/rest/activity/starring#list-stargazers)<br>- `metadata:read` | `stargazers` | `repository` | `vcs.github.repository.stargazers` | `v1.0.0` |
| SSO Authorizations | Single sign-on authorization. | `/orgs/{org}/credential-authorizations`<br>[Organizations - GitHub Docs](https://docs.github.com/en/rest/orgs#list-saml-sso-authorizations-for-an-organization)<br>- `organization_administration:read` | `sso_authorizations` | `organization` | `vcs.github.organization.sso_authorizations` | `v1.0.0` |
| Webhooks | Organization created webhooks. | `/orgs/{org}/hooks`<br>[Organizations - GitHub Docs](https://docs.github.com/en/rest/orgs#list-organization-webhooks)<br>`admin:org_hook` | `webhooks` | `organization` | `vcs.github.organization.webhooks` | `v1.0.0` |
| Dependabot Alerts | GitHub sends Dependabot alerts when we detect that your repository uses a vulnerable dependency or malware. | `/repos/{owner}/{repo}/dependabot/alerts`<br>[Dependabot alerts - GitHub Docs](https://docs.github.com/en/rest/dependabot/alerts#list-dependabot-alerts-for-a-repository)<br>- `vulnerability_alerts:read` | `dependabot_alerts` | `repository` | `vcs.github.organization.dependabot_alerts` | `v2.0.0` |
| Dependabot Secrets | Lists all secrets available in an organization without revealing their encrypted values. | `/orgs/{org}/dependabot/secrets`<br>[Dependabot secrets - GitHub Docs](https://docs.github.com/en/rest/dependabot/secrets#list-organization-secrets)<br>- `admin:org` | `dependabot` | `organization` | `vcs.github.organization.dependabot` | `v2.0.0` |
| Actions | GitHub Actions for a repository. | `/repos/{owner}/{repo}/actions/runs`<br>[Workflow runs - GitHub Docs](https://docs.github.com/en/rest/actions/workflow-runs#list-workflow-runs-for-a-repository)<br>- `actions:read` | `actions` | `repository` | `vcs.github.repository.actions` | `v2.0.0` |
| CodeScan | Code scanning is a feature that you use to analyze the code in a GitHub repository to find security vulnerabilities and coding errors. | `/repos/{owner}/{repo}/code-scanning/alerts`<br>[Code Scanning - GitHub Docs](https://docs.github.com/en/rest/code-scanning#list-code-scanning-alerts-for-a-repository)<br>- `security_events:read` | `codescan` | `repository` | `vcs.github.repository.codescan` | `v2.0.0` |
| Enterprise Audit | Enterprise Auditory Events | `/enterprises/{enterprise}/audit-log`<br>[REST API endpoints for organizations - GitHub Docs](https://docs.github.com/en/rest/orgs#get-the-audit-log-for-an-organization)<br>- `admin:enterprise`
- `read:audit_log`
- `read:enterprise` | `enterprise_audit` | `enterprise` | `vcs.github.enterprise.audit` | `v2.0.0` |
| Organization Audit | Organization Auditory events | `/orgs/{org}/audit-log`<br>[REST API endpoints for organizations - GitHub Docs](https://docs.github.com/en/rest/orgs#get-the-audit-log-for-an-organization)<br>`read:audit_log` | `organization_audit` | `organization` | `vcs.github.organization.audit` | `v3.0.0` |

> ℹ️ **GitHub Documentation**
> ℹ️ 
> ℹ️ Refer to the [GitHub documentation ](https://docs.github.com/en/rest/reference/repos#list-repository-collaborators)to know more about its repositories.

For more information on how the events are parsed, [visit our page](https://docs.devo.com/space/latest/94665779/vcs.github). 

## Vendor setup

> Macro (anchor)



### Personal access token authentication

To retrieve the data, we need to create an access token to authenticate the collector on the GitHub server. 

> Macro (rw-ui-steps-macro)
> 
> > Macro (rw-step)
> 
> Login into your GitHub account as owner. 
> 
> ![image](media://e9a91f8b-ab41-4e86-a0f8-1bfc375e01be)
> 
> > Macro (rw-step)
> 
> Go to **Settings → Developer settings → Personal access tokens**. 
> 
> > Macro (rw-step)
> 
> Click on **Generate new token**.
> 
> ![image](media://4cffb261-9812-44f1-8b71-4b59f6d81722)
> 
> > Macro (rw-step)
> 
> Make sure the following fields are marked:
> 
> - `repo`
> - `repo:status`
> - `repo_deployment`
> - `public_repo`
> - `repo:invite`
> - `security_events`
> - `read:packages`
> - `read:org`
> - `read:public_key`
> - `read:repo_hook`
> - `admin:org_hook`
> - `notifications`
> - `user`
> - `read:user`
> - `user:email`
> - `user:follow`
> - `read:discussion`
> - `read:enterprise`
> - `read:gpg_key`
> 
> > Macro (rw-step)
> 
> Click on **Generate token**.

> ⚠️ If you want to use the Enterprise Audit service, make sure that `read:audit_log` and`read:enterprise` are present.

### Github App Installation authentication

> Macro (rw-ui-steps-macro)
> 
> > Macro (rw-step)
> 
> Login to your Github account. This account needs to be the main or principal account that controls all Github (owner).
> 
> ![image](media://de48a304-70cb-4f6d-88c4-25e66e14477a)
> 
> > Macro (rw-step)
> 
> Go to **Settings → Developer settings → Github Apps**.
> 
> ![image](media://e7ec984d-c7ac-40a2-be2b-a5c454a7df7b)
> 
> > Macro (rw-step)
> 
> Click **Create Github App** and grant *Read-Only* permissions for all the services you want to collect data from.
> 
> > Macro (rw-step)
> 
> Once the app is created, in the **Github Apps** screen, click **Edit → Install App** on the left side menu, and install it in the organization or account you want to get data from.
> 
> ![image](media://f597c3a3-bcf0-476d-a507-dfbb35424354)
> 
> > Macro (rw-step)
> 
> Go to **General** and annotate the App ID.
> 
> ![image](media://393d2170-a40e-4052-bf77-6ba76f85866f)
> 
> > Macro (rw-step)
> 
> On the same screen, scroll down and click **Generate a private key**. The generated file can now be encoded as base64 to be used directly as a string in collector parameters.
> 
> ![image](media://3798aa84-624c-4ae2-86bc-4a20f9523b50)

> ℹ️ If you are using SAML authentication in your account, you’ll need to authorize your token after generating it. Check how to do it in this [article](https://docs.github.com/en/enterprise-cloud@latest/authentication/authenticating-with-saml-single-sign-on/authorizing-a-personal-access-token-for-use-with-saml-single-sign-on).

### Authorization with SAML

#### What is SAML authorization?

**SAML authorization** is a markup language for security confirmations that provides a standardized way to tell external applications and services that a user is who he or she claims to be. SAML uses **single sign-on** (SSO) technology and allows you to authenticate a user once and then communicate that authentication to multiple applications. 

#### Authorizing a personal access token

To use SAML, you need to authorize the token for personal use. There are two ways:

- Authorize existing token
- Create a new token and authorize it.

> ℹ️ **GitHub documentation**
> ℹ️ 
> ℹ️ Refer to the [GitHub documentation](https://docs.github.com/en/enterprise-cloud@latest/authentication/authenticating-with-saml-single-sign-on/authorizing-a-personal-access-token-for-use-with-saml-single-sign-on) to know how to do it.

## Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.

> ℹ️ This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.

| **Setting** | **Details** |
| --- | --- |
| `token` | Set up here requires your access token created in the GitHub console. |
| `username` | Set up here requires your username. |
| `private_key_path` | Set here the path to the .pem file that stores your private key |
| `private_key_base64` | Set here the private key file encoded in base64 |
| `app_id` | Set here the id of your installed app |
| `organization` | Use this parameter to define the name of the organization that owns the repository |

> ℹ️ See the **Accepted authentication methods** section to verify what settings are required based on the desired authentication method.

## Accepted authentication methods

| **Authentication method** | **URL** | **Token** | **Username** | **Private key** | **App ID** | **Organization** |
| --- | --- | --- | --- | --- | --- | --- |
| Personal Access Token | [OPTIONAL]  
(default is `https://api.github.com/`) | [REQUIRED] | [REQUIRED] | [NOT REQUIRED] | [NOT REQUIRED] | [REQUIRED] |
| GitHub App installation | [OPTIONAL]  
(default is `https://api.github.com/`) | [NOT REQUIRED] | [NOT REQUIRED] | [REQUIRED] | [REQUIRED] | [REQUIRED] |

## Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (<u>Cloud collector</u>), or deploy and host the collector in your own machine using a Docker image (<u>On-premise collector</u>).

> Macro (rw-ui-tabs-macro)
> 
> > Macro (rw-tab)
> 
> We use a piece of software called Collector Server to host and manage all our available collectors.
> 
> To enable the collector for a customer:
> 
> 1. In the **Collector Server** **GUI**, access the **domain** in which you want this instance to be created
> 2. Click **Add Collector** and find the one you wish to add.
> 3. In the **Version** field, select the latest value.
> 4. In the **Collector Name** field, set the value you prefer (this name must be unique inside the same Collector Server domain).
> 5. In the sending method select **Direct Send. Direct Send** configuration is optional for collectors that create `Table` events, but mandatory for those that create `Lookups`.
> 6. In the **Parameters** section, establish the **Collector Parameters** as follows below:
> 
> ### Editing the JSON configuration
> 
> ```
> {
>   "global_overrides": {
>     "debug": <debug>
>   },
>   "inputs": {
>     "github": {
>       "id": "<short_unique_id>",
>       "enabled": true,
>       "override_api_url": "override_api_url",
>       "credentials": {
>         "token": "<token>",
>         "username": "<username>",
>         "private_key_path":"<private_key_path>",
>         "private_key_base64": "<private_key_base64>",
>         "app_id": "<app_id>",
>         "org": "<organization>",
>         "enterprise": "<enterprise>",
>         "pager": 100
>       },
>       "rate_limit": {
>         "period_in_seconds": <period_in_seconds>,
>         "requests_limit_in_units": <requests_limit_in_units>
>       },
>       "services": {
>         "actions": {
>           "request_period_in_seconds": <request_period_in_seconds>,
>           "override_devo_tag": "<override_devo_tag>",
>           "since": "<since>"
>         },
>         "codescan": {
>           "request_period_in_seconds": <request_period_in_seconds>,
>           "override_devo_tag": "<override_devo_tag>",
>           "persistence_reset_date": "<persistence_reset_date>"
>         },
>         "collaborators": {
>           "request_period_in_seconds": <request_period_in_seconds>,
>           "override_devo_tag": "<override_devo_tag>",
>           "persistence_reset_date": "<persistence_reset_date>"
>         },
>         "commits": {
>           "request_period_in_seconds": <request_period_in_seconds>,
>           "override_devo_tag": "<override_devo_tag>",
>           "since": "<since>"
>         },
>         "dependabot": {
>           "request_period_in_seconds": <request_period_in_seconds>,
>           "override_devo_tag": "<override_devo_tag>",
>           "persistence_reset_date": "<persistence_reset_date>"
>         },
>         "dependabot_alerts": {
>           "request_period_in_seconds": <request_period_in_seconds>,
>           "override_devo_tag": "<override_devo_tag>",
>           "persistence_reset_date": "<persistence_reset_date>"
>         },
>         "events": {
>           "request_period_in_seconds": <request_period_in_seconds>,
>           "override_devo_tag": "<override_devo_tag>",
>           "persistence_reset_date": "<persistence_reset_date>"
>         },
>         "forks": {
>           "request_period_in_seconds": <request_period_in_seconds>,
>           "override_devo_tag": "<override_devo_tag>",
>           "persistence_reset_date": "<persistence_reset_date>"
>         },
>         "issue_comments": {
>           "request_period_in_seconds": <request_period_in_seconds>,
>           "override_devo_tag": "<override_devo_tag>",
>           "since": "<since>"
>         },
>         "pull_requests": {
>           "request_period_in_seconds": <request_period_in_seconds>,
>           "override_devo_tag": "<override_devo_tag>",
>           "persistence_reset_date": "<persistence_reset_date>"
>         },
>         "releases": {
>           "request_period_in_seconds": <request_period_in_seconds>,
>           "override_devo_tag": "<override_devo_tag>",
>           "persistence_reset_date": "<persistence_reset_date>"
>         },
>         "sso_authorizations": {
>           "request_period_in_seconds": <request_period_in_seconds>,
>           "override_devo_tag": "<override_devo_tag>",
>           "persistence_reset_date": "<persistence_reset_date>"
>         },
>         "stargazers": {
>           "request_period_in_seconds": <request_period_in_seconds>,
>           "override_devo_tag": "<override_devo_tag>",
>           "persistence_reset_date": "<persistence_reset_date>"
>         },
>         "subscribers": {
>           "request_period_in_seconds": <request_period_in_seconds>,
>           "override_devo_tag": "<override_devo_tag>",
>           "persistence_reset_date": "<persistence_reset_date>"
>         },
>         "subscriptions": {
>           "request_period_in_seconds": <request_period_in_seconds>,
>           "override_devo_tag": "<override_devo_tag>",
>           "persistence_reset_date": "<persistence_reset_date>"
>         },
>         "webhooks": {
>           "request_period_in_seconds": <request_period_in_seconds>,
>           "override_devo_tag": "<override_devo_tag>",
>           "persistence_reset_date": "<persistence_reset_date>"
>         },
>         "enterprise_audit": {
>           "request_period_in_seconds": "<request_period_in_seconds>",
>           "override_devo_tag": "<override_devo_tag>",
>           "since": "<since>"
>         },
>         "enterprise_audit": {
>           "request_period_in_seconds": "<request_period_in_seconds>",
>           "override_devo_tag": "<override_devo_tag>",
>           "since": "<since>"
>         }
>       }
>     }
>   }
> }
> ```
> 
> > ℹ️ All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object.
> 
> Please replace the placeholders with real world values following the description table below:
> 
> |  |  |  |  |  |
> | --- | --- | --- | --- | --- |
> | **Parameter** | **Data type** | **Type** | **Value range / Format** | **Details** |
> | `debug` | `bool` | `Mandatory` | `false` / `true` | If the value is `true`, the debug logging traces will be enabled when running the collector. If the value is `false`, only the `info`, `warning,` and `error` logging levels will be printed. |
> | `short_unique_id` | `int` | `Mandatory` | Minimum length: 1  
> Maximum length: 5 | Use this param to give a unique id to this input service.<br>> ⚠️ This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision. |
> | `override_api_url` | `str` | `Optional` | String that matches the next RegEx:  
> `https:\/\/([a-z0-9]+[.]{1})+[a-z]+[\/]{1}` | This URL overrides the default API base URL used for every request made using this collector. The default value is `https://api.github.com/`. Notice that the final slash `/` is required, or else the collector will throw an error.<br>> 📝 This parameter should be removed if it is not used. |
> | `token` | `str` | `Optional`  
> ( `Mandatory` *if private_key_path and private_key_base64 are not present*) | Minimum length: 1 | Set up here your access token created in the GitHub console.<br>> ⚠️ Save this token as if it were a password |
> | `username` | `str` | `Optional`  
> ( `Mandatory` *if token is present*) | Minimum length: 1 | Set up here your username |
> | `organization` | `str` | `Mandatory` | Minimum length: 1 | Use this parameter to define the name of the organization that owns the repository |
> | `enterprise` | `str` | `Optional`<br>( `Mandatory` *if you want to use the *`Enterprise Audit `*service*) | Minimum length: 1 | Use this parameter only if you are using an Enterprise account for GitHub. and you want to use the `Enterprise Audit` service. |
> | `private_key_path` | `str` | `Optional`  
> ( `Mandatory` *if token and private_key_base64 are not present*) | Minimum length: 1 | Set here the path to the .pem file that stores your private key |
> | `private_key_base64` | `str` | `Optional`  
> ( `Mandatory` *if token and private_key_path are not present*) | Minimum length: 1 | Set here the private key file encoded in base64 |
> | `app_id` | `str` | `Optional`  
> ( `Mandatory` *if private_key_path or private_key_base64 are present*) | Minimum length: 1 | Set here the id of your installed app |
> | `pager` | `int` | `Mandatory` | Maximum value: 100  
> *(using the max value is recommended to avoid making additional requests to the API)* | The number of elements returned by the API for each page. Each time a request is made to the API, `pager` elements will be returned. |
> | `period_in_seconds` | `int` | `Optional`  
> *(*`Mandatory`* if rate_limit is present)* | Minimum value: 1 | This is part of the configuration of a custom Rate Limiter. `rate_limit`, which is an object, contains two keys: `period_in_seconds` and `requests_limit_in_units`. In the example below, it will be set a Rate Limiter that allows up to 15 request/min.<br>```
> rate_limit:
>   period_in_seconds: 60
>   requests_limit_in_units: 15
> ```<br>`period_in_seconds` is the reference time frame in which the Rate Limiter will count the number of requests made to the API. |
> | `requests_limit_in_units` | `int` | `Optional`  
> *(*`Mandatory`* if rate_limit is present)* | Minimum value: 1 | This is part of the configuration of a custom Rate Limiter. `rate_limit`, which is an object, contains two keys: `period_in_seconds` and `requests_limit_in_units`. In the example below, it will be set a Rate Limiter that allows up to 15 request/min.<br>```
> "rate_limit": {
>   "period_in_seconds": 60,
>   "requests_limit_in_units": 15
> }
> ```<br>`requests_limit_in_units` is the number of requests made to the API during the time frame established by `period_in_seconds`. |
> | `request_period_in_seconds` | `int` | `Optional` | Minimum value: 1 | Period in seconds used between each data pulling, this value will overwrite the default value (60 seconds)<br>> 📝 This parameter should be removed if it is not used. |
> | `override_devo_tag` | `str` | `Optional` | Minimum length: 1 | This field overrides the tag to which the message will be sent.<br>> 📝 This parameter should be removed if it is not used. |
> | `since` | `str` | `Mandatory`<br>Only for services that use it. | ISO-8601 format: `YYYY-MM-DDTHH:MM:SSZ`<br>RegEx to be matched:  
> `[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z` | This field defines the date from which the data will be pulled. If this field changes its value from previous execution, the persistence will be reset.<br>> 📝 This parameter is required only for services Actions, Commits and Issue Comments. |
> | `persistence_reset_date` | `str` | `Mandatory` for all services that use it | Minimum length: 1 | This is a free text field, which is used to identify when the persistence should be reset. If this value changes from the previous execution, the persistence will be reset.<br>Despite this is a free text field, it is recommended to use a reference to the day the persistence is being reset.<br>For example, if today were October 26th and we are going to reset the persistence, we could set the field to `2022-10-26`.<br>> 📝 This parameter is required for all services except Actions, Commits, and Issue Comments. |
> 
> ![github_10.png](media://5556f47a-fdc2-48c1-a3e4-c61175aaa3d1)
> 
> > Macro (rw-tab)
> 
> This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.
> 
> ### Structure
> 
> The following directory structure should be created for being used when running the collector:
> 
> ```
> <any_directory>
> └── devo-collectors/
>     └── github/
>         ├── certs/
>         │   ├── chain.crt
>         │   ├── <your_domain>.key
>         │   └── <your_domain>.crt
>         ├── state/
>         └── config/ 
>             └── config-github.yaml           
> ```
> 
> > ⚠️ Replace `<product_name>` with the proper value.
> 
> ### Devo credentials
> 
> In Devo, go to **Administration → Credentials → X.509 Certificates**, download the **Certificate**, **Private key** and **Chain CA** and save them in `<product_name>/certs/`. Learn more about security credentials in Devo [here](#).
> 
> ![image](media://26741a29-26bb-423e-8103-afe01122e380)
> 
> > ⚠️ Replace `<product_name>` with the proper value.
> 
> ### Editing the config.yaml file
> 
> ```
> globals:
>   debug: <debug>
>   id: "<collector_id>"
>   name: "<collector_name>"
>   persistence:
>     type: filesystem
>     config:
>       directory_name: state
>   multiprocessing: false
>   queue_max_size_in_mb: 1024
>   queue_max_size_in_messages: 1000
>   queue_max_elapsed_time_in_sec: 60
>   queue_wrap_max_size_in_messages: 100
> 
> outputs:
>   devo_1:
>     type: devo_platform
>     config:
>       address: "<devo_address>"
>       port: 443
>       type: SSL
>       chain: "<chain_filename>"
>       cert: "<cert_filename>"
>       key: "<key_filename>"
> 
> inputs:
>   github:
>     id: "<short_unique_id>"
>     enabled: true
>     override_api_url: "<override_api_url>"
>     credentials:
>       token: "<token>"
>       username: "<username>"
>       org: "<organization>"
>       enterprise: "<enterprise>"
>       private_key_path: <private_key_path>
>       private_key_base64: <private_key_base64>
>       app_id: <app_id>
>       pager: 100
>     rate_limit:
>       period_in_seconds: <period_in_seconds>
>       requests_limit_in_units: <requests_limit_in_units>
>     services:
>       actions:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: "<override_devo_tag>"
>         since: <since>
>       codescan:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: "<override_devo_tag>"
>         persistence_reset_date: "<persistence_reset_date>"
>       collaborators:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: "<override_devo_tag>"
>         persistence_reset_date: "<persistence_reset_date>"
>       commits:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: "<override_devo_tag>"
>         since: <since>
>       dependabot:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: "<override_devo_tag>"
>         persistence_reset_date: "<persistence_reset_date>"
>       dependabot_alerts:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: "<override_devo_tag>"
>         persistence_reset_date: "<persistence_reset_date>"
>       events:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: "<override_devo_tag>"
>         persistence_reset_date: "<persistence_reset_date>"
>       forks:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: "<override_devo_tag>"
>         persistence_reset_date: "<persistence_reset_date>"
>       issue_comments:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: "<override_devo_tag>"
>         since: <since>
>       pull_requests:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: "<override_devo_tag>"
>         persistence_reset_date: "<persistence_reset_date>"
>       releases:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: "<override_devo_tag>"
>         persistence_reset_date: "<persistence_reset_date>"
>       sso_authorizations:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: "<override_devo_tag>"
>         persistence_reset_date: "<persistence_reset_date>"
>       stargazers:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: "<override_devo_tag>"
>         persistence_reset_date: "<persistence_reset_date>"
>       subscribers:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: "<override_devo_tag>"
>         persistence_reset_date: "<persistence_reset_date>"
>       subscriptions:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: "<override_devo_tag>"
>         persistence_reset_date: "<persistence_reset_date>"
>       webhooks:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: "<override_devo_tag>"
>         persistence_reset_date: "<persistence_reset_date>"
>       enterprise_audit:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: <override_devo_tag>
>         since: <since>
>       organization_audit:
>         request_period_in_seconds: <request_period_in_seconds>
>         override_devo_tag: <override_devo_tag>
>         since: <since>
> ```
> 
> > ℹ️ All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object.
> 
> Replace the placeholders with your required values following the description table below:
> 
> |  |  |  |  |  |
> | --- | --- | --- | --- | --- |
> | **Parameter** | **Data type** | **Type** | **Value range** | **Details** |
> | `debug` | `bool` | `Mandatory` | `false` / `true` | If the value is `true`, the debug logging traces will be enabled when running the collector. If the value is `false`, only the `info`, `warning,` and `error` logging levels will be printed. |
> | `collector_id` | `int` | `Mandatory` | Minimum length: 1  
> Maximum length: 5 | Use this param to give an unique id to this collector. |
> | `collector_name` | `str` | `Mandatory` | Minimum length: 1  
> Maximum length: 10 | Use this param to give a valid name to this collector. |
> | `devo_address` | `str` | `Mandatory` | `collector-us.devo.io`  
> `collector-eu.devo.io` | Use this param to identify the Devo Cloud where the events will be sent. |
> | `chain_filename` | `str` | `Mandatory` | Minimum length: 4  
> Maximum length: 20 | Use this param to identify the chain. cert  file downloaded from your Devo domain. Usually this file's name is: `chain.crt` |
> | `cert_filename` | `str` | `Mandatory` | Minimum length: 4  
> Maximum length: 20 | Use this param to identify the `file.cert` downloaded from your Devo domain. |
> | `key_filename` | `str` | `Mandatory` | Minimum length: 4  
> Maximum length: 20 | Use this param to identify the `file.key` downloaded from your Devo domain. |
> | `short_unique_id` | `int` | `Mandatory` | Minimum length: 1  
> Maximum length: 5 | Use this param to give a unique id to this input service.<br>> ⚠️ This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision. |
> | `override_api_url` | `str` | `Optional` | String that matches the next RegEx:  
> `https:\/\/([a-z0-9]+[.]{1})+[a-z]+[\/]{1}` | This URL overrides the default API base URL used for every request made using this collector. The default value is `https://api.github.com/`. Notice that the final slash `/` is required, or else the collector will throw an error.<br>> ℹ️ This parameter should be removed if it is not used. |
> | `token` | `str` | `Mandatory` | Minimum length: 1 | Set up here your access token created in the GitHub console.<br>> ℹ️ Save this token as if it were a password |
> | `username` | `str` | `Mandatory` | Minimum length: 1 | Set up here your username. |
> | `organization` | `str` | `Mandatory` | Minimum length: 1 | Use this parameter to define the name of the organization that owns the repository |
> | `pager` | `int` | `Mandatory` | Maximum value: 100  
> *(using the max value is recommended to avoid making additional requests to the API)* | The number of elements returned by the API for each page. Each time a request is made to the API, `pager` elements will be returned. |
> | `period_in_seconds` | `int` | `Optional`  
> *(*`Mandatory`* if rate_limit is present)* | Minimum value: 1 | This is part of the configuration of a custom Rate Limiter. `rate_limit`, which is an object, contains two keys: `period_in_seconds` and `requests_limit_in_units`. In the example below, it will be set a Rate Limiter that allows up to 15 request/min.<br>```
> rate_limit:
>         period_in_seconds: 60
>         requests_limit_in_units: 15
> ```<br>`period_in_seconds` is the reference time frame in which the Rate Limiter will count the number of requests made to the API. |
> | `requests_limit_in_units` | `int` | `Optional`  
> *(*`Mandatory`* if rate_limit is present)* | Minimum value: 1 | This is part of the configuration of a custom Rate Limiter. `rate_limit`, which is an object, contains two keys: `period_in_seconds` and `requests_limit_in_units`. In the example below, it will be set a Rate Limiter that allows up to 15 request/min.<br>```
> rate_limit:
>        period_in_seconds: 60
>        requests_limit_in_units: 15
> ```<br>`requests_limit_in_units` is the number of requests made to the API during the time frame established by `period_in_seconds`. |
> | `request_period_in_seconds` | `int` | `Optional` | Minimum value: 1 | Period in seconds used between each data pulling, this value will overwrite the default value (60 seconds)<br>> ℹ️ This parameter should be removed if it is not used. |
> | `override_devo_tag` | `str` | `Optional` | Minimum length: 1 | This field overrides the tag to which the message will be sent.<br>> ℹ️ This parameter should be removed if it is not used. |
> | `since` | `str` | `Mandatory`<br>Only for services that use it. | ISO-8601 format: `YYYY-MM-DDTHH:MM:SSZ`<br>RegEx to be matched:  
> `[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z` | This field defines the date from which the data will be pulled. If this field changes its value from previous execution, the persistence will be reset.<br>> ℹ️ This parameter is required only for services Actions, Commits and Issue Comments. |
> | `persistence_reset_date` | `str` | `Mandatory` for services that use it. | Minimum length: 1 | This is a free text field, which is used to identify when the persistence should be reset. If this value changes from the previous execution, the persistence will be reset.<br>Despite this is a free text field, it is recommended to use a reference to the day the persistence is being reset.<br>For example, if today were October 26th and we are going to reset the persistence, we could set the field to `2022-10-26`.<br>> ℹ️ This parameter is required for all services except Actions, Commits and Issue Comments. |
> 
> ### Download the Docker image
> 
> The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
> 
> | **Collector Docker image** | **SHA-256 hash** |
> | --- | --- |
> | [collector-github_collector-docker-image-3.0.0](https://drive.google.com/file/d/1JpWEScz8Ja36Uo6I82R-EQnmu3sBYZhp/view?usp=drive_link) | `1c5caa9b3b36b498a37879be4e73d2a1e89b3590d4f36fbc42ea44cbedbaa7f6` |
> 
> Use the following command to add the Docker image to the system:
> 
> ```
> gunzip -c <image_file>-<version>.tgz | docker load
> ```
> 
> > ⚠️ Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace `<image_file>` and `<version>` with a proper value.
> 
> The Docker image can be deployed on the following services:
> 
> #### Docker
> 
> Execute the following command on the root directory `<any_directory>/devo-collectors/<product_name>/`
> 
> ```
> docker run 
> --name collector-<product_name> 
> --volume $PWD/certs:/devo-collector/certs 
> --volume $PWD/config:/devo-collector/config 
> --volume $PWD/state:/devo-collector/state 
> --env CONFIG_FILE=config.yaml 
> --rm 
> --interactive 
> --tty 
> <image_name>:<version>
> ```
> 
> > ⚠️ Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.
> 
> #### Docker Compose
> 
> The following Docker Compose file can be used to execute the Docker container. It must be created in the `<any_directory>/devo-collectors/<product_name>/` directory.
> 
> ```
> version: '3'
> services:
>   collector-<product_name>:
>     image: <image_name>:${IMAGE_VERSION:-latest}
>     container_name: collector-<product_name>
>     volumes:
>       - ./certs:/devo-collector/certs
>       - ./config:/devo-collector/config
>       - ./credentials:/devo-collector/credentials
>       - ./state:/devo-collector/state
>     environment:
>       - CONFIG_FILE=${CONFIG_FILE:-config.yaml}
> ```
> 
> To run the container using docker-compose, execute the following command from the `<any_directory>/devo-collectors/<product_name>/` directory:
> 
> ```
> IMAGE_VERSION=<version> docker-compose up -d
> ```
> 
> > ⚠️ Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.

## Collector services detail

This section is intended to explain how to proceed with specific actions for services.

<details>
<summary>Common sections for Services</summary>

### Verify data collections

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

| **Component** | **Description** |
| --- | --- |
| Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
| Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |

#### Setup output

A successful run has the following output messages for the setup module:

```
INFO InputProcess::GithubApiserverBasePullerSetup(example_collector,github#444,actions#predefined,all) -> The token/header/authentication is defined
INFO InputProcess::GithubApiserverBasePullerSetup(example_collector,github#444,actions#predefined,all) -> The token/header/authentication is valid
INFO InputProcess::GithubApiserverBasePullerSetup(example_collector,github#444,actions#predefined,all) -> The user whatever-user belongs to whatever-company
INFO InputProcess::GithubApiserverBasePullerSetup(example_collector,github#444,actions#predefined,all) -> Finalizing the execution of setup()
INFO InputProcess::GithubApiserverBasePullerSetup(example_collector,github#444,actions#predefined,all) -> Setup for module <GithubDataPullerActions> has been successfully executed
```
</details>

<details>
<summary>Action Service</summary>

This service lists all workflows that run for a repository in GitHub. All events of this service are ingested into the table `vcs.github.repository.actions`.

### Verify data collections

#### Puller Output

A successful initial run has the following output messages for the puller module:

> ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> Reading persisted data
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> No changes have been made in saved state. Returning saved state: {'pulling_date_from_config': '1640995200.0', 'last_pulled_date': '1641104263.0', 'ids': [1645452345]}
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> GithubDataPullerActions(github,444,actions,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> The collector will start pulling data since 2022-01-02T06:17:43Z
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> Total number of repositories: 2
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> Tag: vcs.github.api.repository.actions
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 3; Number of events received: 1; Number of duplicated events filtered out: 1; Number of events generated and sent: 0; Average of events per second: 0.000.
...
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 3; Number of events received: 1; Number of duplicated events filtered out: 1; Number of events generated and sent: 0; Average of events per second: 0.000.
```

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

```
INFO InputProcess::GithubDataPullerActions(github,444,actions,predefined,all) -> Statistics for this pull cycle Number of requests made: 30; Number of events received: 317; Number of duplicated events filtered out: 11; Number of events generated and sent: 306; Average of events per second: 23.234.
```

> ℹ️ The `@devo_pulling_id` value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that `Pull` action in Loxcope.

> ℹ️ Note that a `Partial` Statistics Report will be displayed after download a page when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> ℹ️ 
> ℹ️ `(Partial) Statistics for this pull cycle Number of requests made: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40; Average of events per second: 23.234.`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `since` parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.
</details>

<details>
<summary>Codescan Service</summary>

Code scanning is a feature that you can use to analyze the code in a GitHub repository to find security vulnerabilities and coding errors. This service returns the codescan results for each repository in case it is enabled.

### Verify data collection

#### Puller output

A successful initial run has the following output messages for the puller module:

> ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> State saved: {'old_persistence_reset_date': '26-Oct', 'codescan': {}}
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> GithubDataPullerCodescan(github,444,codescan,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Starting data collection every 300 seconds
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Get Codescan function called
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Partial statistics: Pages retrieved 10, items buffered 300
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> New items found for repo repo-1 -> 43
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence saved for repo-1 -> 9443322
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Partial statistics: Pages retrieved 10, items buffered 300
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> New items found for repo repo-2 -> 367
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence saved for repo-2 -> 4567887
....
```

After successful execution of the collector, you should be able to see the following log message:

```
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Statistics for this pull cycle Number of requests made: 120; Number of events received: 932; Number of duplicated events filtered out: 0; Number of events generated and sent: 932; Average of events per second: 23.593.
```

> ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

> ℹ️ Note that a `Partial` Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> ℹ️ 
> ℹ️ `Partial statistics: Pages retrieved 10, items buffered 300`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `since` parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.
</details>

<details>
<summary>Collaborators Service</summary>

This service gets a list of collaborators for each repository in GitHub. 

### Verify data collection

#### Puller output

A successful initial run has the following output messages for the puller module:

> ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> State saved: {'old_persistence_reset_date': '26-Oct', 'codescan': {}}
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> GithubDataPullerCodescan(github,444,codescan,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Starting data collection every 300 seconds
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Get Codescan function called
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Partial statistics: Pages retrieved 10, items buffered 300
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> New items found for repo repo-1 -> 43
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence saved for repo-1 -> 9443322
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Partial statistics: Pages retrieved 10, items buffered 300
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> New items found for repo repo-2 -> 367
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence saved for repo-2 -> 4567887
....
```

After the successful execution of the collector, you should be able to see the following log message:

```
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Statistics for this pull cycle Number of requests made: 120; Number of events received: 932; Number of duplicated events filtered out: 0; Number of events generated and sent: 932; Average of events per second: 23.593.
```

> ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

> ℹ️ Note that a `Partial` Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> ℹ️ 
> ℹ️ `Partial statistics: Pages retrieved 10, items buffered 300`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `since` parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.
</details>

<details>
<summary>Commits Service</summary>

This service gets a list of collaborators for each repository in GitHub. 

### Verify data collection

#### Puller output

A successful initial run has the following output messages for the puller module:

> ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> State saved: {'old_persistence_reset_date': '26-Oct', 'codescan': {}}
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> GithubDataPullerCodescan(github,444,codescan,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Starting data collection every 300 seconds
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Get Codescan function called
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Partial statistics: Pages retrieved 10, items buffered 300
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> New items found for repo repo-1 -> 43
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence saved for repo-1 -> 9443322
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Partial statistics: Pages retrieved 10, items buffered 300
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> New items found for repo repo-2 -> 367
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Persistence saved for repo-2 -> 4567887
....
```

After the successful execution of the collector, you should be able to see the following log message:

```
INFO InputProcess::GithubDataPullerCodescan(github,444,codescan,predefined,all) -> Statistics for this pull cycle Number of requests made: 120; Number of events received: 932; Number of duplicated events filtered out: 0; Number of events generated and sent: 932; Average of events per second: 23.593.
```

> ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

> ℹ️ Note that a `Partial` Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> ℹ️ 
> ℹ️ `Partial statistics: Pages retrieved 10, items buffered 300`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `since` parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.
</details>

<details>
<summary>Dependabot</summary>

This service lists all secrets available in an organization without revealing their encrypted values. All events of this service are ingested into table `vcs.github.repository.dependabot`.

### Verify data collection

#### Puller output

A successful initial run has the following output messages for the puller module:

> ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> State saved: {'old_persistence_reset_date': 'test1', 'dependabot': {}}
INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> GithubDataPullerDependabot(github,444,dependabot,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> No more pages have been detected ahead for org my-organization
INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> New items found for org my-organization -> 4
INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> Persistence saved for org my-organization -> 46463737382
....
```

After the successful execution of the collector, you should be able to see the following log message:

```
INFO InputProcess::GithubDataPullerDependabot(github,444,dependabot,predefined,all) -> Statistics for this pull cycle Number of requests made: 1; Number of events received: 4; Number of duplicated events filtered out: 0; Number of events generated and sent: 4; Average of events per second: 5.743.
```

> ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

> ℹ️ Note that a `Partial` Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> ℹ️ 
> ℹ️ `Partial statistics: Pages retrieved 10, items buffered 300`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `persistence_reset_date` parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.
</details>

<details>
<summary>Dependabot Alerts</summary>

This service returns the Dependabot Alerts for each repository. GitHub generates an alert when a repository uses a vulnerable dependency or malware. All events of this service are ingested into table `vcs.github.repository.dependabot_alerts`.

### Verify data collection

#### Puller output

A successful initial run has the following output messages for the puller module:

> ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> State saved: {'old_persistence_reset_date': '26-Oct', 'dependabot_alerts': {}}
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Starting data collection every 300 seconds
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Get Dependabot Alerts function called
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> New items found for repo repo-1 -> 12
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Persistence saved for repo-1 -> 94445
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> New items found for repo repo-2 -> 3
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Persistence saved for repo-2 -> 45556
....
```

After the successful execution of the collector, you should be able to see the following log message:

```
INFO InputProcess::GithubDataPullerDependabotAlerts(github,444,dependabot_alerts,predefined,all) -> Statistics for this pull cycle Number of requests made: 30; Number of events received: 122; Number of duplicated events filtered out: 0; Number of events generated and sent: 122; Average of events per second: 13.63.
```

> ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

> ℹ️ Note that a `Partial` Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> ℹ️ 
> ℹ️ `Partial statistics: Pages retrieved 10, items buffered 300`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `persistence_reset_date` parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.
</details>

<details>
<summary>Events</summary>

This service returns the events for each repository. All events of this service are ingested into table `vcs.github.repository.events`.

### Verify data collection

#### Puller output

A successful initial run has the following output messages for the puller module:

> ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> State saved: {'old_persistence_reset_date': 'prueba-2', 'events': {}}
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> GithubDataPullerEvents(github,444,events,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Starting data collection every 300 seconds
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Get Events function called
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Partial statistics: Pages retrieved 10, items buffered 300
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Partial statistics: Pages retrieved 20, items buffered 600
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> New items found for repo repo-1 -> 740
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Persistence saved for repo-1 -> 456789
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Partial statistics: Pages retrieved 10, items buffered 300
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> New items found for repo repo-2 -> 356
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Persistence saved for repo-2 -> 5678567
....
```

After the successful execution of the collector, you should be able to see the following log message:

```
INFO InputProcess::GithubDataPullerEvents(github,444,events,predefined,all) -> Statistics for this pull cycle Number of requests made: 89; Number of events received: 1562; Number of duplicated events filtered out: 0; Number of events generated and sent: 1562; Average of events per second: 79.13.
```

> ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

> ℹ️ Note that a `Partial` Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> ℹ️ 
> ℹ️ `Partial statistics: Pages retrieved 10, items buffered 300`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `persistence_reset_date` parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.
</details>

<details>
<summary>Forks</summary>

This service returns the forks for each repository. All events of this service are ingested into table `vcs.github.repository.forks`.

### Verify data collection

#### Puller output

A successful initial run has the following output messages for the puller module:

> ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> State saved: {'old_persistence_reset_date': 'prueba-3', 'forks': {}}
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> GithubDataPullerForks(github,444,forks,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Starting data collection every 300 seconds
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Get Forks function called
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> New items found for repo repo-1 -> 3
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Persistence saved for repo-1 -> 9623469344
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> New items found for repo repo-2 -> 2
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Persistence saved for repo-2 -> 5678234564
....
```

After the successful execution of the collector, you should be able to see the following log message:

```
INFO InputProcess::GithubDataPullerForks(github,444,forks,predefined,all) -> Statistics for this pull cycle Number of requests made: 30; Number of events received: 56; Number of duplicated events filtered out: 0; Number of events generated and sent: 56; Average of events per second: 12.128.
```

> ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

> ℹ️ Note that a `Partial` Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> ℹ️ 
> ℹ️ `Partial statistics: Pages retrieved 10, items buffered 300`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `persistence_reset_date` parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.
</details>

<details>
<summary>Issue Comments Service</summary>

This service gets the list of issue comments made into a repository in GitHub. All events of this service are ingested into table `vcs.github.repository.issue_comments`.

### Verify data collection

#### Puller output

A successful initial run has the following output messages for the puller module:

> ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> Reading persisted data
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> No changes have been made in saved state. Returning saved state: {'pulling_date_from_config': '1640995200.0', 'last_pulled_date': '1641852073.0', 'ids': [1009384492]}
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> The collector will start pulling data since 2022-01-10T22:01:13Z
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> Total number of repositories: 2
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> Tag: vcs.github.api.repository.issue_comments
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 2; Number of events received: 60; Number of duplicated events filtered out: 60; Number of events generated and sent: 0; Average of events per second: 0.000.
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 4; Number of events received: 120; Number of duplicated events filtered out: 105; Number of events generated and sent: 15; Average of events per second: 1.307.
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 6; Number of events received: 180; Number of duplicated events filtered out: 160; Number of events generated and sent: 20; Average of events per second: 1.294.
...
```

After the successful execution of the collector, you should be able to see the following log message:

```
INFO InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> Statistics for this pull cycle Number of requests made: 6; Number of events received: 180; Number of duplicated events filtered out: 160; Number of events generated and sent: 20; Average of events per second: 1.294.
```

> ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

> ℹ️ Note that a `Partial` Statistics Report will be displayed after download a page when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> ℹ️ 
> ℹ️ `(Partial) Statistics for this pull cycle Number of requests made: 6; Number of events received: 180; Number of duplicated events filtered out: 160; Number of events generated and sent: 20; Average of events per second: 1.294.`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `since` parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.
</details>

<details>
<summary>Pull Request Service</summary>

This service returns the pull request for each repository, and the associate commits for each pull request. All events of this service are ingested into table `vcs.github.repository.pull_requests` and `vcs.github.repository.pull_requests`.

### Verify data collection

#### Puller output

A successful initial run has the following output messages for the puller module:

> ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> State saved: {'old_persistence_reset_date': '27-Oct-2022', 'pull_requests': {}, 'pull_request_commits': {}}
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Starting data collection every 300 seconds
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> New items found for Commits PR commits -> 1
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Persistence saved for Commits PR commits -> bdb806bd6218552c7b3b6507803e48694b5591b7
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> New items found for Commits PR commits -> 1
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Persistence saved for Commits PR commits -> ccc00f42d56d59bcb375a327f163bb8b737f376d
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> New items found for Commits PR commits -> 17
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Persistence saved for Commits PR commits -> 8f1af254f6fffad8718c7c68a7f67778bc6c6b3f
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> New items found for Commits PR commits -> 1
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Persistence saved for Commits PR commits -> a0d5d609978be33b7b7c37b2ea2a400d6102ccda
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> New items found for Commits PR commits -> 8
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Persistence saved for Commits PR commits -> e99dfc7b7075250ca685106bab18f31f7e62f3c5
...
```

After the successful execution of the collector, you should be able to see the following log message:

```
INFO InputProcess::GithubDataPullerPullRequests(github,444,pull_requests,predefined,all) -> Statistics for this pull cycle Number of requests made: 172; Number of events received: 1333; Number of duplicated events filtered out: 0; Number of events generated and sent: 1333; Average of events per second: 15.183.
```

> ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

> ℹ️ Note that a `Partial` Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> ℹ️ 
> ℹ️ `Partial statistics: Pages retrieved 10, items buffered 300`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `persistence_reset_date` parameter must be changed from the user configuration. This is a free text field. It is recommended to use a reference to the day the persistence is being reset. For further details, go to the settings section.
</details>

<details>
<summary>Releases Service</summary>

This service returns the releases for each repository. All events of this service are ingested into table `vcs.github.repository.releases`.

### Verify data collection

#### Puller output

A successful initial run has the following output messages for the puller module:

> ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> State saved: {'old_persistence_reset_date': '27-Oct', 'releases': {}}
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> GithubDataPullerReleases(github,444,releases,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> Starting data collection every 300 seconds
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> Get Releases function called
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> New items found for repo repo-1 -> 0
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> New items found for repo repo-2 -> 0
....
```

After the successful execution of the collector, you should be able to see the following log message:

```
INFO InputProcess::GithubDataPullerReleases(github,444,releases,predefined,all) -> Statistics for this pull cycle Number of requests made: 30; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.0.
```

> ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

> ℹ️ Note that a `Partial` Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> ℹ️ 
> ℹ️ `Partial statistics: Pages retrieved 10, items buffered 300`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `persistence_reset_date` parameter must be changed from the user configuration. This is a free text field. It is recommended to use a reference to the day the persistence is being reset. For further details, go to the settings section.
</details>

<details>
<summary>SSO Authorization Service</summary>

This service returns Single Sign On authorization for all organizations. All events of this service are ingested into table `vcs.github.organizations.sso_authorizations`.

### Verify data collection

#### Puller output

A successful initial run has the following output messages for the puller module:

> ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> State saved: {'old_persistence_reset_date': '2022-10-16T12:00:00Z', 'sso_authorizations': {}}
INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> No more pages have been detected ahead for org my-organization
INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> New items found for org my-organization -> 58
INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> Persistence saved for org my-organization -> 40487930
.....
```

After the successful execution of the collector, you should be able to see the following log message:

```
INFO InputProcess::GithubDataPullerSSOAuthorizations(github,444,sso_authorizations,predefined,all) -> Statistics for this pull cycle Number of requests made: 2; Number of events received: 58; Number of duplicated events filtered out: 0; Number of events generated and sent: 58; Average of events per second: 80.759.
```

> ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

> ℹ️ Note that a `Partial` Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> ℹ️ 
> ℹ️ `Partial statistics: Pages retrieved 10, items buffered 300`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `persistence_reset_date` parameter must be changed from the user configuration. This is a free text field. It is recommended to use a reference to the day the persistence is being reset. For further details, go to the settings section.
</details>

<details>
<summary>Stargazers</summary>

This service returns information about the users who starts each repository, making it a favorite. All events of this service are ingested into table `vcs.github.repository.stargazers`.

### Verify data collection

#### Puller output

A successful initial run has the following output messages for the puller module:

> ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> State saved: {'old_persistence_reset_date': '2022-11-01T12:34:21Z', 'stargazers': {}}
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> GithubDataPullerStargazers(github,444,stargazers,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> New items found for repo repo-1 -> 9
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> Persistence saved for repo-1 -> 1966093
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> New items found for repo repo-2 -> 0
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> No more pages have been detected ahead for repo repo-3
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> New items found for repo repo-3 -> 0
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> No more pages have been detected ahead for repo repo-4
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> New items found for repo repo-4 -> 0
...
```

After the successful execution of the collector, you should be able to see the following log message:

```
INFO InputProcess::GithubDataPullerStargazers(github,444,stargazers,predefined,all) -> Statistics for this pull cycle Number of requests made: 30; Number of events received: 33; Number of duplicated events filtered out: 0; Number of events generated and sent: 33; Average of events per second: 3.049.
```

> ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

> ℹ️ Note that a `Partial` Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> ℹ️ 
> ℹ️ `Partial statistics: Pages retrieved 10, items buffered 300`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `persistence_reset_date` parameter must be changed from the user configuration. This is a free text field. It is recommended to use a reference to the day the persistence is being reset. For further details, go to the settings section.
</details>

<details>
<summary>Subscribers</summary>

This service returns information about users subscribed to one repository. All events of this service are ingested into table `vcs.github.repository.subscribers`.

### Verify data collection

#### Puller output

A successful initial run has the following output messages for the puller module:

> ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> State saved: {'old_persistence_reset_date': '2022-10-27', 'subscribers': {}}
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> GithubDataPullerSubscribers(github,444,subscribers,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> New items found for repo repo-1 -> 60
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Persistence saved for repo-1 -> 1234236
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> New items found for repo repo-2 -> 54
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Persistence saved for repo-2 -> 2342343
...
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> No more pages have been detected ahead for repo repo-N
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> New items found for repo repo-N -> 1
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Persistence saved for repo-N -> 3453457
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Statistics for this pull cycle Number of requests made: 10; Number of events received: 178; Number of duplicated events filtered out: 0; Number of events generated and sent: 178; Average of events per second: 11.194.
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> The data is up to date!
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Data collection completed. Elapsed time: 15.904 seconds. Waiting for 44.096 second(s) until the next one
```

After the successful execution of the collector, you should be able to see the following log message:

```
INFO InputProcess::GithubDataPullerSubscribers(github,444,subscribers,predefined,all) -> Statistics for this pull cycle Number of requests made: 10; Number of events received: 178; Number of duplicated events filtered out: 0; Number of events generated and sent: 178; Average of events per second: 11.194.
```

> ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

> ℹ️ Note that a `Partial` Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> ℹ️ 
> ℹ️ `Partial statistics: Pages retrieved 10, items buffered 300`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `persistence_reset_date` parameter must be changed from the user configuration. This is a free text field. It is recommended to use a reference to the day the persistence is being reset. For further details, go to the settings section.
</details>

<details>
<summary>Subscription</summary>

This service returns information about the repositories the user (used to make the pulling) is subscribed to. All events of this service are ingested into table `vcs.github.repository.subscriptions`.

> ℹ️ Depending on the kind of user you are using to make the data pulling, it could make no sense for you checking if the user is subscribed to the Organization repositories.

### Verify data collection

#### Puller output

A successful initial run has the following output messages for the puller module:

> ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> State saved: {'old_persistence_reset_date': '2022-10-27', 'subscriptions': {}}
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> Pull Started
WARNING InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) ->  404 Did not found any watchers/subscriptions for the repository
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> No more pages have been detected ahead for repo repo-1
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> New items found for repo repo-1 -> 0
WARNING InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) ->  404 Did not found any watchers/subscriptions for the repository
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> No more pages have been detected ahead for repo repo-2
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> New items found for repo repo-2 -> 0
...
WARNING InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) ->  404 Did not found any watchers/subscriptions for the repository
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> No more pages have been detected ahead for repo repo-N
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> New items found for repo repo-N -> 0
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> Statistics for this pull cycle Number of requests made: 5; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> The data is up to date!
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> Data collection completed. Elapsed time: 8.960 seconds. Waiting for 51.040 second(s) until the next one
```

After the successful execution of the collector, you should be able to see the following log message:

```
INFO InputProcess::GithubDataPullerSubscriptions(github,444,subscriptions,predefined,all) -> Statistics for this pull cycle Number of requests made: 5; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
```

> ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

> ℹ️ Note that a `Partial` Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> ℹ️ 
> ℹ️ `Partial statistics: Pages retrieved 10, items buffered 300`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `persistence_reset_date` parameter must be changed from the user configuration. This is a free text field. It is recommended to use a reference to the day the persistence is being reset. For further details, go to the settings section.
</details>

<details>
<summary>Webhooks</summary>

List of webhooks created by the organization. All events of this service are ingested into table `vcs.github.organizations.webhooks`.

### Verify data collection

#### Puller output

A successful initial run has the following output messages for the puller module:

> ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> Reading persisted data
WARNING InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> Persistence not found, persistence will be initialized
WARNING InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> State saved: {'old_persistence_reset_date': '27-October', 'webhooks': {}}
INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> GithubDataPullerWebhooks(github,444,webhooks,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> No more pages have been detected ahead for org my-organization
INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> New items found for org my-organization -> 1
INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> Persistence saved for org my-organization -> 324567455
.....
```

After the successful execution of the collector, you should be able to see the following log message:

```
INFO InputProcess::GithubDataPullerWebhooks(github,444,webhooks,predefined,all) -> Statistics for this pull cycle Number of requests made: 1; Number of events received: 1; Number of duplicated events filtered out: 0; Number of events generated and sent: 1; Average of events per second: 5.679.
```

> ℹ️ The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

> ℹ️ Note that a `Partial` Statistics Report will be displayed after download a set of 10 pages when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> ℹ️ 
> ℹ️ `Partial statistics: Pages retrieved 10, items buffered 300`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `persistence_reset_date` parameter must be changed from the user configuration. This is a free text field. It is recommended to use a reference to the day the persistence is being reset. For further details, go to the settings section.
</details>

<details>
<summary>Enterprise audit service</summary>

### Description

This service gets the audit log (a sequence of activities) for an enterprise in GitHub.

> ⚠️ This service generates a huge amount of events and it takes a lot of time and requests to the API to be up-to-date with this service. Use the parameter `since` to set a near date as the beginning.

### Devo categorization and destination

All events of this service are ingested into the table `vcs.github.enterprise.audit`.

### Verify data collection

#### Puller output

A successful initial run has the following output messages for the puller module:

> ℹ️ Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Reading persisted data
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> GithubDataPullerAudit(github,444,audit,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Tag: vcs.github.api.organization.audit
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 1; Number of events received: 30; Number of duplicated events filtered out: 0; Number of events generated and sent: 30; Average of events per second: 14.773.
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 2; Number of events received: 60; Number of duplicated events filtered out: 0; Number of events generated and sent: 60; Average of events per second: 14.709.
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 3; Number of events received: 90; Number of duplicated events filtered out: 0; Number of events generated and sent: 90; Average of events per second: 14.685.
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 4; Number of events received: 120; Number of duplicated events filtered out: 0; Number of events generated and sent: 120; Average of events per second: 14.865.
...
```

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message. However, it takes a lot of time to reach the end of this service, as it generates a huge amount of events and starts pulling 90 days back:

```
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Statistics for this pull cycle Number of requests made: 10000; Number of events received: 300000; Number of duplicated events filtered out: 0; Number of events generated and sent: 300000; Average of events per second: 14.865.
```

> ℹ️ The `@devo_pulling_id` value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that `Pull` action in the **Data Search** area of Devo.

> ℹ️ Note that a `Partial` Statistics Report will be displayed after download a page when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> ℹ️ 
> ℹ️ `(Partial) Statistics for this pull cycle Number of requests made: 4; Number of events received: 120; Number of duplicated events filtered out: 0; Number of events generated and sent: 120; Average of events per second: 14.865.`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `since` parameter must be changed in the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.
</details>

<details>
<summary>Organization audit service</summary>

### Description

This service gets the audit log (a sequence of activities) for an organization in Github.

> ⚠️ This service generates a huge amount of events and it takes a lot of time and requests to the API to be up-to-date with this service. Use the parameter `since` to set a near date as the beginning

### Devo categorization and destination

All events of this service are ingested into table `vcs.github.organization.audit`

### Verify data collection

#### Puller Output

A successful initial run has the following output messages for the puller module:

> 📝 Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

```
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Starting the execution of pre_pull()
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Reading persisted data
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> GithubDataPullerAudit(github,444,audit,predefined,all) Finalizing the execution of pre_pull()
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Starting data collection every 60 seconds
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Pull Started
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Tag: vcs.github.api.organization.audit
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 1; Number of events received: 30; Number of duplicated events filtered out: 0; Number of events generated and sent: 30; Average of events per second: 14.773.
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 2; Number of events received: 60; Number of duplicated events filtered out: 0; Number of events generated and sent: 60; Average of events per second: 14.709.
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 3; Number of events received: 90; Number of duplicated events filtered out: 0; Number of events generated and sent: 90; Average of events per second: 14.685.
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> (Partial) Statistics for this pull cycle Number of requests made: 4; Number of events received: 120; Number of duplicated events filtered out: 0; Number of events generated and sent: 120; Average of events per second: 14.865.
...
```

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message. However, it takes a lot of time to reach the end of this service, as it generates a huge amount of events and starts pulling 90 days back:

```
INFO InputProcess::GithubDataPullerAudit(github,444,audit,predefined,all) -> Statistics for this pull cycle Number of requests made: 10000; Number of events received: 300000; Number of duplicated events filtered out: 0; Number of events generated and sent: 300000; Average of events per second: 14.865.
```

> ℹ️ The `@devo_pulling_id` value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that `Pull` action in Loxcope.

> 📝 Note that a `Partial` Statistics Report will be displayed after download a page when the pagination is required to pull all available events. Look for the report without the `Partial` reference.
> 📝 
> 📝 `(Partial) Statistics for this pull cycle Number of requests made: 4; Number of events received: 120; Number of duplicated events filtered out: 0; Number of events generated and sent: 120; Average of events per second: 14.865.`

### Restart the persistence

This service makes use of persistence. To restart the persistence, the `since` parameter must be changed from the user configuration. This field indicates the date from which to start pulling data. For further details, go to the settings section.
</details>

## Collector operations

This section is intended to explain how to proceed with specific operations of this collector.

<details>
<summary>Verify collector operations</summary>

### Initialization

The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.

A successful run has the following output messages for the initializer module:

```
INFO MainProcess::MainThread -> Loading configuration using the following files: {"full_config": "config.yaml", "job_config_loc": null, "collector_config_loc": null}
INFO MainProcess::MainThread -> Using the default location for "job_config_loc" file: "/etc/devo/job/job_config.json"
INFO MainProcess::MainThread -> "/etc/devo/job" does not exists
INFO MainProcess::MainThread -> Using the default location for "collector_config_loc" file: "/etc/devo/collector/collector_config.json"
INFO MainProcess::MainThread -> "/etc/devo/collector" does not exists
INFO MainProcess::MainThread -> Results of validation of config files parameters: {"config": "/path/to/config.yaml", "config_validated": True, "job_config_loc": "/etc/devo/job/job_config.json", "job_config_loc_default": True, "job_config_loc_validated": False, "collector_config_loc": "/etc/devo/collector/collector_config.json", "collector_config_loc_default": True, "collector_config_loc_validated": False}
INFO MainProcess::MainThread -> {"build_time": "UNKNOWN", "os_info": "Linux-5.14.0-1054-oem-x86_64-with-glibc2.31", "collector_name": "example_collector", "collector_version": "2.0.0", "collector_owner": "integrations_factory@devo.com", "started_at": "2022-10-26T15:24:51.619878Z"}
INFO MainProcess::MainThread -> [OUTPUT] OutputMultiprocessingController::__init__ Configuration -> {'devo_1': {'type': 'devo_platform', 'config': {'address': 'devo_address', 'port': 443, 'type': 'SSL', 'chain': 'chain_file', 'cert': 'cert_file', 'key': 'key_file', 'concurrent_connections': 1, 'period_sender_stats_in_seconds': 300, 'activate_final_queue': False, 'threshold_for_using_gzip_in_transport_layer': 1.1, 'compression_level': 6, 'compression_buffer_in_bytes': 51200, 'generate_metrics': False}}}
INFO MainProcess::MainThread -> OutputProcess - Starting thread (executing_period=60s)
INFO MainProcess::MainThread -> InputProcess - Starting thread (executing_period=60s)
INFO OutputProcess::MainThread -> Process started
INFO InputProcess::MainThread -> Process Started
INFO InputProcess::MainThread -> There is not defined any submodule, using the default one with value "none"
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: devo_address
INFO InputProcess::MainThread -> GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) Starting the execution of init_variables()
INFO InputProcess::MainThread -> Validating settings from collector definitions
INFO InputProcess::MainThread -> Validating settings from user configuration
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: devo_address
INFO InputProcess::MainThread -> Populating collector_variables store
INFO InputProcess::MainThread -> Created new rate_limiter 1 seconds, 1 calls
INFO InputProcess::MainThread -> Initialization of api_base_url has started.
INFO InputProcess::MainThread -> Base url is not provided in the config.yaml. Considering the base url specified in the collector definitions
INFO InputProcess::MainThread -> api_base_url has been initialized
INFO InputProcess::MainThread -> GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) Finalizing the execution of init_variables()
INFO InputProcess::MainThread -> InputThread(github,444) - Starting thread (execution_period=600s)
INFO InputProcess::MainThread -> ServiceThread(github,444,issue_comments,predefined) - Starting thread (execution_period=600s)
INFO InputProcess::MainThread -> GithubApiserverBasePullerSetup(example_collector,github#444,issue_comments#predefined,all) -> Starting thread
INFO InputProcess::MainThread -> GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) - Starting thread
WARNING InputProcess::GithubDataPullerIssueComments(github,444,issue_comments,predefined,all) -> Waiting until setup will be executed
INFO InputProcess::GithubApiserverBasePullerSetup(example_collector,github#444,issue_comments#predefined,all) -> Starting the execution of setup()
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: devo_address
INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread
INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> Starting thread
INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread
```

### Events delivery and Devo ingestion

The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.

A successful run has the following output messages for the initializer module:

```
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Sender: SyslogSender(standard_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Standard - Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 44 (elapsed 0.007 seconds)
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Sender: SyslogSender(internal_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Internal - Total number of messages sent: 1, messages sent since "2022-06-28 10:39:22.516313+00:00": 1 (elapsed 0.019 seconds)
```

> ℹ️ By default, these information traces will be displayed every 10 minutes.

#### Sender services

The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (`internal`, `standard`, and `lookup`). This collector uses the following Sender Services:

| **Sender services** | **Description** |
| --- | --- |
| `internal_senders` | In charge of delivering internal metrics to Devo such as logging traces or metrics. |
| `standard_senders` | In charge of delivering pulled events to Devo. |

#### Sender statistics

Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:

| **Logging trace** | **Description** |
| --- | --- |
| `Number of available senders: 1` | Displays the number of concurrent senders available for the given Sender Service. |
| `sender manager internal queue size: 0` | Displays the items available in the internal sender queue.<br>> ℹ️ This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders. |
| `Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds)` | Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:<br>- 44 events were sent to Devo since the collector started.
- The last checkpoint timestamp was `2022-06-28 10:39:22.511671+00:00`.
- 21 events where sent to Devo between the last UTC checkpoint and now.
- Those 21 events required `0.007 seconds` to be delivered.<br>> ℹ️ By default these traces will be shown every 10 minutes. |
</details>

<details>
<summary>Check memory usage</summary>

To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.

- The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
- The global pressure of the available memory is displayed in the `global` value.
- All metrics (Global, RSS, VMS) include the value before freeing and after `previous -> after freeing memory`

```
INFO InputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(34.50MiB -> 34.08MiB), VMS(410.52MiB -> 410.02MiB)
INFO OutputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(28.41MiB -> 28.41MiB), VMS(705.28MiB -> 705.28MiB)
```

> ℹ️ Differences between `RSS` and `VMS` memory usage:
> ℹ️ 
> ℹ️ - `RSS` is the Resident Set Size, which is the actual physical memory the process is using
> ℹ️ - `VMS` is the Virtual Memory Size which is the virtual memory that process is using
</details>

<details>
<summary>Enable/disable the logging debug mode</summary>

Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.

- To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector.
- To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector.

For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode.
</details>

<details>
<summary>Troubleshooting</summary>

### Restarting the collector with an old configuration after upgrading from version 1.x to 2.x

The format of the configuration file has changed from version 1.x to version 2.x. It is not possible to use directly an old config file from 1.x for the deployment of a 2.x collector. If you try to start a new collector using an old config file, you will see an error like this in the log file:

`2022-10-26T15:02:22.850 ERROR InputProcess::MainThread -> InputThread(github,123) - No service definition found for service name: repository`  
`2022-10-26T15:02:22.850 ERROR InputProcess::MainThread -> InputThread(github,123) - No service definition found for service name: organization`

You should create a new config file, using the old version 1.x file as a base to fill in the correspondent values in the new 2.x format.

### Runtime errors

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the common errors for the current services.

| **Error type** | **Error ID** | **Error message** | **Cause** | **Solution** |
| --- | --- | --- | --- | --- |
| `InitVariablesError` | 0 | `The internal config did not pass the format validation. Contact with Support.` | The internal configuration of the collector did not match the required JSON Schema to pass validation. | This is an internal issue. Contact with Devo Support team. |
| 1 | `The user config did not pass the format validation. Check error traces for details and visit our documentation.` | The user configuration of the collector did not match the required JSON Schema to pass validation. | Check the error traces to see the field that does not match the format validation and make the necessary change. |
| 7 | `api_base_url not of expected type: str` | `override_api_url` is not of type string. | Change `override_api_url` to be of type string. The default API URL could also work; to use it, just remove `override_api_url` from user configuration. |
| 8 | `api_base_url must match regex: {api_url_regex}` | `override_api_url` does not match the RegEx. | Make `override_api_url` match the next RegEx:  
`https:\/\/([a-z0-9]+[.]{1})+[a-z]+[\/]{1}`<br>The default API URL could also work; to use it, just remove `override_api_url` from user configuration. |
| `SetupError` | 100 | `The remote data is not pullable with the given credentials. Check the error traces for details.` | The personal access `token` is not set, in user configuration. | Follow the steps described in this document to get a valid the Personal Access Token from Github and use that `token` in the user configuration file. |
| 101 | `The token/header/authentication was refreshed but is still expired. Check the error traces for details.` | The personal access `token` is not valid, in user configuration. | Follow the steps described in this document to get a valid the Personal Access Token from Github and use that `token` in the user configuration file. |
| 102 | `Regenerate the expired/invalid token from Developer settings in your Github account.` | The personal access `token` is not valid, in user configuration. | Follow the steps described in this document to get a valid the Personal Access Token from Github and use that `token` in the user configuration file. |
| 201 | `Error, org {org} is not valid for user {username}` | The `username` set in user configuration is not valid. | Follow the steps described in this document to get a valid Personal Access Token from Github and use that `username` in the user configuration file. |
| `RequestError` | 600 | `Error in the connection, API response code {status_code}: {error_message} for {endpoint}` | There was an unexpected runtime error. | Read the message and try to understand the error. If it is a `5xx` error it means the error is in the API server side; wait and it will be solved.<br>Else, please, contact with Devo Support team. |
| 601 | `Neither "x-ratelimit-reset" nor "retry-after" headers found in the response headers. This is an unexpected response.` | There was an unexpected runtime error thrown by the API, related to the rate limiter. | This error is not handled and it should be. Please, contact with Devo Support team. |
</details>

## Change log

| **Release** | **Released on** | **Release type** | **Details** | **Recommendations** |
| --- | --- | --- | --- | --- |
| `v3.0.0` | 7/2/2024 | [NEW FEATURE]  
[IMPROVEMENT]  
[BUG FIXING] | **New features**<br>- Audit substituted by new improved Organization Audit service
- Requests now send Time Zone UTC explicitly<br>**Bug fix**<br>- INT-2482 Github events delayed (Organization Audit)
- INT-2177 Github collector does not specify timezone<br>**Improvements**<br>- Upgraded DCSDK from 1.11.1 to 1.12.1
  - Added new sender for relay in house + TLS
  - Added persistence functionality for gzip sending buffer
  - Added Automatic activation of gzip sending
  - Improved behaviour when persistence fails
  - Upgraded DevoSDK dependency
  - Fixed console log encoding
  - Restructured python classes
  - Improved behaviour with non-utf8 characters
  - Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB)
  - New persistence format/structure (compression in some cases)
  - Removed dmesg execution (It was invalid for docker execution) | `Recommended version` |
| `v2.3.0` | 5/7/2024 | [NEW FEATURE]  
[IMPROVEMENT]  
[BUG FIXING] | **New features**<br>- Added new Enterprise Audit service<br>**Bug fix**<br>- Fix missing parentheses<br>**Improvements**<br>- Upgraded DCSDK from 1.10.0 to 1.11.1:
  - Introduced pyproject.toml
  - Added requirements-dev.txt
  - Fixed error in pyproject.toml related to project scripts endpoint
  - Updated DevoSDK to v5.1.9
  - Fixed some bug related to development on MacOS
  - Added an extra validation and fix when the DCSDK receives a wrong timestamp format
  - Added an optional config property for use the Syslog timestamp format in a strict way
  - Updated DevoSDK to v5.1.10
  - Fix for SyslogSender related to UTF-8
  - Enhace of troubleshooting. Trace Standardization, Some traces has been introduced.
  - Introduced a machanism to detect "Out of Memory killer" situation.
  - Changed default number for connection retries (now 7)
  - Fix for Devo connection retries
  - Added extra check for not valid message timestamps | `Update` |
| `v2.1.0` | 8/14/2023 | [IMPROVEMENT] | - Upgraded DCSDK from 1.4.4 to 1.9.1:
  - Store lookup instances into DevoSender to avoid creation of new instances for the same lookup
  - Ensure service_config is a dict into templates
  - Ensure special characters are properly sent to the platform
  - Changed log level to some messages from info to debug
  - Changed some wrong log messages
  - Upgraded some internal dependencies
  - Changed queue passed to setup instance constructor
  - Ability to validate collector setup and exit without pulling any data
  - Ability to store in the persistence the messages that couldn't be sent after the collector stopped
  - Ability to send messages from the persistence when the collector starts and before the puller begins working
  - Ensure special characters are properly sent to the platform
  - Added a lock to enhance sender object
  - Added new class attrs to the **setstate** and **getstate** queue methods
  - Fix sending attribute value to the **setstate** and **getstate** queue methods
  - Added log traces when queues are full and have to wait
  - Added log traces of queues time waiting every minute in debug mode
  - Added method to calculate queue size in bytes
  - Block incoming events in queues when there are no space left
  - Send telemetry events to Devo platform
  - Upgraded internal Python dependency Redis to v4.5.4
  - Upgraded internal Python dependency DevoSDK to v5.1.3
  - Fixed obfuscation not working when messages are sent from templates d
  - New method to figure out if a puller thread is stopping
  - Upgraded internal Python dependency DevoSDK to v5.0.6
  - Improved logging on messages/bytes sent to Devo platform
  - Fixed wrong bytes size calculation for queues
  - New functionality to count bytes sent to Devo Platform (shown in console log)
  - Upgraded internal Python dependency DevoSDK to v5.0.4
  - Fixed bug in persistence management process, related to persistence reset
  - Aligned source code typing to be aligned with Python 3.9.x
  - Inject environment property from user config
  - Obfuscation service can be now configured from user config and module definiton
  - Obfuscation service can now obfuscate items inside arrays | `Update` |
| `v2.0.0` | 10/28/2022 | [NEW FEATURE]  
[IMPROVEMENT]    
[BUG FIXING]  
[VULNS] | **New features**<br>- Actions data source: Lists all Github Actions “workflows runs” for a repository.
- CodeScan data source: List alerts after analyzing the code in a repository to find security vulnerabilities and coding errors.
- Dependabot Alerts data source: List alerts created due to detecting vulnerable dependencies or malware.
- Dependabot Secrets data source: List all secrets available in an organization without revealing their encrypted values.
- A new Rate Limiter service has been added providing a higher granularity.
- A feature has been added to initialize the persistence of services in a granular way through the configuration file.<br>**Improvements**<br>- The pulling logic for all the services have been improved reducing the risk of duplicates.
- Improved error management for connection issues.
- Upgrade underlying `Devo Collector SDK` from `v1.1.3` to `v1.4.1`.
- Upgraded the underlying `DevoSDK` package to `v3.6.4` and dependencies, this upgrade increases the resilience of the collector when the connection with Devo or the Syslog server is lost. The collector can reconnect in some scenarios without running the auto reboot service.
- Support for stopping the collector when a `GRACEFULL_SHUTDOWN` system signal is received.
- Re-enabled the logging to `Devo.collector.out` for Input threads.
- Added functionality for detecting some system signals for starting the controlled stopping.
- Added log traces for knowing system memory usage and execution environment status. Added more details in logs.
- Added a new template functionality for easing the developing collectors (not used by this collector).
- Refactored source code structure.
- The Docker container exits with the proper error code.
- Minimized probabilities of suffering a `DevoSDK` bug related to "sender" to be null.
- When an exception is raised by the Collector Setup, the collector retries after 5 seconds. For consecutive exceptions, the waiting time is multiplied by 5 until hits 1800 seconds, which is the maximum waiting time allowed. No maximum retries are applied.
- When an exception is raised by the Collector Pull method, the collector retries after 5 seconds. For consecutive exceptions, the waiting time is multiplied by 5 until hits 1800 seconds, which is the maximum waiting time allowed. No maximum retries are applied.
- When an exception is raised by the Collector pre-pull method, the collector retries after 30 seconds. No maximum retries are applied.<br>**Bug fixing**<br>- Fixed pagination and persistence bugs when pulling thousands of target repositories and events.
- Fixed a bug in the Webhook data source that prevented complete downloading.
- Fixed bugs related to ingestions outages.<br>**Vulnerabilities mitigation**<br>- `CVE-2022-1664`
- `CVE-2021-33574`
- `CVE-2022-23218`
- `CVE-2022-23219`
- `CVE-2019-8457`
- `CVE-2022-1586`
- `CVE-2022-1292`
- `CVE-2022-2068`
- `CVE-2022-1304`
- `CVE-2022-1271`
- `CVE-2021-3999`
- `CVE-2021-33560`
- `CVE-2022-29460`
- `CVE-2022-29458`
- `CVE-2022-0778`
- `CVE-2022-2097`
- `CVE-2020-16156`
- `CVE-2018-2503` | `Update`  
`(breaking release)` |