Microsoft Azure collector

Microsoft Azure is an ever-expanding set of cloud computing services to help your organization meet its business challenges. Azure gives you the freedom to build, manage, and deploy applications on a massive, global network using your preferred tools and frameworks. Azure generates monitoring information from its resources and thanks to the Diagnostics Settings option, users can configure that information, centralizing, and working with it in different ways.

We use an Event Hub as a means to gather the data and make it easier to send it to Devo. Although Event Hub is the service used to centralize data, it also generates data that can be sent to itself. 

You can use the Microsoft Azure collector to send this information to your Devo domain. Once the gathered information arrives at Devo, it will be processed and included in different tables in the associated Devo domain. 

Microsoft Azure services

Setup

Structure

The collector will be executed as a Docker container, and some files will be necessary for the proper functioning of the container. The following directory structure will be required as part of the setup procedure (it can be created under any directory):

<any_directory>
└── devo-collectors/
    └── azure/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        └── config/
            └── config-azure.yaml

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <any directory>/devo-collectors/azure/certs. Learn more about security credentials in Devo here.

Configuration

Prerequisites

To configure the required settings to send your data to your Devo domain, you'll need:

• An Azure subscription.
• An Azure user with at least "Contributor" privileges.

Data source setup

In order to make the collector run properly, the data source (in this case Microsoft Azure) must have several service structures configured. Specifically, an Event Hub resource per "region" will be required.

Currently, the required Azure configuration can only be done manually, following these steps:

1. In your Azure account, search the Event Hubs service and click on it. 

   
   

   

2. Create an Event Hub resource per region (repeat the steps below for each region):

   • Click Add.
     
     

     

   • Fill the mandatory fields keeping in mind that the Event Hub must be in the same region as the resources that you are going to monitor (and only need one per region).

     The Throughput Units option refers to the ingress/egress limit in MB/s (each unit is 1 MB/s or 1000 events/second ingress, 2 MB/s, or 4096 events/second egress). You should adjust it according to the data volume (this can be modified later).

     

   • The previous steps create an EventHub namespace; now go to Event Hubs, search the created one and click on it.
     
     

     

   • Now click on the + Event Hub button and create a new resource. You only need to fill the Name and Partition Count fields (the Partition Count field will divide the data into different partitions to make it easier to read large volumes of data).
     

     Write down the EventHub name to be used later in the configuration file. If you modify the Partition Count field, remember to add it to the config file later.

     
     

     

   • Now, in the Event Hub Namespace, click on Shared access policies, search the default policy named RootManageSharedAccessKey and click it.
     
     

     
   • Copy and write down the primary (or secondary) connection string to be used later in the configuration file.

3. Now, search the Monitor service and click on it.
   
   

   

4. Click the Diagnostic Settings option in the left area.

5. A list of the deployed resources will be shown. Search for the resources that you want to monitor, select them, and click Add diagnostic setting.
   
   
   

6. Type a name for the rule and check the required category details (logs will be sent to the cloud.azure.eh.events table, and metrics will be sent to the cloud.azure.eh.events table).
   
   
   
7. Check Stream to an Event Hub, and select the corresponding Event hub namespace, Event hub name and Event hub policy name.
   
   
   
8. Click Save to finish the process.
   
   
   

Config file examples

The configuration file config-azure.yaml has two sections, one section used to generate internal logs and send the data (base configuration) and another section with the necessary information to connect with the subscriptions or APIs to retrieve the data. (custom configuration).

The base configuration section is as follows:

config:
    path: ./config                                    # <- relative path where configuration will be stored
    version: 1
devo_connection:
    address: collector-eu.devo.io                     # <- URL used for sending messages
    port: 443
    type: SSL                                         # <- SSL for using certificates or TCP for sending without encryption
    chain: ./certs/chain.crt                          # \
    cert: ./certs/<devo_domain>.crt                   #   > Credential files downloaded from a Devo domain
    key: ./certs/<devo_domain>.key                    # /
log:
    backup_count: 5
    devo_out: 'true'
    devo_stats: 'true'
    format: '%(asctime)s %(levelname)s %(message)s'
    level: info
    max_size: 2097152
    path: ./log                                       # <- Directory to be used for storing the application internal logs
    stdout: 'false'
state:
    path: ./state                                     # <- Directory to be used for storing the "state" files

The custom configuration section can be set with many different values depending on the collector and their requirements. Find below an example of the custom configuration section file:

custom:
  inputs:
    azure:
      id: 1                                           # <- "input_id", used for internal identifications
      enabled: true                                   # <- If "false" value this "input" will not be used by collector
      services:
        my_service:                                   # <- Descriptive name of the services that you are going to monitor
          types:                                      # <- Kind of data that will be retrieved (currently azure only uses "eh_services")
            - eh_services
          regions:
            eu_west:                                  # <- EventHub can only receive data from resources in the same region. This field indicates the region that will be monitored. (Nomenclature is not strict)
              event_hub_name: "name"                  # <- EventHub name previously obtained in the Configuration section
              connection_str: "connection string"     # <- EventHub connection string, previously obtained in the Configuration section
              partition_number: 2                     # <- MANDATORY parameter if you modify the partition number in the EventHub creation step

The following is a complete example where we configure the "network security groups" and the "Load Balancer" resources from "Europe West" and "North Europe" regions:

config:
    path: ./config                                    # <- relative path where configuration will be stored
    version: 1
devo_connection:
    address: collector-eu.devo.io                     # <- URL to use for sending messages
    port: 443
    type: SSL                                         # <- SSL for using certificates or TCP for sending without encryption
    chain: ./certs/chain.crt                          # \
    cert: ./certs/<devo_domain>.crt                   #   > Credential files downloaded from a Devo domain
    key: ./certs/<devo_domain>.key                    # /
log:
    backup_count: 5
    devo_out: 'true'
    devo_stats: 'true'
    format: '%(asctime)s %(levelname)s %(message)s'
    level: info
    max_size: 2097152
    path: ./log                                       # <- Directory to be used for storing the application internal logs
    stdout: 'false'
state:
    path: ./state                                     # <- Directory to be used for storing the "state" files
custom:
  inputs:
    azure:
      id: 1                                           # <- "input_id", used for internal identifications
      enabled: true                                   # <- If "false" value this "input" will not be used by collector
      services:
        all_nsg_and_lb:                               # <- Descriptive name of the services that you are going to monitor
          types:                                      # <- Kind of data that will be retrieved (currently azure only uses "eh_services")
            - eh_services
          regions:
            eu_west:                                  # <- EventHub can only receive data from resources in the same region. This field indicates the region that will be monitored. (Nomenclature is not strict)
              event_hub_name: "name"                  # <- EventHub name previously obtained in the Configuration section
              connection_str: "connection string"     # <- EventHub connection string, previously obtained in the Configuration section
            eu_north:
              event_hub_name: "name"
              connection_str: "connection string"
              partition_number: 3

Run the collector

The collector should be deployed as a Docker container. Click here to download the Docker image of the collector as a .tgz file.

Use the following command to add the Docker image to the system:

gunzip -c collector_cloud.tgz | docker load

The Docker image can be deployed on the following services:

• Docker

• Docker Compose

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/azure/

docker run \
--name collector-azure \
--volume $PWD/certs:/devo-collector/certs \
--volume $PWD/config:/devo-collector/config \
--env CONFIG_FILE=config-azure.yaml \
--rm -it docker.devo.internal/collector/cloud:<version>

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/azure/ directory.

version: '3'
services:
  collector-azure:
    image: docker.devo.com/collector/cloud:${IMAGE_VERSION:-latest}
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config-azure.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/azure/ directory:

IMAGE_VERSION=<version> docker-compose up -d