Microsoft Azure collector

Service description

Microsoft Azure is an ever-expanding set of cloud computing services to help your organization meet its business challenges. Azure gives you the freedom to build, manage, and deploy applications on a massive, global network using your preferred tools and frameworks.

Data source description

You can use the Microsoft Azure collector to send the following types of data to your Devo domain. Once the gathered information arrives at Devo, it will be processed and included in different tables in the associated Devo domain. 

Virtual Machine metrics

With the advantages of the Azure API, you can obtain basic metrics about your deployed Virtual Machines, gather them on our platform, and transform it as required.

Event Hub Services

Thanks to the Azure Event Hub resource combined with the Diagnostic Settings option offered by Azure, you can easily centralize all the supported services in the Event Hub and send them to Devo. Although Event Hub is the service used to centralize data, it also generates data that can be sent to itself.

Microsoft Azure services

Listed in the table below are some service names, details, and how the Devo platform treats the data.

Setup

The Microsoft Azure collector centralizes the data with an Event Hub using the Azure SDK. To use it, you need to configure the resources in the Azure Portal and set the right permissions to access the information.

• Virtual Machine metrics
• Event Hub events

Virtual Machine metrics

Getting credentials

To log in to the Azure subscription, the collector uses a Service Principal object. You need to get the subscription ID, Active Directory ID, Application ID (service principal identificator) and the secret (service principal "password"). To get them, follow these steps:

1. Log in to your Azure account and search for Azure Active Directory.
   
   
   

2. Now, click App registrations in the left menu and click the app (or Service Principal) that you are going to use.
   
   
   
3. In the Overview area, find the Application (client) ID and the Directory (tenant) ID.
   
   
   

4. Now click Certificates & Secrets on the menu and create a new secret.
   
   
   

   Don't forget to save the secret, it will be only shown upon creation.

5. Get the subscription ID searching for Subscriptions on the home page.
   
   
   
6. Find the correct subscription and note down the subscription ID.
   
   
   

Setting up permissions

1. After creating the App registration (or Service Principal), go to the desired Resource Group (or subscription if you want to retrieve metrics from all the available virtual machines).
   

2. Select Access control (IAM) in the left menu and click Add.
   
   
   
3. Select at least the Reader role and choose the previously created App registration.
   

4. Confirm the changes.

Event Hub events

Getting credentials

1. In your Azure account, search the Event Hubs service and click on it. 

   
   

   

2. Create an Event Hub resource per region (repeat the steps below for each region):

   • Click Add.
     
     

     

   • Fill the mandatory fields keeping in mind that the Event Hub must be in the same region as the resources that you are going to monitor (and only need one per region).

     The Throughput Units option refers to the ingress/egress limit in MB/s (each unit is 1 MB/s or 1000 events/second ingress, 2 MB/s, or 4096 events/second egress). You should adjust it according to the data volume (this can be modified later).

     

   • The previous steps create an EventHub namespace; now go to Event Hubs, search the created one and click on it.
     
     

     

   • Now click on the + Event Hub button and create a new resource. You only need to fill the Name and Partition Count fields (the Partition Count field will divide the data into different partitions to make it easier to read large volumes of data).
     

     Write down the EventHub name to be used later in the configuration file.

     
     

     

   • Once the Event Hub is created in the namespace, click it and select Consumer Group in the left menu.
     
     

     

   • Here you will see the Event Hub consumer groups. This will be used by the collector (or other applications) for reading data from the Event Hub. Write down the Consumer group name that you will use later in the configuration file.
     
     

   • Now, in the Event Hub Namespace, click on Shared access policies, search the default policy named RootManageSharedAccessKey and click it.
     
     

     
   • Copy and write down the primary (or secondary) connection string to be used later in the configuration file.

Setting up the Event Hubs

1. Now, search the Monitor service and click on it.
   
   

   

2. Click the Diagnostic Settings option in the left area.

3. A list of the deployed resources will be shown. Search for the resources that you want to monitor, select them, and click Add diagnostic setting.
   
   
   

4. Type a name for the rule and check the required category details (logs will be sent to the cloud.azure.eh.events table, and metrics will be sent to the cloud.azure.eh.metrics table).
   
   
   
5. Check Stream to an Event Hub, and select the corresponding Event hub namespace, Event hub name and Event hub policy name.
   
   
   
6. Click Save to finish the process.
   
   

Run the collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure will be required as part of the setup procedure (it can be created under any directory):

<any_directory>
└── devo-collectors/
    └── azure/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        └── config/ 
            └── config-azure.yaml

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <any directory>/devo-collectors/azure/certs. Learn more about security credentials in Devo here.

Editing the config-azure.yaml file

In the config-azure.yaml file, replace the <app_id> , <active_directory_id>, <subscription_id> and <secret> values and enter the ones that you got in the previous steps. In the <short_unique_identifier> placeholder, enter the value that you choose.

globals:
  debug: false
  id: not_used
  name: azure
  persistence:
    type: filesystem                                             # File system persistence ON
    config:
      directory_name: state                                      # Directory where the persistence will be saved in case of using filesystem
outputs:
  devo_1:                                                        # Cloud Devo config EU (for US use us.elb.relay.logtrust.net)
    type: devo_platform
    config:        
      address: eu.elb.relay.logtrust.net
      port: 443
      type: SSL
      chain: chain.crt
      cert: <your_domain>.crt
      key: <your_domain>.key
inputs:
  azure:
    id: <short_unique_identifier>                                # The value of this field will be used internally for having independent persistence areas
    enabled: true
    requests_per_second: 5                                       # Setup how many request API por second
    autoconfig:
      enable: true                                               # Autoconfig always True for this collector
      refresh_interval_in_seconds: 600                           # Setting up the time for executing autoconfig interval. 60sec is recommended.
    credentials:
      app_id: "<app_id_value>"                                   # Service principal identificator. Obtained in the Setup steps.
      active_directory_id: "<active_directory_id_value>"         # Active directory identificator. Obtained in the Setup steps.
      subscription_id: "<subscription_id_value>"                 # Subscription identificator. Obtained in the Setup steps.
      secret: "<secret_value>"                                   # Service principal secret. Obtained in the Setup steps.
    services:                                                    # Services available for this collector.
      my_service_1:                                              # Service name, has only aclarative purpose, can be any string.
        request_period_in_seconds: 60                            # Setting up time interval between API requests. 60sec is recommended.    
        types:                                                   # Kind of data that will be retrieved (can be "eh_services" or "vm_metrics", both are shown in this example)
          - eh_services
        queues:                                                  # List of eventHubs that will be readed. Needed because you can read more than one EventHub in the same Namespace
          queue_a:                                               # Queue name. As service name has only aclarative purpose, can be any string.
            event_hub_name: "<event_hub_name_value>"             # Name of the EventHub that will be readed, obtained in the Setup steps.
            connection_str: Endpoint=sb://<domain_name>.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<shared_access_key_value>    # Connection string of the EventHub that will be readed, obtained in the Setup steps.
            consumer_group: "<consumer_group_value>"             # Consumer group of the EventHub that will be used, obtained in the Setup steps.
            tag: my.app.azure.sample                             # Optional. Custom tag if you want to send the event to other Devo table.
      my_service_2:
        request_period_in_seconds: 60
        types:
          - vm_metrics

Download the Docker image

The collector should be deployed as a Docker container. Click here to download the Docker image of the collector as a .tgz file.

Use the following command to add the Docker image to the system:

gunzip -c collector-azure-docker-image-<version>.tgz | docker load

The Docker image can be deployed on the following services:

• Docker

• Docker Compose

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/azure/

docker run \
--name collector-azure \
--volume $PWD/certs:/devo-collector/certs \
--volume $PWD/config:/devo-collector/config \
--volume $PWD/state:/devo-collector/state \
--env CONFIG_FILE=config-azure.yaml \
--rm -it docker.devo.internal/collector/azure:<version>

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/azure/ directory.

version: '3'
services:
  collector-azure:
    build:
      context: .
      dockerfile: Dockerfile
    image: docker.devo.internal/collector/azure:${IMAGE_VERSION:-latest}
    container_name: collector-azure
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config-azure.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/azure/ directory:

IMAGE_VERSION=<version> docker-compose up -d