AWS S3 Buckets
Devo furnishes you with Python scripts that you deploy as a function on AWS Lambda to listen for changes in an AWS S3 bucket. New bucket objects are detected, collected, tagged, and forwarded securely to the Devo Cloud.
Due to the nature of services logging to S3, there will be a time gap from the generation of the event in the original source and its arrival to Devo. Log events will only be ingested once they are written to the S3 bucket. You should keep this in mind when searching log events by time range and when setting write frequency.
This article takes you step-by-step through the configuration process:
Download the Devo domain certificate files
In the Devo web application, go to Administration → Credentials → X.509 Certificates and download the X.509 Certificate, Private Key, and Chain CA to a new folder.
Download the script files
Click here to download the aws_s3_lambda_devo.zip file that contains the necessary script and configuration files.
Decompress the zip file and copy these three files to the folder where you saved the Devo domain certificates:
- config.ini.example
- devo_engine.py
- lambda_function.py
Have a look at the README for a description of these files.
Edit and rename the config.ini.example file
Open the config.ini.example file in an editor and edit the values for the following parameters.
Parameter Description client_key The name of the Devo domain private key file.
Ex: devo_domain.key
client_crt The name of the Devo domain certificate file.
Ex: devo_domain.key
chain The name of the Devo domain Chain CA file.
This is usually chain.crt.
realy_add This is the host address for the Devo Cloud for the region you are using. It should be one of:
- USA: us.elb.relay.logtrust.net
- Europe: eu.elb.relay.logtrust.net
- South America: collector-sa.devo.io
relay_port The inbound port number of the Devo Platform host should always be 443. tag This is the Devo tag that corresponds to the technology that generated the events you are sending to Devo. There are hundreds of supported technologies.
There are also Devo tags for events generated by two Amazon services; CloudFront and the Elastic Load Balancer. The first four levels of these tags are fixed and the final two levels should be assigned so as to identify the specific event source. These two levels will appear in the region and instance columns of the resulting data table in Devo.
Amazon CloudFront CDM: web.aws.cloudfront.access-w3c.<region>.<instance>
AWS Elastic Load Balancer: web.aws.elb.access.<region>.<instance>Example: web.aws.elb.access.eu-west-1.pro-frontend-elb
For log files in common event format (CEF), it is not necessary to set this parameter and you can just leave it blank. Just be sure that the technology is one that we support in CEF.
In the case that there is no Devo tag that corresponds to the event's technology, you can assign a tag that starts with my.app. In this case, the event's fields will not be parsed.
- Save the file as config.ini in the folder where the domain certificates and Python scripts are saved. Delete the original config.ini.example file.
Prepare a ZIP file for upload
You should have a folder with only the following six files: your updated and renamed configuration file, two Python script files, and the three certificate files you downloaded from your Devo domain. Note that two of the certificate files should have the name of your Devo domain (domain_name in the example below).
Create a ZIP file containing only these six files, and name it whatever you like.
Create a new Lambda function
This procedure guides you through creating the new Lambda function that will monitor the S3 bucket for changes.
Create a new AWS Lambda function in the same zone in which the S3 bucket resides.
Click Blueprints, then click the s3-get-object-python blueprint tile.
Click the Configure button. The next page contains three sections; Basic information, S3 trigger, and Lambda function code.
In the Basic information section, enter a Name for the new function.
If using an existing role, make sure that it has Lambda execution and S3 read permissions.
If not using an existing role, create a new one. Under Role, select Create new role from AWS Policy Templates. Enter a role name and select Amazon S3 object read-only permissions as the Policy Template.
In the S3 trigger section, select the Bucket that contains the events, set the Event type to All object create events, then select Enable trigger.
Click Create function. The next page contains several sections in which you configure the details of your new function.
Modify the Function code section as indicated below and for Function package, click Upload to select the .zip file you created earlier. Then, click Save to upload the file.
- In the Execution role section, select the role you specified/created for the function. In the Basic settings section, set the Memory and Timeout to an interval that is close to, but less than, the event creation frequency. For example, if the log file creation frequency is 5 minutes, set the Timeout to 4 minutes and 30 seconds. In the Network section, select No VPC for the VPC value.
Click Save.
Now you can confirm that the Lambda function has been correctly associated to the bucket. Go to S3 and open the bucket. In the bucket's Properties tab, make sure that there's an active notification associated with Events.
If there is no active notification, click the Events tile, then click Add notification. Set up a new event as shown below and click Save.
Now, every time there a new object file is written to the S3 bucket, it will be sent to your Devo domain with the tag specified in the config.ini file.