AWS S3 Buckets

Devo furnishes you with Python scripts that you deploy as a function on AWS Lambda to listen for changes in an AWS S3 bucket. New bucket entries are detected, collected, tagged, and forwarded securely to the Devo Cloud.   

Due to the nature of services logging to S3, there will be a time gap from the generation of the event in the original source and its arrival to Devo. Log events will only be ingested once they are written to the S3 bucket. You should keep this in mind when searching log events by time range and when setting write frequency.

This article takes you step-by-step through the configuration process:

Download the Devo domain certificate files

In the Devo web application, go to Administration → Credentials → X.509 Certificates and download the X.509 Certificate, Private Key, and Chain CA to a new folder.

Download the script files

Click here to download the lt_aws_s3_lambda script files. They are contained in a ZIP file. 

Decompress the ZIP file and copy these three files to the folder where you saved the Devo domain certificates:

• config.ini.example
• lt_engine.py
• lt_s3_lambda.py

You can ignore the other two files, however, the README contains the same instructions as in this article.

Edit and rename the config.ini.example file

1. Open the config.ini.example file in an editor and edit the values for the following parameters.

   Parameter	Description
   client_key	The name of the Devo domain private key file.
   client_crt	The name of the Devo domain certificate file.
   chain	The name of the Devo domain Chain CA file.
   realy_add	This is the host address for the Devo Cloud for the region you are using. It should be one of: us.elb.relay.logtrust.net eu.elb.relay.logtrust.net es.elb.relay.logtrust.net
   relay_port	The inbound port number of the Devo Cloud host.
   tag	The Devo tag that should be applied to events from the AWS S3 bucket. Here are two supported tag structures for different web technologies: Cloud Front: web.aws.cloudfront.accessW3c.<region>.<instance> Elastic Load Balancing: web.aws.elb.access.<region>.<instance> The tag levels for region and instance identify the server's region and the server itself.

2. Save the file as config.ini. in the folder where the domain certificates and Python scripts are saved. Delete the original config.ini.example file.  

Prepare a ZIP file for upload

You should have a folder with only the following six files: your updated and renamed configuration file, two Python script files, and the three certificate files you downloaded from the Devo web application.

Create a ZIP file containing only these six files, and name it whatever you like.  

Create a new Lambda function

This procedure guides you through creating the new lambda function that will monitor the S3 bucket for changes. During this procedure we will upload the ZIP file we have already prepared.

1. Create a new AWS Lambda function in the same zone in which the S3 bucket resides.

2. Double-click to select the s3-get-object-python blueprint. The Configure triggers page opens.

   

3. Select the bucket that you will monitor for new log events and select Object Created (all) as the Event type. Select Enable trigger to start sending logs immediately once the Lambda function has been created. Click the Next button. The Configure function page opens.
   
   

4. Enter a Name and a Description for your new function. Select Python 2.7 for Runtime.  Select Upload a .ZIP file for Code entry type and upload the ZIP file you created earlier containing the six files.
   
   

5. On the same page, enter lt_s3_lambda.lambda_handler as the Handler. Select to Create a new role from template(s), enter a Role name, and choose S3 object read-only permissions as the Policy templates.
   
   

6. In the Advanced settings section, select the maximum value (1536 MB) for Memory. Set the Timeout to a value that is close but less than log file creation frequency. For example, if the log file creation frequency is 5 minutes, set the Timeout to 4 minutes and 30 seconds. Select No VPC for the VPC value. 
   
   
   
7. Finally, review the Lambda function configuration. If it is correct, click the Create function button.
   
   
   

The logs become visible in your Devo domain as new files are created in the S3 bucket.