Forwarding query responses to S3

Before sending data to S3, contact Devo customer support to validate the destination parameter values to use in your request.

Request

Using the destination object in a query request you can have query responses forwarded to your organization's data storage services, including the Amazon S3 service. 

To send your response to S3, include the destination object in your request and set type to s3 like this:

"destination": {
 "type":"s3"
}

Parameters

With the destination.type set to s3, you need to specify some additional parameters specific to Amazon Web Services to authenticate the connection and save the data contained in the response:

Parameter Description
aws.bucket Bucket name that should exist in your organization's AWS account.
aws.region AWS region. See the available region codes here.
aws.accesskey This is your Access Key ID for your organization's AWS account.
aws.secretkey This is your Secret Access Key for your organization's AWS account.
format The format used to store the information in the file uploaded to S3. It may be either zip or gzip. If not specified, the default format will be zip.
file.password Add a password for the compressed file. Only valid for zip format.

You can still send a response to S3 successfully without specifying the aws. parameters, however the data contained in the response will only be saved temporally to a generic account.

Query using QueryID and forwarding to S3

In the example below, we are using our own AWS account to deliver the S3 files.

{
  "queryId": "Query_Id_from_web_app",
  "from": 1481471880,
  "to": 1481558280,
  "destination": [
            {"type":"s3",
             "params":[
                    "format":"csv",
                    "aws.accesskey":"aaaaaaaaaabbbbbbbcccccc",
                    "aws.secretkey":"cccccccccbbbbbbbaaaaaaa",
                    "aws.bucket":"bucket_name"
                }
     
  ],
 "mode": {
    "type":"json"
    }
}

Response

When the query request is received and the results are successfully being forwarded to S3, a response that includes the job ID is returned to the query sender. This job ID is needed to manage the task. For more information, see the Job requests article.

{
    "msg": "",
    "status": 0,
    "object": {
        "id": "193507c8-93df-405f-bdf7-acf1a91890e2"
    }
}

The query results will be upload to AWS bucket in one or many files, depending for the size of the results. The maximum file size can be configured using the aws.file.size parameter. By default, the file size is 500 Mbs.

Files are uploaded in this format:

apiv2_<task id>_<index>.csv.(zip|gz)

Examples:

apiv2_983c5a6b-7081-40eb-a631-7492d9badbfe_001.csv.zip
apiv2_983c5a6b-7081-40eb-a631-7492d9badbfe_002.csv.zip
apiv2_983c5a6b-7081-40eb-a631-7492d9badbfe_003.csv.zip
apiv2_983c5a6b-7081-40eb-a631-7492d9badbfe_004.csv.zip

apiv2_4c92c89e-19a3-4e49-bb8a-4dbf30d47214_001.csv.gz
apiv2_4c92c89e-19a3-4e49-bb8a-4dbf30d47214_002.csv.gz

If you want to see the paths of the uploaded files you can query the task status to see the current upload files (not finished tasks) or see all the files paths (finished task).

{
    "status": 0,
    "cid": "tjIqEhbIKZ",
    "timestamp": 1533548393561,
    "object": {
        "status": "COMPLETED",
        "id": "983c5a6b-7081-40eb-a631-7492d9badbfe",
        "eventGenerated": 2619588,
        "eventsSent": 2619588,
        "lastDatetime": 0,
        "extra": {
            "paths": [
                "https://sis-2523.s3.eu-west-1.amazonaws.com/apiv2_983c5a6b-7081-40eb-a631-7492d9badbfe_001.csv.zip",
                "https://sis-2523.s3.eu-west-1.amazonaws.com/apiv2_983c5a6b-7081-40eb-a631-7492d9badbfe_002.csv.zip",
                "https://sis-2523.s3.eu-west-1.amazonaws.com/apiv2_983c5a6b-7081-40eb-a631-7492d9badbfe_003.csv.zip",
                "https://sis-2523.s3.eu-west-1.amazonaws.com/apiv2_983c5a6b-7081-40eb-a631-7492d9badbfe_004.csv.zip"
            ]
        },
        "type": "s3",
        "friendlyName": "s3_ezilvjQ1Sh",
        "query": "LinQ_Query",
        "cid": "lSypa1lvKv",
        "owner": "kW5Ule1m*omUatc8tu4S5LuQSnX9UpLs",
        "table": "siem.logtrust.web.activity",
        "error": []
    }
}

For more information about Amazon S3, visit the Amazon S3 website.

Have we answered your question?

If not, please contact our technical support team via email by clicking the button below.

CONTACT US