Forward response to Amazon S3

Request

Using the destination object in a query request you can have query responses forwarded to your organization's data storage services, including Amazon S3. 

To send your response to S3, include the destination object in your request and set type to s3 like this:

"destination": {
 "type":"s3"
}

Parameters

With the destination.type set to s3, you need to specify some additional parameters specific to Amazon Web Services to authenticate the connection and save the data contained in the response:

You can still send a response to S3 successfully without specifying the aws. parameters, but the data contained in the response will only be saved temporally in a generic account.

Query using QueryID and forwarding to S3

In the example below, we are using our own AWS account to deliver the S3 files.

{
    "query": "from demo.ecommerce.data select *",
    "from": 1546083949,
    "to": 1546947949,
    "destination": {
        "type": "s3",
        "params": {
            "aws.bucket": "my-bucket",
            "aws.region": "eu-west-1",
            "aws.accesskey": "abcdefghijkl",
            "aws.secretkey": "Tg4T0aGKvd/aaaaaaaWWQv3Vs0kS15tpn3kbd0V2UZ",
			"notification.email":"john.smith@company.com",
            "format": "zip"
        }
    }
}

Response

When the query request is received and the results are successfully being forwarded to S3, a response that includes the job ID is returned to the query sender. This job ID is needed to manage the task. For more information, see the Job requests article.

{
    "msg": "",
    "status": 0,
    "object": {
        "id": "193507c8-93df-405f-bdf7-acf1a91890e2"
    }
}

Files are uploaded in this format:

Examples:

If you want to see the paths of the uploaded files you can query the task status to see the current upload files (not finished tasks) or see all the files paths (finished task).

{
    "status": 0,
    "cid": "tjIqEhbIKZ",
    "timestamp": 1533548393561,
    "object": {
        "status": "COMPLETED",
        "id": "983c5a6b-7081-40eb-a631-7492d9badbfe",
        "eventGenerated": 2619588,
        "eventsSent": 2619588,
        "lastDatetime": 0,
        "extra": {
            "paths": [
                "https://sis-2523.s3.eu-west-1.amazonaws.com/apiv2_983c5a6b-7081-40eb-a631-7492d9badbfe_001.csv.zip",
                "https://sis-2523.s3.eu-west-1.amazonaws.com/apiv2_983c5a6b-7081-40eb-a631-7492d9badbfe_002.csv.zip",
                "https://sis-2523.s3.eu-west-1.amazonaws.com/apiv2_983c5a6b-7081-40eb-a631-7492d9badbfe_003.csv.zip",
                "https://sis-2523.s3.eu-west-1.amazonaws.com/apiv2_983c5a6b-7081-40eb-a631-7492d9badbfe_004.csv.zip"
            ]
        },
        "type": "s3",
        "friendlyName": "s3_ezilvjQ1Sh",
        "query": "LinQ_Query",
        "cid": "lSypa1lvKv",
        "owner": "kW5Ule1m*omUatc8tu4S5LuQSnX9UpLs",
        "table": "demo.ecommerce.data",
        "error": []
    }
}