Forwarding query responses to HDFS

Before sending data to HDFS, contact Devo customer support to validate the destination.

Using the destination object in a query request, you can have query responses forwarded to your organization's data storage services, including Hadoop Distributed File System (HDFS)-type systems. 

To send your response to HDFS, include the destination object in your request and set type to hdfs like this:

"destination": {
 "type":"hdfs"
}

Parameters

With the destination.type set to hdfs, you need to specify some additional parameters specific to HDFS to identify where and how to save the data contained in the response:

Parameter Description
fs.defaultFS Specifies where the HDFS server is listening (for example, hdfs://172.16.1.17:8020).
full path Specifies the path where the data will be written (for example, /tmp/devo/test_05).
file Specifies the file to which the data will be saved. Data files have the following format: {file_name}_{node_name}.data

dfs.replication

Defines the default block replication factor. The actual number of replications can be specified when the file is created.

Query using QueryID and forwarding to HDFS

{
  "queryId": "Query_Id_from_web_app",
  "from": 1481471880,
  "to": 1481558280,
  "destination": [
            {"type":"hdfs",
             "params":[
                    "fs.defaultFS":"hdfs://server:port",
                    "dfs.replication":"1",
                    "file":"/tmp/devo/test_05"
                    ...........
                }
     
  ],
 "mode": {
    "type":"json"
    }
}

Query using LINQ and forwarding to HDFS

{
  "query": "LinQ_Query",
  "from": 1481471880,
  "to": 1481558280,
  "destination": [
            {"type":"hdfs",
             "params":{
                    "fs.defaultFS":"hdfs://server:port",
                    "dfs.replication":"1",
                    "file":"/tmp/devo/test_05"
                    ...........
                }
     
  ],
 "mode": {
    "type":"json"
    }
}
Learn more about HDFS by reviewing the Hadoop HDFS Architecture Guide .

Have we answered your question?

If not, please contact our technical support team via email by clicking the button below.

CONTACT US