Forwarding query responses to Kafka

Before sending data to Kafka, contact Devo customer support to validate the destination parameters to use in the query request.

Using the destination object in a query request, you can have query responses forwarded to your organization's data storage services, including Apache Kafka systems. 

To forward a query's response to Kafka, include the destination object in the query request and set type to kafka like this:

"destination": {
 "type":"kafka"
}

Parameters

With the destination.type set to kafka, you need to specify some additional parameters specific to Kafka systems to identify where and how to save the data contained in the response:

Parameter Description
topic

Name of the Kafka topic to which you want to save the data contained in the query response.

bootstrap.servers The host/port pair used to establish the initial connection to the Kafka cluster. The format is hostname:port.

All the configuration parameters that Kafka Java Library accepts to configure a producer are allowed. All values, even those Kafka documentation defines as Integer, Long or Boolean, must be sent as String.

Query using QueryID and forwarding to Kafka queue

{
  "queryId": "Query_Id_from_web_app",
  "from": 1481471880,
  "to": 1481558280,
  "destination": [
            {"type":"kafka",
             "params":[
                    "topic":"my_topic",
                    "bootstrap.servers":"kafka.queues.location:port"
                    ...........
                }
     
  ],
 "mode": {
    "type":"json"
    }
}

Query using LINQ and forwarding to Kafka queue

{
  "query": "LinQ_Query",
  "from": 1481471880,
  "to": 1481558280,
  "destination": [
            {"type":"kafka",
             "params":{
                    "topic":"my_topic",
                    "bootstrap.servers":"kafka.queues.location:port"
                    ...........
                }
     
  ],
 "mode": {
    "type":"json"
    }
}

For more information about Kafka, visit the official Kafka website.

Have we answered your question?

If not, please contact our technical support team via email by clicking the button below.

CONTACT US