Kafka Sender

A list of host/port pairs to use for establishing the initial connection to the Kafka cluster.

The client will make use of all servers, irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should use the following format:
host1:port1,host2:port2,...

Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).

The number of acknowledgments the producer requires the leader to have received before considering a request as complete.

This controls the durability of records that are sent. The following settings are allowed:

0 - If set to zero, the producer will not wait for any acknowledgment from the server. The record will be immediately added to the socket buffer and considered sent. No guarantee can be made that the server has received the record in this case, and the retries configuration will not take effect (as the client won't generally know of any failures). The offset given back for each record will always be set to -1.

1 - The leader will write the record to its local log but will respond without awaiting full acknowledgment from all followers. In this case, should the leader fail immediately after acknowledging the record but before the followers have replicated it, the record will be lost.

all - The leader will wait for the full set of in-sync replicas to acknowledge the record. This guarantees that the record will not be lost as long as at least one in-sync replica remains alive. This is the strongest available guarantee. This is equivalent to the acks=-1 setting.

Amount of time to wait (in seconds) before sending requests in the absence of a full batch size. Leave at 0 (default) for no delay.

The producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they can be sent out. However, in some circumstances the client may want to reduce the number of requests even under moderate load. This setting acomplishes this by adding a small amount of artificial delay—that is, rather than immediately sending out a record, the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together.

This can be thought of as analogous to Nagle's algorithm in TCP. 

For example, entering a linger of 5ms would have the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of a load.

Once the batch size of records for a partition is fulfilled, it will be sent immediately regardless of this setting. However, if there are fewer than this many bytes accumulated for this partition, the unit will 'linger' for the specified time waiting for more records to come in. 

Enter the default batch size in bytes to avoid batching records larger than this size. Enter 0 to disable batching entirely.

The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server.
Requests sent to brokers will contain multiple batches, one for each partition with data available to be sent.

A small batch size will make batching less common and may reduce throughput.

A very large batch size will use more memory to allocate a buffer of the specified batch size in anticipation of additional records.

Enter the total bytes of memory to use to buffer records waiting to be sent to the server.

If records are sent faster than they can be delivered to the server, the producer will block for maxBlocks and will throw an exception.

This setting should correspond roughly to the total memory, but is not a hardbound since not all memory is used for buffering. Some additional memory will be used for compression (if compression is enabled) as well as for maintaining in-flight requests.

Enter the compression type for all data generated by the producer.

The default is none (i.e. no compression). Valid values are none, gzip, snappy, or lz4.

Compression is of full batches of data, so the efficacy of batching will also impact the compression ratio (more batching means better compression).

The number of retries on error.

Setting a value greater than zero will cause the client to resend any failed record with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error.

The maximum amount of time the client will wait for the response of a request (in seconds).

If the response is not received before the timeout elapses, the client will resend the request if necessary or fail the request if retries are exhausted.

Fields

There are static (unit level) and dynamic (event level) settings.
Everything Kafka needs to create a producer must be a static setting.

Everything that can be defined by a Kafka record can be a dynamic setting.
topic & partition can be both static and dynamic.

timestamp & key can be dynamic, or not sent.

value must always be dynamic.

Events to be sent as Kafka records.

Outputs events that requested metrics, repeated once for every obtained metric, and enriched with fields for metric name and value.