Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KAFKA output: codec, acks, and retry configuration #945

Closed
wants to merge 3 commits into from

Conversation

framiere
Copy link
Contributor

The following configuration is now possible

  ## CompressionCodec represents the various compression codecs recognized by Kafka in messages.
  ##  0 : No compression
  ##  1 : Gzip compression
  ##  2 : Snappy compression
  compression_codec = 0

  ##  RequiredAcks is used in Produce Requests to tell the broker how many replica acknowledgements it must see before responding
  ##  0 : the producer never waits for an acknowledgement from the broker. This option provides the lowest latency but the weakest durability guarantees (some data will be lost when a server fails).
  ##  1 : the producer gets an acknowledgement after the leader replica has received the data. This option provides better durability as the client waits until the server acknowledges the request as successful (only messages that were written to the now-dead leader but not yet replicated will be lost).
  ##  -1 : the producer gets an acknowledgement after all in-sync replicas have received the data. This option provides the best durability, we guarantee that no messages will be lost as long as at least one in sync replica remains.
  required_acks = -1

  ##  The total number of times to retry sending a message
  max_retry = 3

The following configuration is now possible

	## CompressionCodec represents the various compression codecs
recognized by Kafka in messages.
	##  "none" : No compression
	##  "gzip" : Gzip compression
	##  "snappy" : Snappy compression
	# compression_codec = "none"

	##  RequiredAcks is used in Produce Requests to tell the broker how
many replica acknowledgements it must see before responding
	##  "none" : the producer never waits for an acknowledgement from the
broker. This option provides the lowest latency but the weakest
durability guarantees (some data will be lost when a server fails).
	##  "leader" : the producer gets an acknowledgement after the leader
replica has received the data. This option provides better durability
as the client waits until the server acknowledges the request as
successful (only messages that were written to the now-dead leader but
not yet replicated will be lost).
	##  "leader_and_replicas" : the producer gets an acknowledgement after
all in-sync replicas have received the data. This option provides the
best durability, we guarantee that no messages will be lost as long as
at least one in sync replica remains.
	# required_acks = "leader_and_replicas"

	##  The total number of times to retry sending a message
	# max_retry = "3"
@@ -53,6 +61,21 @@ var sampleConfig = `
## ie, if this tag exists, it's value will be used as the routing key
routing_tag = "host"

## CompressionCodec represents the various compression codecs recognized by Kafka in messages.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these should be indented by two spaces, not a tab

@sparrc
Copy link
Contributor

sparrc commented Mar 31, 2016

thanks for the contribution @framiere, just a few comments to address and then I can merge!

@sparrc
Copy link
Contributor

sparrc commented Mar 31, 2016

also please update the changelog with your change (under features)

@framiere framiere changed the title Add compression/acks/retry conf to Kafka output plugin KAFKA output: codec, acks, and retry configuration Mar 31, 2016
@framiere
Copy link
Contributor Author

done

@sparrc sparrc closed this in 51f4e9c Apr 1, 2016
@sparrc
Copy link
Contributor

sparrc commented Apr 1, 2016

great, thanks @framiere

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants