Added support for Idempotent Producer #1152

mimaison · 2018-08-20T12:18:52Z

This PR adds support for the Idempotent Producer.

If the Idempotent configuration is enabled, upon startup the producer will retrieve a ProducerID and add it as well as sequence numbers to all produce requests.

This is still not complete and has not been properly tested yet. Quick tests showed that when retrying with idempotency, messages that have already been written to disk by brokers are correctly marked as duplicates.

This should be enough to start a discussion. Have a look and see if it's something you'd like merged. If so, I'll complete testing.

mimaison · 2018-09-03T16:30:53Z

@eapache Is this a feature you would be interested in? If I'm not mistaken, this would be the first 3rd party client to support it.

It still needs more work and I plan to complete it this week. So I'd like to know if for any reasons it's not something you'd want

buyology · 2018-09-04T14:21:33Z

@mimaison — great to see someone take this on! 🎉

I have done some work on this myself (and the txn producer (which I more or less got up and running)) — and I was blocked by #1070. Curious to hear your thoughts about that issue.

birdayz · 2018-09-04T16:48:42Z

i've also just started working on this, i was not aware that you already got the TX producer running.
i wonder if we can some up with a plan to get the EOS producer and consumer in, with everything that's required. i'm interested in helping.
@buyology mind to share your tx producer? i could't find a branch for this in your repo, just some other stuff like the Coordinator request refactor :)

mimaison · 2018-09-04T16:49:25Z

@buyology We've not solved that issue yet.

Our plan was to try requeuing failed PartitionSets directly into the output channel of BrokerProducer. Then assuming MaxOpenRequests is 1 (like when Idempotent was first released in Java in 0.11), the producer should recover. We've not tested that yet though.

Did you try something similar or some other methods?

edoardocomar · 2018-09-10T16:54:46Z

@buyology we've reimplemented the PR with retry logic applied to whole batches
could you please help review ?
cc @eapache

buyology

Made a first pass over the PR and left a few comments. The overall approach taken seems very reasonable to me! ☺️

buyology · 2018-09-12T07:39:40Z

client.go

@@ -67,6 +67,9 @@ type Client interface {
 	// in local cache. This function only works on Kafka 0.8.2 and higher.
 	RefreshCoordinator(consumerGroup string) error

+	// InitProducerID retrieves information required for Idempotent Producer
+	InitProducerID() (*InitProducerIDResponse, error)


Does it make sense to allow a Client to be reused to create multiple transactional producers? I'm thinking of the fact that the transactional producer would also use the InitProducerID-request but also specify the TransactionID and TransactionTimeout somewhere and I guess that would then have to be in the Config — and then I guess the answer would be — no

We think a future implementation of the transactional producer shall change the implementation of InitProducerID() to get the TXID from the config.
Reusing a client with a TXID set in the config for creating a new producer would result in the old producer being fenced off as a zombie. Similar to the Java client if you were to reuse the config.

buyology · 2018-09-12T07:44:20Z

config.go

+		}
+		if c.Net.MaxOpenRequests > 1 {
+			return ConfigurationError("Idempotent producer requires Net.MaxOpenRequests to be 1")
+		}


The Java client provides some fallbacks for these automatically, but I guess this is more in line with how sarama does configuration in general

yes we stuck to the sarama way of not rewriting the config under the covers

buyology · 2018-09-12T08:00:30Z

async_producer.go

+		msg.retries++
+	}
+	// extremely pessimistic strategy - refreshing metadata for every batch retried. Should be improved
+	err := p.client.RefreshMetadata(topic)


I guess would basically be the InvalidMetadataExceptions?

Sorry, can you expand on the comment? we're unsure what you are pointing out.

About this snippet we're not ecstatic about the very pessimistic strategy of refreshing md on each , we're going to push a small commit to make one refresh only

buyology · 2018-09-12T08:03:29Z

async_producer.go

 		// Other non-retriable errors
 		default:
-			bp.parent.returnErrors(msgs, block.Err)
+			bp.parent.returnErrors(pSet.msgs, block.Err)


The Java client compensates subsequent sequence numbers due to any non-retriable errors:

https://github.com/apache/kafka/blob/05ba5aa00847b18b74369a821e972bbba9f155eb/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java#L508

I guess we would have to do something similar.

Correct, currently upon hitting a non retriable error, we do not attempt to recover automatically from it. Users will have to create a new Producer to get a new ProducerId.

As you pointed out, the Java client tries to recover automatically (by fetching a new producerId and even resequencing message in some cases) but we feel like this can be done as a further improvement/PR.

buyology · 2018-09-12T08:20:00Z

@birdayz — sorry for the belated reply: I uploaded my tx-branch. It is hacky in places, but I think there are e.g. some tests (and perhaps some details I figured out by dissecting the Java implementation) that can be of help.

mimaison · 2018-09-30T20:24:42Z

@bai Can you take a look?

bai · 2018-10-01T03:22:22Z

Thanks @mimaison, looks solid. Could you please rebase your branch to get it tested against Go 1.11 too?

mimaison · 2018-10-01T12:45:10Z

Sure, I'll do that this week (hopefully today or tomorrow!)

Co-authored-by: Edoardo Comar <ecomar@uk.ibm.com> Co-authored-by: Mickael Maison <mickael.maison@gmail.com>

Unit test with mock broker Co-authored-by: Edoardo Comar <ecomar@uk.ibm.com> Co-authored-by: Mickael Maison <mickael.maison@gmail.com>

Co-authored-by: Edoardo Comar <ecomar@uk.ibm.com> Co-authored-by: Mickael Maison <mickael.maison@gmail.com>

mimaison · 2018-10-01T18:10:49Z

@bai We rebased the PR. One of the Travis jobs failed while installing dependencies but all the other passed. I don't seem to be able to retrigger a build so I can rerun it.

mimaison · 2018-10-03T07:37:18Z

Thank you!

bai · 2018-10-03T07:38:15Z

Many thanks for contributing this 👍

This reverts commit f21e149, reversing changes made to f1aa91b.

mimaison force-pushed the idempotent_producer branch from 0e0f112 to d6af3ca Compare August 30, 2018 14:47

mimaison force-pushed the idempotent_producer branch from d6af3ca to 1f91986 Compare September 10, 2018 16:53

buyology reviewed Sep 12, 2018

View reviewed changes

edoardocomar force-pushed the idempotent_producer branch from 50ec331 to 186fb18 Compare September 12, 2018 12:48

mimaison and others added 3 commits October 1, 2018 17:20

Added support for Idempotent Producer

8e2b04b

Co-authored-by: Edoardo Comar <ecomar@uk.ibm.com> Co-authored-by: Mickael Maison <mickael.maison@gmail.com>

Idempotent Producer - batch retry logic

4e5d477

Unit test with mock broker Co-authored-by: Edoardo Comar <ecomar@uk.ibm.com> Co-authored-by: Mickael Maison <mickael.maison@gmail.com>

single metadata refresh for all retries

eb38bbc

Co-authored-by: Edoardo Comar <ecomar@uk.ibm.com> Co-authored-by: Mickael Maison <mickael.maison@gmail.com>

edoardocomar force-pushed the idempotent_producer branch from 186fb18 to eb38bbc Compare October 1, 2018 17:19

bai merged commit f21e149 into IBM:master Oct 3, 2018

muirdm pushed a commit to retailnext/sarama that referenced this pull request Oct 8, 2018

Revert "Merge pull request IBM#1152 from mimaison/idempotent_producer"

b1ae913

This reverts commit f21e149, reversing changes made to f1aa91b.

muirdm pushed a commit to retailnext/sarama that referenced this pull request Nov 21, 2018

Revert "Merge pull request IBM#1152 from mimaison/idempotent_producer"

6c91dad

This reverts commit f21e149, reversing changes made to f1aa91b.

pmalekn mentioned this pull request Oct 31, 2019

Plans to support upcoming Kafka v0.11 ? #901

Closed

dnwe mentioned this pull request Jun 24, 2020

Revisit publishing of uncommited messages in functional_test #1733

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for Idempotent Producer #1152

Added support for Idempotent Producer #1152

mimaison commented Aug 20, 2018

mimaison commented Sep 3, 2018

buyology commented Sep 4, 2018 •

edited

Loading

birdayz commented Sep 4, 2018

mimaison commented Sep 4, 2018

edoardocomar commented Sep 10, 2018

buyology left a comment

buyology Sep 12, 2018 •

edited

Loading

edoardocomar Sep 12, 2018

buyology Sep 12, 2018

edoardocomar Sep 12, 2018

buyology Sep 12, 2018

edoardocomar Sep 12, 2018 •

edited

Loading

buyology Sep 12, 2018

edoardocomar Sep 12, 2018

buyology commented Sep 12, 2018

mimaison commented Sep 30, 2018

bai commented Oct 1, 2018

mimaison commented Oct 1, 2018

mimaison commented Oct 1, 2018

mimaison commented Oct 3, 2018

bai commented Oct 3, 2018

Added support for Idempotent Producer #1152

Added support for Idempotent Producer #1152

Conversation

mimaison commented Aug 20, 2018

mimaison commented Sep 3, 2018

buyology commented Sep 4, 2018 • edited Loading

birdayz commented Sep 4, 2018

mimaison commented Sep 4, 2018

edoardocomar commented Sep 10, 2018

buyology left a comment

Choose a reason for hiding this comment

buyology Sep 12, 2018 • edited Loading

Choose a reason for hiding this comment

edoardocomar Sep 12, 2018

Choose a reason for hiding this comment

buyology Sep 12, 2018

Choose a reason for hiding this comment

edoardocomar Sep 12, 2018

Choose a reason for hiding this comment

buyology Sep 12, 2018

Choose a reason for hiding this comment

edoardocomar Sep 12, 2018 • edited Loading

Choose a reason for hiding this comment

buyology Sep 12, 2018

Choose a reason for hiding this comment

edoardocomar Sep 12, 2018

Choose a reason for hiding this comment

buyology commented Sep 12, 2018

mimaison commented Sep 30, 2018

bai commented Oct 1, 2018

mimaison commented Oct 1, 2018

mimaison commented Oct 1, 2018

mimaison commented Oct 3, 2018

bai commented Oct 3, 2018

buyology commented Sep 4, 2018 •

edited

Loading

buyology Sep 12, 2018 •

edited

Loading

edoardocomar Sep 12, 2018 •

edited

Loading