SinkContext: ability to seek/pause/resume consumer for a topic #10498

dlg99 · 2021-05-07T01:07:54Z

Motivation

Allow sink to rewind a topic to given offset and pause/resume consumer for given topic
This is needed for #9927 https://github.com/apache/pulsar/pull/9927/files#r595722189

Modifications

SinkContext API:
New methods to seek/pause/resume
Matching implementations in ContextImpl

Added ExtendedSourceContext interface (not public) to provide access to the PulsarSource's consumers.
Updated ContextImpl and PulsarSource's implementations to provide this functionality.

Verifying this change

Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change added tests and can be verified as follows:

Added unit tests

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

The public API: yes

SinkContext added new methods, default implementation provided.

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? JavaDocs

...ons/instance/src/main/java/org/apache/pulsar/functions/source/MultiConsumerPulsarSource.java

jerrypeng · 2021-05-07T04:38:25Z

@dlg99 can you also please explain why this is needed for the Pulsar IO Sink and Kafka Sink compatibility layer?

dlg99 · 2021-05-07T04:59:29Z

@jerrypeng kafka-connect-adaptor implements some of the kafka interfaces. For the sink it is SinkTaskContext https://kafka.apache.org/23/javadoc/org/apache/kafka/connect/sink/SinkTaskContext.html - it has offset/pause/resume methods that need seek/pause/resume from the Sink context.

eolivelli

I left a few comments, overall the approach is good to me.

either this approach or the proposal from @jerrypeng works for me.

...ons/instance/src/main/java/org/apache/pulsar/functions/source/MultiConsumerPulsarSource.java

...ns/instance/src/main/java/org/apache/pulsar/functions/source/SingleConsumerPulsarSource.java

pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java

eolivelli

Lgtm

jerrypeng

I think there is a simpler approach to referencing consumers created in PulsarSource from Context. Please see my comment.

dlg99 · 2021-05-13T20:41:25Z

@jerrypeng @eolivelli I did the changes that @jerrypeng requested, also figured that MultiTopicsConsumerImpl (and, transitively, PatternMultiTopicsConsumerImpl) need special handling because getTopic()for such consumers does not return the topic.

pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java

eolivelli

LGTM

jerrypeng · 2021-05-14T16:39:44Z

pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java

+        if (inputConsumers == null) {
+            throw new PulsarClientException("Getting consumer is not supported");
+        }
+        for (int i = 0; i < 2; i++) {


Why is this for loop needed?

Two attempts to get the consumer:

Try to get the consumer.

If not found, reprocess MultiTopicsConsumers in case new consumers appeared (happens on repartition or a new topic that matches provided pattern if the pattern is used),

give it another try.

This doesn't seem like the right way to do this. Retrying twice seems arbitrary. I would rather let the user implement how many retries they want.

@jerrypeng this is not arbitrary.
I have no access to callbacks/events when the MultiTopicsConsumers adds another partition or a topic (for pattern-based subscription). MultiTopicsConsumers do have that internally.
So if the check of previously cached topic/partition -> consumer map finds nothing it means that it:

either doesn't exist

or the topic/partition (and consumer) got added after the initial map was built.
In that case it makes sense to check MultiTopicsConsumers once to update the map and try again.
I don't think blocking and polling is justified, at least I don't have such usecase.

@dlg99 if partitions / topics are added dynamically because of topic pattern match or increases in partition count, how would the function know when that happened and why would trying to the consumers twice help with that? There is an inherent race condition there. If can always argue there can be a scenario that topics/partitions got created after you called this method. If the goal of the method is to get new consumers for new topics / partitions then this API is not appropriate.

We talked offline about reasons it is needed.
I expressed the retry more clearly and added comments. Please take a look.

dlg99 · 2021-05-17T17:24:49Z

@jerrypeng do you have any additional feedback or is it good to go?

…added comments

…e#10498)

dlg99 marked this pull request as draft May 7, 2021 01:08

dlg99 mentioned this pull request May 7, 2021

Add ability to use Kafka's sinks as pulsar sinks #9927

Merged

1 task

jerrypeng reviewed May 7, 2021

View reviewed changes

...ons/instance/src/main/java/org/apache/pulsar/functions/source/MultiConsumerPulsarSource.java Outdated Show resolved Hide resolved

SinkContext: ability to seek/pause/resume consumer for a topic

70ab10d

dlg99 force-pushed the sink-rewind-topic branch from ad6fc9d to 70ab10d Compare May 10, 2021 22:03

dlg99 marked this pull request as ready for review May 10, 2021 23:24

eolivelli requested changes May 11, 2021

View reviewed changes

CR feedback

b04f4a5

dlg99 requested a review from eolivelli May 11, 2021 20:11

eolivelli approved these changes May 11, 2021

View reviewed changes

jerrypeng requested changes May 11, 2021

View reviewed changes

CR feedback, handling of multi-topic consumers

90faca4

dlg99 requested a review from jerrypeng May 13, 2021 22:14

eolivelli reviewed May 14, 2021

View reviewed changes

eolivelli approved these changes May 14, 2021

View reviewed changes

jerrypeng reviewed May 14, 2021

View reviewed changes

sijie assigned dlg99 May 15, 2021

sijie added this to the 2.8.0 milestone May 15, 2021

sijie added the area/connector label May 15, 2021

dlg99 requested a review from jerrypeng May 15, 2021 17:40

Expressed reload of consumers from MultiTopicsConsumer more clearly; …

87eed03

…added comments

jerrypeng approved these changes May 17, 2021

View reviewed changes

eolivelli merged commit 8183a1e into apache:master May 18, 2021

yangl pushed a commit to yangl/pulsar that referenced this pull request Jun 23, 2021

SinkContext: ability to seek/pause/resume consumer for a topic (apach…

1e40e2b

…e#10498)

dlg99 deleted the sink-rewind-topic branch October 14, 2021 23:30

bharanic-dev pushed a commit to bharanic-dev/pulsar that referenced this pull request Mar 18, 2022

SinkContext: ability to seek/pause/resume consumer for a topic (apach…

b5ca8c9

…e#10498)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SinkContext: ability to seek/pause/resume consumer for a topic #10498

SinkContext: ability to seek/pause/resume consumer for a topic #10498

dlg99 commented May 7, 2021

jerrypeng commented May 7, 2021

dlg99 commented May 7, 2021

eolivelli left a comment

eolivelli left a comment

jerrypeng left a comment

dlg99 commented May 13, 2021

eolivelli left a comment

jerrypeng May 14, 2021

dlg99 May 14, 2021

jerrypeng May 17, 2021

dlg99 May 17, 2021

jerrypeng May 17, 2021

dlg99 May 17, 2021

dlg99 commented May 17, 2021

SinkContext: ability to seek/pause/resume consumer for a topic #10498

SinkContext: ability to seek/pause/resume consumer for a topic #10498

Conversation

dlg99 commented May 7, 2021

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

jerrypeng commented May 7, 2021

dlg99 commented May 7, 2021

eolivelli left a comment

Choose a reason for hiding this comment

eolivelli left a comment

Choose a reason for hiding this comment

jerrypeng left a comment

Choose a reason for hiding this comment

dlg99 commented May 13, 2021

eolivelli left a comment

Choose a reason for hiding this comment

jerrypeng May 14, 2021

Choose a reason for hiding this comment

dlg99 May 14, 2021

Choose a reason for hiding this comment

jerrypeng May 17, 2021

Choose a reason for hiding this comment

dlg99 May 17, 2021

Choose a reason for hiding this comment

jerrypeng May 17, 2021

Choose a reason for hiding this comment

dlg99 May 17, 2021

Choose a reason for hiding this comment

dlg99 commented May 17, 2021