STORM-2542: Remove KafkaConsumer.subscribe API option, make KafkaConsumer.assign the default #2151
Conversation
|
If this goes in I'll follow up with a PR against 1.x to deprecate the classes using |
|
@srdo was this agreed upon ? |
|
@hmcl No. I got no response on the mailing list, so now I'm trying here. I figure if anyone objects they'll be able to do so on this PR. |
|
@srdo I am +1 on this PR. Let's just clean the commits that that make it such that we this PR consist of three commits only. The first commit should be STORM-2548 PR, the 2nd commit STORM-2541 PR, plus its own commit moving classes into the appropriate packages. |
|
@srdo Thanks for your diligence and awesome work refactoring this code. It just made it much better. |
srdo
force-pushed
the
STORM-2542
branch
2 times, most recently
from
548e938 to
f473b5d
Compare
docs/storm-kafka-client.md
Outdated
| This can cause less churn in the assignments when spouts go down and come back up, but it can result in a lot of issues if not done right. This can all be handled by subclassing | ||
| Subscription and we have a few implementations that you can look at for examples on how to do this. ManualPartitionNamedSubscription and ManualPartitionPatternSubscription. Again | ||
| please be careful when using these or implementing your own. | ||
| By default the KafkaSpout instancs will be assigned partitions by round robin assignment. If you need to customize this assignment, you can implement the `ManualPartitioner` interface. The implementation can be passed to the `ManualPartitionSubscription` constructor, and the `Subscription` can then be set in the `KafkaSpoutConfig` via the `KafkaSpoutConfig.Builder` constructor. Please take care when supplying a custom implementation, since an incorrect `ManualPartitioner` implementation could leave some partitions unread, or concurrently read by multiple spout instances. See the `RoundRobinManualPartitioner` for an example of how to implement this functionality. |
There was a problem hiding this comment.
By default the KafkaSpout instances will be assigned partitions using a round robin strategy. If you need to customize partitions assignment, you must implement the ManualPartitioner interface.
|
+1. Thanks @srdo this looks great. |
srdo
force-pushed
the
STORM-2542
branch
7 times, most recently
from
269e24d to
3f44dca
Compare
|
Thanks for the reviews. |
|
@srdo are we not planning on pushing this into 1.x-branch? |
4 participants
This builds on #2150, which is the first commit in this. The third commit is purely class moves because the kafka.spout package was getting a bit unwieldy.
This should only go on 2.x if it goes in at all.
Please see https://issues.apache.org/jira/browse/STORM-2542 for the justification for why I believe we should make this change.