SingleProcessorTopology implements Processor#3
Merged
Guozhang Wang (guozhangwang) merged 4 commits intoconfluentinc:streamingfrom Aug 6, 2015
Merged
SingleProcessorTopology implements Processor#3Guozhang Wang (guozhangwang) merged 4 commits intoconfluentinc:streamingfrom
Guozhang Wang (guozhangwang) merged 4 commits intoconfluentinc:streamingfrom
Conversation
There was a problem hiding this comment.
Guozhang Wang (@guozhangwang)
We need to create a new instance of Processor here, don't we?
There was a problem hiding this comment.
ymatsuda
I think we can just use the SingleProcessorTopology instance here, why do we need to create another one?
There was a problem hiding this comment.
topology() is called for each partition group to instantiate the chain of instances of KStream, Processor, StateStore, etc. In general, you shouldn't use the same Processor instance for different partition groups.
BTW, I don't know if we still want to do a special handing of a single processor use case. I am afraid that It confuses user.
There was a problem hiding this comment.
You are right, I will go ahead and change accordingly.
Guozhang Wang (guozhangwang)
added a commit
that referenced
this pull request
Aug 6, 2015
Remove SingleProcessorTopology.
Guozhang Wang (guozhangwang)
added a commit
that referenced
this pull request
Mar 3, 2016
…tech-preview Backport recent changes from trunk/streams: batch #3
José Armando García Sancio (jsancio)
pushed a commit
that referenced
this pull request
Sep 13, 2019
… State (#3) Initial infrastructure to support broker storage tiering. This PR implements several components to support tiered storage. Tier Topic & Tier Topic Manager Implements a mechanism for storing tier metadata store. Implemented via an internal "Tier Topic", and a TierTopicManager that manages writing and materializing this Tier Topic into Tier Partition State. CPKAFKA-1727: Implement Tier Topic CPKAFKA-1760: Tier topic metrics CPKAFKA-1761: TierTopicManager: implement failure handling. CPKAFKA-1731: TierTopicManager: topic creation and configuration CPKAFKA-1770: TierTopicManager: Serialization / Deserialization mechanism. CPKAFKA-1734: TierTopicManager: enable custom hash partitioning of tier topic. CPKAFKA-1855: TierTopicManager: implement catch up consumer. Tier Partition State Implements a file based log format that contains tier metadata for a given tiered partition for efficient access. The Tier Partition State is materialized from the Tier Topic by the TierTopicManager. CPKAFKA-1759: Implement Tier Partition State Tier Archiver Implements a first pass at an archiver component, that detects when data for a topic partition is ready to be tiered, and uploads that data to an object store. Upon successfully uploading tiered data, writes to the Tier Topic in order to commit tier metadata durably. CPKAFKA-1728: Archiver CPKAFKA-1735: Archiver: thread pool CPKAFKA-1738: Archiver: basic next segment to archive logic. CPKAFKA-1744: Archiver: retry logic. Broker Config and Startup: CPKAFKA-1816: Tier Enable TopicConfig. CPKAFKA-1901: Tiered storage feature flag. CPKAFKA-1732: Enable startup with the rest of KafkaServer Co-authored-by: Lucas Bradstreet <lucas@confluent.io> Co-authored-by: Gardner Vickers <gardner@confluent.io> Co-authored-by: Colin Hicks <colin.hicks@confluent.io>
Confluent Jenkins Bot (ConfluentJenkins)
pushed a commit
that referenced
this pull request
Oct 17, 2019
…pache#7305) A partition log in initialized in following steps: 1. Fetch log config from ZK 2. Call LogManager.getOrCreateLog which creates the Log object, then 3. Registers the Log object Step #3 enables Configuration update thread to deliver configuration updates to the log. But if any update arrives between step #1 and #3 then that update is missed. It breaks following use case: 1. Create a topic with default configuration, and immediately after that 2. Update the configuration of topic There is a race condition here and in random cases update made in second step will get dropped. This change fixes it by tracking updates arriving between step #1 and #3 Once a Partition is done initializing log, it checks if it has missed any update. If yes, then the configuration is read from ZK again. Added unit tests to make sure a dirty configuration is refreshed. Tested on local cluster to make sure that topic configuration and updates are handled correctly. Reviewers: Jason Gustafson <jason@confluent.io>
David Arthur (mumrah)
pushed a commit
that referenced
this pull request
Oct 29, 2019
…pache#7305) A partition log in initialized in following steps: 1. Fetch log config from ZK 2. Call LogManager.getOrCreateLog which creates the Log object, then 3. Registers the Log object Step #3 enables Configuration update thread to deliver configuration updates to the log. But if any update arrives between step #1 and #3 then that update is missed. It breaks following use case: 1. Create a topic with default configuration, and immediately after that 2. Update the configuration of topic There is a race condition here and in random cases update made in second step will get dropped. This change fixes it by tracking updates arriving between step #1 and #3 Once a Partition is done initializing log, it checks if it has missed any update. If yes, then the configuration is read from ZK again. Added unit tests to make sure a dirty configuration is refreshed. Tested on local cluster to make sure that topic configuration and updates are handled correctly. Reviewers: Jason Gustafson <jason@confluent.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ymatsuda
Let SingleProcessorTopology implements Processor so that uers do not need to pass-in the Processor class.