KAFKA-14493: Introduce Zk to KRaft migration state machine STUBs in KRaft controller. by akhileshchg · Pull Request #12998 · apache/kafka

akhileshchg · 2022-12-15T02:42:08Z

KAFKA-14493: Introduce Zk to KRaft migration state machine STUBs in KRaft controller.

This patch introduces a preliminary state machine that can be used by the KRaft controller to drive online migration from Zk to KRaft.

MigrationState -- Defines the states we can have during migration from Zk to KRaft.

KRaftMigrationDriver -- Defines the state transitions and events to handle actions like controller change, metadata change, and broker change and has interfaces through which it claims Zk controllership, performs zk writes, and sends RPCs to ZkBrokers.

MigrationClient -- Interface that defines the functions used to claim and relinquish Zk controllership, read to and write from Zk.

BrokersRpcClient -- Interface that defines the functions used to send RPCs to Zk brokers.

…Raft controller. This patch introduces a preliminary state machine that can be used by KRaft controller to drive online migration from Zk to KRaft. MigrationState -- Defines the states we can have while migration from Zk to KRaft. KRaftMigrationDriver -- Defines the state transitions, and events to handle actions like controller change, metadata change, broker change and have interfaces through which it claims Zk controllership, performs zk writes and sends RPCs to ZkBrokers. MigrationClient -- Interface that defines the functions used to claim and relinquish Zk controllership, read to and write from Zk. BrokersRpcClient -- Interface that defines the functions used to send RPCs to Zk brokers.

…ot generator. This PR introduces the new metadata loader and snapshot generator. For the time being, they are only used by the controller, but a PR for the broker will come soon. The new metadata loader supports adding and removing publishers dynamically. (In contrast, the old loader only supported adding a single publisher.) It also passes along more information about each new image that is published. This information can be found in the LogDeltaManifest and SnapshotManifest classes. The new snapshot generator replaces the previous logic for generating snapshots in QuorumController.java and associated classes. The new generator is intended to be shared between the broker and the controller, so it is decoupled from both. There are a few small changes to the old snapshot generator in this PR. Specifically, we move the batch processing time and batch size metrics out of BrokerMetadataListener.scala and into BrokerServerMetrics.scala.

Conflicts: core/src/main/scala/kafka/server/ControllerServer.scala core/src/main/scala/kafka/server/SharedServer.scala metadata/src/main/java/org/apache/kafka/image/loader/MetadataLoader.java metadata/src/main/java/org/apache/kafka/image/publisher/MetadataPublisher.java

akhileshchg · 2022-12-16T01:48:57Z

metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationDriver.java

+        Set<Integer> zkRegisteredZkBrokers = zkMigrationClient.readBrokerIds();
+        zkRegisteredZkBrokers.removeAll(kraftRegisteredZkBrokers);
+        if (zkRegisteredZkBrokers.isEmpty()) {
+            return true;
+        } else {


I forgot about the approach we decided to take. Remind me again, are we not looking at topic assignments at this point? I thought that is more decisive regarding the zkBrokers which need to register with the kraft controller.

I'm tenatively adding the change. We can remove it if it is not required.

…gration_driver_stub

Still an issue with broker epochs in UMR from KRaft

Conflicts: clients/src/main/resources/common/message/BrokerRegistrationRequest.json core/src/main/scala/kafka/server/BrokerLifecycleManager.scala core/src/main/scala/kafka/server/BrokerToControllerChannelManager.scala core/src/main/scala/kafka/server/KafkaServer.scala metadata/src/main/java/org/apache/kafka/metadata/BrokerRegistration.java metadata/src/main/resources/common/metadata/RegisterBrokerRecord.json

Conflicts: core/src/main/scala/kafka/cluster/Partition.scala core/src/main/scala/kafka/common/RecordValidationException.scala core/src/main/scala/kafka/log/LocalLog.scala core/src/main/scala/kafka/log/LogLoader.scala core/src/main/scala/kafka/log/LogSegment.scala core/src/main/scala/kafka/log/ProducerStateManager.scala core/src/main/scala/kafka/log/UnifiedLog.scala core/src/main/scala/kafka/server/ReplicaManager.scala core/src/main/scala/kafka/tools/DumpLogSegments.scala core/src/main/scala/kafka/utils/CoreUtils.scala core/src/test/scala/unit/kafka/coordinator/group/GroupMetadataManagerTest.scala core/src/test/scala/unit/kafka/coordinator/transaction/TransactionStateManagerTest.scala core/src/test/scala/unit/kafka/log/LocalLogTest.scala core/src/test/scala/unit/kafka/log/LogLoaderTest.scala core/src/test/scala/unit/kafka/log/ProducerStateManagerTest.scala core/src/test/scala/unit/kafka/log/UnifiedLogTest.scala core/src/test/scala/unit/kafka/server/IsrExpirationTest.scala core/src/test/scala/unit/kafka/server/ReplicaManagerQuotasTest.scala core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala core/src/test/scala/unit/kafka/utils/CoreUtilsTest.scala jmh-benchmarks/src/main/java/org/apache/kafka/jmh/partition/UpdateFollowerFetchStateBenchmark.java metadata/src/main/java/org/apache/kafka/metadata/migration/ZkRecordConsumer.java storage/src/main/java/org/apache/kafka/server/log/internals/CorruptSnapshotException.java

…gration_driver_stub

clients/src/main/resources/common/message/BrokerRegistrationRequest.json

core/src/main/scala/kafka/server/KafkaServer.scala

metadata/src/main/java/org/apache/kafka/metadata/migration/BrokersRpcClient.java

metadata/src/main/java/org/apache/kafka/controller/MigrationControlManager.java

…gration_driver_stub Conflicts: core/src/main/scala/kafka/migration/KRaftControllerToZkBrokersRpcClient.scala

cmccabe

LGTM pending jenkins

…gration_driver_stub

…Raft controller. (#12998) This patch introduces a preliminary state machine that can be used by KRaft controller to drive online migration from Zk to KRaft. MigrationState -- Defines the states we can have while migration from Zk to KRaft. KRaftMigrationDriver -- Defines the state transitions, and events to handle actions like controller change, metadata change, broker change and have interfaces through which it claims Zk controllership, performs zk writes and sends RPCs to ZkBrokers. MigrationClient -- Interface that defines the functions used to claim and relinquish Zk controllership, read to and write from Zk. Co-authored-by: David Arthur <mumrah@gmail.com> Reviewers: Colin P. McCabe <cmccabe@apache.org>

rishiraj88 · 2023-01-09T21:50:21Z

Thanks

With the new broker epoch validation logic introduced in #12998, we no longer need the ZK broker epoch to be sent to the KRaft controller. This patch removes that epoch and replaces it with a boolean. Another small fix is included in this patch for controlled shutdown in migration mode. Previously, if a ZK broker was in migration mode, it would always try to do controlled shutdown via BrokerLifecycleManager. Since there is no ordering dependency between bringing up ZK brokers and the KRaft quorum during migration, a ZK broker could be running in migration mode, but talking to a ZK controller. A small check was added to see if the current controller is ZK or KRaft before decided which controlled shutdown to attempt. Reviewers: Colin P. McCabe <cmccabe@apache.org>

* apache-github/trunk: KAFKA-14601: Improve exception handling in KafkaEventQueue apache#13089 KAFKA-14367; Add `OffsetCommit` to the new `GroupCoordinator` interface (apache#12886) KAFKA-14530: Check state updater more often (apache#13017) KAFKA-14304 Use boolean for ZK migrating brokers in RPC/record (apache#13103) KAFKA-14003 Kafka Streams JUnit4 to JUnit5 migration part 2 (apache#12301) KAFKA-14607: Move Scheduler/KafkaScheduler to server-common (apache#13092) KAFKA-14367; Add `OffsetFetch` to the new `GroupCoordinator` interface (apache#12870) KAFKA-14557; Lock metadata log dir (apache#13058) MINOR: Implement toString method for TopicAssignment and PartitionAssignment (apache#13101) KAFKA-12558: Do not prematurely mutate internal partition state in Mirror Maker 2 (apache#11818) KAFKA-14540: Fix DataOutputStreamWritable#writeByteBuffer (apache#13032) KAFKA-14600: Reduce flakiness in ProducerIdExpirationTest (apache#13087) KAFKA-14279: Add 3.3.x streams system tests (apache#13077) MINOR: bump streams quickstart pom versions and add to list in gradle.properties (apache#13064) MINOR: Update KRaft cluster upgrade documentation for 3.4 (apache#13063) KAFKA-14493: Introduce Zk to KRaft migration state machine STUBs in KRaft controller. (apache#12998) KAFKA-14570: Fix parenthesis in verifyFullFetchResponsePartitions output (apache#13072) MINOR: Remove public mutable fields from ProducerAppendInfo (apache#13091)

…Raft controller. (apache#12998) This patch introduces a preliminary state machine that can be used by KRaft controller to drive online migration from Zk to KRaft. MigrationState -- Defines the states we can have while migration from Zk to KRaft. KRaftMigrationDriver -- Defines the state transitions, and events to handle actions like controller change, metadata change, broker change and have interfaces through which it claims Zk controllership, performs zk writes and sends RPCs to ZkBrokers. MigrationClient -- Interface that defines the functions used to claim and relinquish Zk controllership, read to and write from Zk. Co-authored-by: David Arthur <mumrah@gmail.com> Reviewers: Colin P. McCabe <cmccabe@apache.org>

…e#13103) With the new broker epoch validation logic introduced in apache#12998, we no longer need the ZK broker epoch to be sent to the KRaft controller. This patch removes that epoch and replaces it with a boolean. Another small fix is included in this patch for controlled shutdown in migration mode. Previously, if a ZK broker was in migration mode, it would always try to do controlled shutdown via BrokerLifecycleManager. Since there is no ordering dependency between bringing up ZK brokers and the KRaft quorum during migration, a ZK broker could be running in migration mode, but talking to a ZK controller. A small check was added to see if the current controller is ZK or KRaft before decided which controlled shutdown to attempt. Reviewers: Colin P. McCabe <cmccabe@apache.org>

akhileshchg and others added 6 commits December 14, 2022 18:29

add missing files

fcde47a

Merge remote-tracking branch 'origin/trunk' into migration_driver_stub

35c12dc

Register the migration driver with the controller

5221e33

akhileshchg commented Dec 16, 2022

View reviewed changes

akhileshchg and others added 22 commits December 15, 2022 18:06

Fix build issue and add helper functions to cluster image and delta

88aa6d7

WIP

d52cd95

RPC support from KRaft controller to brokers

df21677

Progress on KRaft to ZK dual write

7da1d1f

Merge remote-tracking branch 'akhilesh/migration_driver_stub' into mi…

032aa3f

…gration_driver_stub

Merge remote-tracking branch 'origin/trunk' into migration_driver_stub

543324a

Progress on end-to-end topic creation

c38c7df

Still an issue with broker epochs in UMR from KRaft

Send ZK broker epoch in registration

d4226dc

track ZK controller epoch and version

69e87f9

Add better logging for migration

de66d21

Merge remote-tracking branch 'origin/trunk' into migration_driver_stub

e526d10

fixup after merge

b93409c

Add system test for online migration

1f874b9

Change the state machine a bit, get system test working

c9fb0e6

Progress on end-to-end test

2f13655

Use the correct epoch on ZK broker

8167ea5

Merge remote-tracking branch 'akhilesh/migration_driver_stub' into mi…

4352a4b

…gration_driver_stub

wip on epoch

4660527

WIP on end-to-end test

4dbb5a3

Merge remote-tracking branch 'upstream/trunk' into migration_driver_stub

e4a7342

cmccabe reviewed Jan 5, 2023

View reviewed changes

clients/src/main/resources/common/message/BrokerRegistrationRequest.json Show resolved Hide resolved

cmccabe reviewed Jan 5, 2023

View reviewed changes

core/src/main/scala/kafka/server/KafkaServer.scala Outdated Show resolved Hide resolved

cmccabe reviewed Jan 5, 2023

View reviewed changes

metadata/src/main/java/org/apache/kafka/metadata/migration/BrokersRpcClient.java Outdated Show resolved Hide resolved

cmccabe reviewed Jan 5, 2023

View reviewed changes

metadata/src/main/java/org/apache/kafka/controller/MigrationControlManager.java Outdated Show resolved Hide resolved

mumrah and others added 4 commits January 5, 2023 12:49

end to end test is passing

dacff92

Merge remote-tracking branch 'akhilesh/migration_driver_stub' into mi…

9e2d4a9

…gration_driver_stub Conflicts: core/src/main/scala/kafka/migration/KRaftControllerToZkBrokersRpcClient.scala

Address comments

cd5141c

Fixup after merge from trunk

028b3e6

cmccabe approved these changes Jan 5, 2023

View reviewed changes

Merge remote-tracking branch 'akhilesh/migration_driver_stub' into mi…

1f937b0

…gration_driver_stub

cmccabe marked this pull request as ready for review January 5, 2023 22:01

cmccabe and others added 8 commits January 6, 2023 10:34

Fix un-thread-safe access to QC fields, fix some variable names

c39b37d

fix checkstyle

960aed5

Revert default exception handler

358f535

Pin metadata.version to 3.4, fix some tests

c5aa43b

Merge remote-tracking branch 'akhilesh/migration_driver_stub' into mi…

b1e2c19

…gration_driver_stub

Fix failing integration tests

9d051dd

Close ZK client before KRaftMigrationDriver in shutdown

c1ddfca

fix checkstyle

894a917

cmccabe merged commit db49070 into apache:trunk Jan 9, 2023

mumrah mentioned this pull request Jan 10, 2023

KAFKA-14304 Use boolean for ZK migrating brokers in RPC/record #13103

Merged

chia7712 mentioned this pull request Apr 18, 2024

KAFKA-16563: retry pollEvent in KRaftMigrationDriver for retriable errors #15732

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-14493: Introduce Zk to KRaft migration state machine STUBs in KRaft controller.#12998

KAFKA-14493: Introduce Zk to KRaft migration state machine STUBs in KRaft controller.#12998
cmccabe merged 41 commits intoapache:trunkfrom
akhileshchg:migration_driver_stub

akhileshchg commented Dec 15, 2022

Uh oh!

akhileshchg Dec 16, 2022

Uh oh!

akhileshchg Dec 16, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cmccabe left a comment

Uh oh!

rishiraj88 commented Jan 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

akhileshchg commented Dec 15, 2022

Uh oh!

akhileshchg Dec 16, 2022

Choose a reason for hiding this comment

Uh oh!

akhileshchg Dec 16, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cmccabe left a comment

Choose a reason for hiding this comment

Uh oh!

rishiraj88 commented Jan 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants