KAFKA-14558: Move/Rewrite LastRecord, TxnMetadata, BatchMetadata, ProducerStateEntry, and ProducerAppendInfo to the storage module by satishd · Pull Request #13043 · apache/kafka

satishd · 2022-12-23T07:04:53Z

For broader context on this change, you may want to look at KAFKA-14470: Move log layer to storage module.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

core/src/main/scala/kafka/log/LogCleaner.scala

ijuma

Thanks for the PR. Please see my comments below.

core/src/main/scala/kafka/log/UnifiedLog.scala

core/src/test/scala/unit/kafka/log/LogSegmentTest.scala

storage/src/main/java/org/apache/kafka/server/log/internals/TxnMetadata.java

storage/src/main/java/org/apache/kafka/server/log/internals/ProducerAppendInfo.java

storage/src/main/java/org/apache/kafka/server/log/internals/ProducerStateEntry.java

ijuma · 2023-01-04T16:42:56Z

@satishd Just checking that you saw the review for this PR (there was an issue with the notifications in a previous one, I believe). I was planning to merge this before reviewing the next one.

satishd · 2023-01-05T05:09:59Z

@ijuma I rebased it with trunk to resolve the conflicts, working on addressing your review comments. I will let you know once I am done with my changes.

Minor cleanup in classes move to storage module. Resolved upstream trunk merge conflicts.

storage/src/main/java/org/apache/kafka/server/log/internals/ProducerStateEntry.java

…of ProducerStateEntry instances.

…e of BatchMetadata.

satishd · 2023-01-06T17:57:53Z

Thanks @ijuma for the review. I address them in the code and replied to your comment.

storage/src/main/java/org/apache/kafka/server/log/internals/ProducerStateEntry.java

core/src/main/scala/kafka/log/LogCleaner.scala

storage/src/main/java/org/apache/kafka/server/log/internals/ProducerAppendInfo.java

ijuma

Thanks for the updates. This is pretty close. Left a few more comments. I resolved all the items that were already addressed.

core/src/test/scala/unit/kafka/log/ProducerStateManagerTest.scala

storage/src/main/java/org/apache/kafka/server/log/internals/TxnMetadata.java

storage/src/main/java/org/apache/kafka/server/log/internals/ProducerAppendInfo.java

storage/src/main/java/org/apache/kafka/server/log/internals/ProducerStateEntry.java

satishd · 2023-01-07T13:47:23Z

Thanks @ijuma for the review comments. Replied inline and/or addressed with the latest commits.

ijuma

Thanks for the updates. See comments below.

storage/src/main/java/org/apache/kafka/server/log/internals/ProducerAppendInfo.java

storage/src/main/java/org/apache/kafka/server/log/internals/ProducerStateEntry.java

storage/src/main/java/org/apache/kafka/server/log/internals/ProducerAppendInfo.java

satishd · 2023-01-08T01:12:59Z

Thanks @ijuma for the review. Addressed them with the latest commit.

ijuma

LGTM, thanks!

ijuma · 2023-01-08T04:13:00Z

JDK 8 build passed, the other builds had unrelated flaky failures.

ijuma · 2023-01-08T04:25:38Z

@satishd I submitted a follow-up PR that removes the public mutable fields from ProducerStateEntry: #13091

…9-jan-2023 * apache/trunk: (16 commits) KAFKA-14570: Fix parenthesis in verifyFullFetchResponsePartitions output (apache#13072) MINOR: Remove public mutable fields from ProducerAppendInfo (apache#13091) KAFKA-14558: Move/Rewrite LastRecord, TxnMetadata, BatchMetadata, ProducerStateEntry, and ProducerAppendInfo to the storage module (apache#13043) KAFKA-14535: Fix flaky EndToEndAuthorization tests which were sensitive to ACL change reordering (apache#13086) KAFKA-9087 Replace log high watermark by future log high watermark wh… (apache#13075) MINOR: add error reason when controller failed to handle events (apache#13050) MINOR: doc: note how JDK-8136913 can affect client SASL (apache#13071) 2023 (apache#13083) KAFKA-14279; Add 3.3.x to core compatibility tests (apache#13076) MINOR Fixed doc generation for LogConfig class in genTopicConfigDocs. (apache#13079) ...

…ducerStateEntry, and ProducerAppendInfo to the storage module (apache#13043) For broader context on this change, see: * KAFKA-14470: Move log layer to storage module. Reviewers: Ismael Juma <ismael@juma.me.uk>

ijuma reviewed Dec 23, 2022

View reviewed changes

core/src/main/scala/kafka/log/LogCleaner.scala Outdated Show resolved Hide resolved

satishd force-pushed the mv-producer-statemanager-auxiliary-classes branch 2 times, most recently from b00e944 to 2c7a350 Compare December 29, 2022 12:39

satishd changed the title ~~[Draft] Move ProducerStateManager auxiliary classes to storage module.~~ KAFKA-14558 Move LastRecord, TxnMetadata, BatchMetadata, ProducerStateEntry, and ProducerAppendInfo to the storage module. Dec 30, 2022

satishd force-pushed the mv-producer-statemanager-auxiliary-classes branch from 2c7a350 to 4eef3a2 Compare December 30, 2022 06:23

satishd marked this pull request as ready for review December 30, 2022 06:24

satishd marked this pull request as draft December 30, 2022 06:25

satishd force-pushed the mv-producer-statemanager-auxiliary-classes branch from 4eef3a2 to 4316d11 Compare December 30, 2022 07:27

satishd marked this pull request as ready for review December 30, 2022 08:03

satishd force-pushed the mv-producer-statemanager-auxiliary-classes branch from 4316d11 to 72f2872 Compare December 31, 2022 13:10

ijuma reviewed Jan 2, 2023

View reviewed changes

satishd force-pushed the mv-producer-statemanager-auxiliary-classes branch from f41ef30 to 060de9e Compare January 5, 2023 01:13

ijuma changed the title ~~KAFKA-14558 Move LastRecord, TxnMetadata, BatchMetadata, ProducerStateEntry, and ProducerAppendInfo to the storage module.~~ KAFKA-14558: Move LastRecord, TxnMetadata, BatchMetadata, ProducerStateEntry, and ProducerAppendInfo to the storage module. Jan 5, 2023

ijuma changed the title ~~KAFKA-14558: Move LastRecord, TxnMetadata, BatchMetadata, ProducerStateEntry, and ProducerAppendInfo to the storage module.~~ KAFKA-14558: Move LastRecord, TxnMetadata, BatchMetadata, ProducerStateEntry, and ProducerAppendInfo to the storage module Jan 5, 2023

satishd added 5 commits January 5, 2023 13:34

Moved LastRecord and TxnMetadata to storage module

e0ebe0a

Moved BatchMetadata and ProducerStateEntry to storage module

c6710e0

Moved ProducerAppendInfo to storage module.

472998b

Minor cleanup in classes move to storage module. Resolved upstream trunk merge conflicts.

Resolved a few failing tests

2d9b9ef

Addressed reveiw comments

fc77d30

satishd force-pushed the mv-producer-statemanager-auxiliary-classes branch from 7edd1e9 to 7a1f9b9 Compare January 5, 2023 09:14

Made producerId as private with getter

3ecb555

satishd force-pushed the mv-producer-statemanager-auxiliary-classes branch from 7a1f9b9 to d181994 Compare January 5, 2023 15:54

ijuma reviewed Jan 5, 2023

View reviewed changes

storage/src/main/java/org/apache/kafka/server/log/internals/ProducerStateEntry.java Outdated Show resolved Hide resolved

Minor cleanup to pass batachmetadata list and other minro changes.

89e2ec3

satishd force-pushed the mv-producer-statemanager-auxiliary-classes branch from d181994 to 89e2ec3 Compare January 5, 2023 16:32

satishd added 2 commits January 6, 2023 18:30

Fixed a few tests failed in ProducerManagerTest introducing equality …

499bf70

…of ProducerStateEntry instances.

Addressed review comments. List arg is replaced with a single instanc…

a01c9ec

…e of BatchMetadata.

ijuma reviewed Jan 6, 2023

View reviewed changes

storage/src/main/java/org/apache/kafka/server/log/internals/ProducerStateEntry.java Outdated Show resolved Hide resolved

ijuma reviewed Jan 6, 2023

View reviewed changes

core/src/main/scala/kafka/log/LogCleaner.scala Outdated Show resolved Hide resolved

storage/src/main/java/org/apache/kafka/server/log/internals/ProducerAppendInfo.java Outdated Show resolved Hide resolved

ijuma reviewed Jan 6, 2023

View reviewed changes

satishd changed the title ~~KAFKA-14558: Move LastRecord, TxnMetadata, BatchMetadata, ProducerStateEntry, and ProducerAppendInfo to the storage module~~ KAFKA-14558: Move/Rewrite LastRecord, TxnMetadata, BatchMetadata, ProducerStateEntry, and ProducerAppendInfo to the storage module Jan 7, 2023

Addressed review comments

ef3b5cd

satishd force-pushed the mv-producer-statemanager-auxiliary-classes branch from 6e726dc to ef3b5cd Compare January 7, 2023 13:35

ijuma reviewed Jan 7, 2023

View reviewed changes

Addressed review comments

a4aaf83

ijuma approved these changes Jan 8, 2023

View reviewed changes

ijuma merged commit 2dec39d into apache:trunk Jan 8, 2023

Conversation

satishd commented Dec 23, 2022 • edited by ijuma Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Committer Checklist (excluded from commit message)

Uh oh!

Uh oh!

ijuma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ijuma commented Jan 4, 2023

Uh oh!

satishd commented Jan 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

satishd commented Jan 6, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ijuma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

satishd commented Jan 7, 2023

Uh oh!

ijuma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

satishd commented Jan 8, 2023

Uh oh!

ijuma left a comment

Choose a reason for hiding this comment

Uh oh!

ijuma commented Jan 8, 2023

Uh oh!

ijuma commented Jan 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

satishd commented Dec 23, 2022 •

edited by ijuma

Loading

satishd commented Jan 5, 2023 •

edited

Loading

ijuma commented Jan 8, 2023 •

edited

Loading