KAFKA-18052: Decouple the dependency of feature stable version to the metadata version #17886

dongnuo123 · 2024-11-20T20:46:32Z

Currently the validation of feature upgrade relies on the supported version range generated during registration. For a given feature, its max supported feature version in production is set to be the default version value (the latest feature version with bootstrap metadata value smaller or equal to the latest production metadata value).

This patch introduces a LATEST_PRODUCTION value independent from the metadata version to each feature so that the highest supported feature version can be customized by the feature owner.

The change only applies to dynamic feature upgrade. During formatting, we still use the default value associated the metadata version.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

server-common/src/main/java/org/apache/kafka/server/common/TestFeatureVersion.java

junrao

@dongnuo123 : Thanks for the PR. Left a comment.

server-common/src/main/java/org/apache/kafka/server/common/Features.java

junrao

@dongnuo123 : Thanks for the updated PR. A few minor comments.

server-common/src/main/java/org/apache/kafka/server/common/Features.java

server-common/src/test/java/org/apache/kafka/server/common/FeaturesTest.java

server-common/src/main/java/org/apache/kafka/server/common/TestFeatureVersion.java

metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java

junrao

@dongnuo123 : Thanks for the updated PR. A few more comments.

server-common/src/main/java/org/apache/kafka/server/common/Features.java

server-common/src/main/java/org/apache/kafka/server/common/UnitTestFeatureVersion.java

junrao · 2024-11-25T20:01:50Z

server-common/src/main/java/org/apache/kafka/server/common/Features.java

+    UNIT_TEST_VERSION_2(UnitTestFeatureVersion.FEATURE_NAME + ".2", new FeatureVersion[]{UT_FV2_0, UT_FV2_1}, UT_FV2_0),
+    UNIT_TEST_VERSION_3(UnitTestFeatureVersion.FEATURE_NAME + ".3", new FeatureVersion[]{UT_FV3_0, UT_FV3_1}, UT_FV3_1),
+    UNIT_TEST_VERSION_4(UnitTestFeatureVersion.FEATURE_NAME + ".4", new FeatureVersion[]{UT_FV4_0, UT_FV4_1}, UT_FV4_1),
+    UNIT_TEST_VERSION_5(UnitTestFeatureVersion.FEATURE_NAME + ".5", new FeatureVersion[]{UT_FV5_0, UT_FV5_1}, UT_FV5_1);


It's kind of weird to include unit test code in production code.

Yeah it's a bit weird. The formatter unit tests are also failing because of these unrecognized features. I've tried several ways of addressing it but still don't have a rather good solution.

The issues are that 1) Feature is an enum so we can't extend from it to create a new enum specially for unit tests 2) The validation calls featureFromName which needs the Features[] array, so the Features for unit tests have to really exist.

Another way to separate the production features and the unit test features is to create an exactly same enum called UnitTestFeatures, but that will be much duplication and pretty hard to maintain, which I don't really want to adopt

server-common/src/test/java/org/apache/kafka/server/common/FeaturesTest.java

…roduction

junrao

@dongnuo123 : Thanks for the updated PR. A few more comments.

server-common/src/main/java/org/apache/kafka/server/common/Feature.java

junrao · 2024-12-04T00:14:36Z

server-common/src/main/java/org/apache/kafka/server/common/Feature.java

+                            dependencyFeature.defaultLevel(MetadataVersion.LATEST_PRODUCTION)));
+                    }
+                } else {
+                    if (dependency.getValue() > defaultVersion.bootstrapMetadataVersion().featureLevel()) {


defaultVersion.bootstrapMetadataVersion().featureLevel() => metadataVersion ?

The two are not necessarily equal. For instance, when metadataVersion=120, the feature has versions

FV_1, bootstrap MV = 119

FV_2, bootstrap MV = 121

Then defaultVersion.bootstrapMetadataVersion() will be 119. Using metadataVersion would also work since we do check when the two are equal, but I feel comparing with the bootstrap MV makes more sense.

server-common/src/main/java/org/apache/kafka/server/common/Feature.java

server-common/src/main/java/org/apache/kafka/server/common/TestFeatureVersion.java

junrao · 2024-12-04T00:54:45Z

server-common/src/main/java/org/apache/kafka/server/common/TestFeatureVersion.java

+
+    // For testing the default value has MV dependency that is ahead of the bootstrap MV.
+    UT_FV7_0(0, MetadataVersion.MINIMUM_KRAFT_VERSION, Collections.singletonMap(MetadataVersion.FEATURE_NAME, MetadataVersion.IBP_3_7_IV0.featureLevel())),
+    UT_FV7_1(1, MetadataVersion.IBP_3_8_IV0, Collections.singletonMap(MetadataVersion.FEATURE_NAME, MetadataVersion.IBP_3_8_IV0.featureLevel()));


It seems that we don't really need UT_FV7_1?

Adding UT_FV7_1 was meant to test when the failed feature version is not the default version of latest production mv, but yes there's no impact removing it

server-common/src/test/java/org/apache/kafka/server/common/FeatureTest.java

server-common/src/main/java/org/apache/kafka/server/common/TestFeatureVersion.java

server-common/src/main/java/org/apache/kafka/server/common/Feature.java

junrao

@dongnuo123 : Thanks for the updated PR. Just a few minor comments.

server-common/src/main/java/org/apache/kafka/server/common/GroupVersion.java

server-common/src/main/java/org/apache/kafka/server/common/KRaftVersion.java

server-common/src/main/java/org/apache/kafka/server/common/TestFeatureVersion.java

junrao

@dongnuo123 : Thanks for the updated PR. LGTM. The failed tests are marked as flaky. So, will merge this PR.

… metadata version (apache#17886) Currently the validation of feature upgrade relies on the supported version range generated during registration. For a given feature, its max supported feature version in production is set to be the default version value (the latest feature version with bootstrap metadata value smaller or equal to the latest production metadata value). This patch introduces a LATEST_PRODUCTION value independent from the metadata version to each feature so that the highest supported feature version can be customized by the feature owner. The change only applies to dynamic feature upgrade. During formatting, we still use the default value associated the metadata version. Reviewers: Justine Olshan <jolshan@confluent.io>, Jun Rao <junrao@gmail.com>

add latest production to features

0efae34

github-actions bot added core Kafka Broker small Small PRs labels Nov 20, 2024

dongnuo123 changed the title ~~[Draft] add latest production to features~~ KAFKA-18052: Decouple the dependency of feature stable version to the metadata version Nov 20, 2024

dongnuo123 changed the title ~~KAFKA-18052: Decouple the dependency of feature stable version to the metadata version~~ [Draft] KAFKA-18052: Decouple the dependency of feature stable version to the metadata version Nov 20, 2024

jolshan reviewed Nov 20, 2024

View reviewed changes

server-common/src/main/java/org/apache/kafka/server/common/TestFeatureVersion.java Outdated Show resolved Hide resolved

dongnuo123 added 2 commits November 20, 2024 17:28

check latestProduction == latestTesting in TestFeature

540f99e

check latestProduction == latestTesting in TestFeature

381ca63

dongnuo123 changed the title ~~[Draft] KAFKA-18052: Decouple the dependency of feature stable version to the metadata version~~ KAFKA-18052: Decouple the dependency of feature stable version to the metadata version Nov 21, 2024

junrao reviewed Nov 21, 2024

View reviewed changes

server-common/src/main/java/org/apache/kafka/server/common/Features.java Outdated Show resolved Hide resolved

validateDefaultValueAndLatestProductionValue

c8c290e

github-actions bot removed the small Small PRs label Nov 21, 2024

dajac added the ci-approved label Nov 22, 2024

junrao reviewed Nov 22, 2024

View reviewed changes

more unit tests + rename defaultValue

d2ea645

github-actions bot added tools kraft labels Nov 23, 2024

dongnuo123 added 2 commits November 23, 2024 15:48

validate test version

d49643d

spotlessapply

c7731f7

dongnuo123 commented Nov 25, 2024

View reviewed changes

metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java Show resolved Hide resolved

junrao reviewed Nov 25, 2024

View reviewed changes

dongnuo123 added 8 commits December 3, 2024 10:35

Merge remote-tracking branch 'ak/trunk' into feature-version-latest-p…

60d5c9e

…roduction

lateset production

198d95a

comments

a831bce

move UnitTestFeatureVersion to TestFeatureVersion

0a9b35a

rename Features to Feature

52f3cb8

validate with all MVs' default value

4485c8c

spotless apply

361cbfa

Merge remote-tracking branch 'ak/trunk' into feature-version-latest-p…

74f1efe

…roduction

junrao reviewed Dec 4, 2024

View reviewed changes

dongnuo123 added 4 commits December 4, 2024 11:48

minor change in comments

15e2540

move unit test features out to a separate file

61e0542

move unit test features out to a separate file

4232407

minor

1ecfed8

junrao reviewed Dec 4, 2024

View reviewed changes

remove valuesWithoutUnitTestVersions

2c1c9a7

junrao approved these changes Dec 5, 2024

View reviewed changes

junrao merged commit e30edb3 into apache:trunk Dec 5, 2024
7 of 8 checks passed

junrao deleted the feature-version-latest-production branch December 5, 2024 19:07

junrao mentioned this pull request Dec 11, 2024

KAFKA-17705: Add Transactions V2 system tests and mark as production ready #18132

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-18052: Decouple the dependency of feature stable version to the metadata version #17886

KAFKA-18052: Decouple the dependency of feature stable version to the metadata version #17886

dongnuo123 commented Nov 20, 2024 •

edited

Loading

junrao left a comment

junrao left a comment

junrao left a comment

junrao Nov 25, 2024

dongnuo123 Dec 3, 2024 •

edited

Loading

junrao left a comment

junrao Dec 4, 2024

dongnuo123 Dec 4, 2024

junrao Dec 4, 2024

dongnuo123 Dec 4, 2024

junrao left a comment

junrao left a comment

KAFKA-18052: Decouple the dependency of feature stable version to the metadata version #17886

KAFKA-18052: Decouple the dependency of feature stable version to the metadata version #17886

Conversation

dongnuo123 commented Nov 20, 2024 • edited Loading

Committer Checklist (excluded from commit message)

junrao left a comment

Choose a reason for hiding this comment

junrao left a comment

Choose a reason for hiding this comment

junrao left a comment

Choose a reason for hiding this comment

junrao Nov 25, 2024

Choose a reason for hiding this comment

dongnuo123 Dec 3, 2024 • edited Loading

Choose a reason for hiding this comment

junrao left a comment

Choose a reason for hiding this comment

junrao Dec 4, 2024

Choose a reason for hiding this comment

dongnuo123 Dec 4, 2024

Choose a reason for hiding this comment

junrao Dec 4, 2024

Choose a reason for hiding this comment

dongnuo123 Dec 4, 2024

Choose a reason for hiding this comment

junrao left a comment

Choose a reason for hiding this comment

junrao left a comment

Choose a reason for hiding this comment

dongnuo123 commented Nov 20, 2024 •

edited

Loading

dongnuo123 Dec 3, 2024 •

edited

Loading