[SPARK-41197][BUILD] Upgrade Kafka to 3.3.1 by tedyu · Pull Request #38715 · apache/spark

tedyu · 2022-11-18T18:46:53Z

What changes were proposed in this pull request?

This PR upgrades Kafka to 3.3.1 release.

The new default partitioner keeps track of how many bytes are produced per-partition and once the amount exceeds batch.size, it switches to the next partition. For spark kafka tests, this will result in records being sent to only one partition in some tests.
KafkaTestUtils.producerConfiguration is modified to use DefaultPartitioner.

Why are the changes needed?

Kafka 3.3.1 release has new features along with bug fixes: https://www.confluent.io/blog/apache-kafka-3-3-0-new-features-and-updates/

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing test suite

tedyu · 2022-11-18T21:53:38Z

@HeartSaVioR
Can you take a look ?

bjornjorgensen · 2022-11-19T20:53:33Z

Hi, 3.3.0 "A significant bug was found in the 3.3.0 release after artifacts were pushed to Apache and Maven central but prior to the release announcement. As a result, the decision was made to not announce 3.3.0 and instead release 3.3.1 with the fix. It is recommended that 3.3.0 not be used."

Can you try with 3.3.1

tedyu · 2022-11-19T21:56:07Z

@bjornjorgensen
I have updated the PR.

dongjoon-hyun

Hi, @tedyu . Could you fix UT failures?

[info] *** 90 TESTS FAILED ***
[error] Failed tests:
[error] 	org.apache.spark.sql.kafka010.KafkaMicroBatchV2SourceSuite
[error] 	org.apache.spark.sql.kafka010.KafkaMicroBatchV1SourceWithAdminSuite
[error] 	org.apache.spark.sql.kafka010.KafkaMicroBatchV1SourceSuite
[error] 	org.apache.spark.sql.kafka010.KafkaContinuousSourceSuite
[error] 	org.apache.spark.sql.kafka010.KafkaMicroBatchV2SourceWithAdminSuite

tedyu · 2022-11-21T14:43:13Z

I noticed the test failures.
Let me analyze the test output.

bjornjorgensen · 2022-11-21T19:29:40Z

Can it have something to do with apache/kafka#12794 ?

dengziming · 2022-11-22T04:31:07Z

These failures comes from apache/kafka#12049 and is described here: https://kafka.apache.org/documentation/#upgrade_33_notable
The new default partitioner keeps track of how many bytes are produced per-partition and once the amount exceeds batch.size, switches to the next partition. In spark kafka tests, this will result in records being sent to only one partition in some tests.
One simplest solution is add props.put("partitioner.class",classOf[org.apache.kafka.clients.producer.internals.DefaultPartitioner].getName) in KafkaTestUtils.producerConfiguration, or we can implement our own partitioner, or set a smallbatch.size config.

HyukjinKwon · 2022-11-22T07:04:16Z

cc @dongjoon-hyun and @HeartSaVioR FYI

tedyu · 2022-11-22T14:13:26Z

Thanks @dengziming for the information.

tedyu · 2022-11-22T14:14:01Z

@HyukjinKwon @dongjoon-hyun @HeartSaVioR
Please take a look - tests pass.

dongjoon-hyun

+1, LGTM. Thank you, @tedyu , @bjornjorgensen , @dengziming, @HyukjinKwon .
Merged to master for Apache Spark 3.4.

This PR upgrades Kafka to 3.3.1 release. The new default partitioner keeps track of how many bytes are produced per-partition and once the amount exceeds `batch.size`, it switches to the next partition. For spark kafka tests, this will result in records being sent to only one partition in some tests. `KafkaTestUtils.producerConfiguration` is modified to use `DefaultPartitioner`. Kafka 3.3.1 release has new features along with bug fixes: https://www.confluent.io/blog/apache-kafka-3-3-0-new-features-and-updates/ No Existing test suite Closes #38715 from tedyu/k-33. Authored-by: Ted Yu <yuzhihong@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

dongjoon-hyun · 2022-11-28T20:58:38Z

This is merged via 0ff201c .

### What changes were proposed in this pull request? This PR upgrades Kafka to 3.3.1 release. The new default partitioner keeps track of how many bytes are produced per-partition and once the amount exceeds `batch.size`, it switches to the next partition. For spark kafka tests, this will result in records being sent to only one partition in some tests. `KafkaTestUtils.producerConfiguration` is modified to use `DefaultPartitioner`. ### Why are the changes needed? Kafka 3.3.1 release has new features along with bug fixes: https://www.confluent.io/blog/apache-kafka-3-3-0-new-features-and-updates/ ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing test suite Closes apache#38715 from tedyu/k-33. Lead-authored-by: Dongjoon Hyun <dongjoon@apache.org> Co-authored-by: Ted Yu <yuzhihong@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

This PR upgrades Kafka to 3.3.1 release. The new default partitioner keeps track of how many bytes are produced per-partition and once the amount exceeds `batch.size`, it switches to the next partition. For spark kafka tests, this will result in records being sent to only one partition in some tests. `KafkaTestUtils.producerConfiguration` is modified to use `DefaultPartitioner`. Kafka 3.3.1 release has new features along with bug fixes: https://www.confluent.io/blog/apache-kafka-3-3-0-new-features-and-updates/ No Existing test suite Closes apache#38715 from tedyu/k-33. Authored-by: Ted Yu <yuzhihong@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? This PR upgrades Kafka to 3.3.1 release. The new default partitioner keeps track of how many bytes are produced per-partition and once the amount exceeds `batch.size`, it switches to the next partition. For spark kafka tests, this will result in records being sent to only one partition in some tests. `KafkaTestUtils.producerConfiguration` is modified to use `DefaultPartitioner`. ### Why are the changes needed? Kafka 3.3.1 release has new features along with bug fixes: https://www.confluent.io/blog/apache-kafka-3-3-0-new-features-and-updates/ ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing test suite Closes apache#38715 from tedyu/k-33. Lead-authored-by: Dongjoon Hyun <dongjoon@apache.org> Co-authored-by: Ted Yu <yuzhihong@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

This PR upgrades Kafka to 3.3.1 release. The new default partitioner keeps track of how many bytes are produced per-partition and once the amount exceeds `batch.size`, it switches to the next partition. For spark kafka tests, this will result in records being sent to only one partition in some tests. `KafkaTestUtils.producerConfiguration` is modified to use `DefaultPartitioner`. Kafka 3.3.1 release has new features along with bug fixes: https://www.confluent.io/blog/apache-kafka-3-3-0-new-features-and-updates/ No Existing test suite Closes apache#38715 from tedyu/k-33. Authored-by: Ted Yu <yuzhihong@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

github-actions bot added the BUILD label Nov 18, 2022

tedyu force-pushed the k-33 branch from 588d0ff to 4ea889c Compare November 19, 2022 21:55

dongjoon-hyun requested changes Nov 21, 2022

View reviewed changes

tedyu force-pushed the k-33 branch from 4ea889c to 9d7d1ca Compare November 22, 2022 04:37

github-actions bot added SQL STRUCTURED STREAMING labels Nov 22, 2022

[SPARK-41197] Upgrade Kafka version to 3.3 release

6355f90

tedyu force-pushed the k-33 branch from 9d7d1ca to 6355f90 Compare November 22, 2022 10:44

dongjoon-hyun changed the title ~~[SPARK-41197] Upgrade Kafka version to 3.3 release~~ [SPARK-41197][BUILD] Upgrade Kafka to 3.3.1 Nov 28, 2022

dongjoon-hyun approved these changes Nov 28, 2022

View reviewed changes

dongjoon-hyun added 2 commits November 28, 2022 12:48

Fix indentation

264d2a9

Fix indentation

de32457

dongjoon-hyun closed this in 60ab363 Nov 28, 2022

wayneguow mentioned this pull request Jun 13, 2024

[SPARK-48614][SQL][SS] Cleanup deprecated api usage related to kafka-clients #46968

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-41197][BUILD] Upgrade Kafka to 3.3.1#38715

[SPARK-41197][BUILD] Upgrade Kafka to 3.3.1#38715
tedyu wants to merge 3 commits intoapache:masterfrom
tedyu:k-33

tedyu commented Nov 18, 2022 •

edited

Loading

Uh oh!

tedyu commented Nov 18, 2022

Uh oh!

bjornjorgensen commented Nov 19, 2022 •

edited

Loading

Uh oh!

tedyu commented Nov 19, 2022

Uh oh!

dongjoon-hyun left a comment

Uh oh!

tedyu commented Nov 21, 2022

Uh oh!

bjornjorgensen commented Nov 21, 2022

Uh oh!

dengziming commented Nov 22, 2022 •

edited

Loading

Uh oh!

HyukjinKwon commented Nov 22, 2022

Uh oh!

tedyu commented Nov 22, 2022

Uh oh!

tedyu commented Nov 22, 2022

Uh oh!

dongjoon-hyun left a comment

Uh oh!

dongjoon-hyun commented Nov 28, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

tedyu commented Nov 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

tedyu commented Nov 18, 2022

Uh oh!

bjornjorgensen commented Nov 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tedyu commented Nov 19, 2022

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

tedyu commented Nov 21, 2022

Uh oh!

bjornjorgensen commented Nov 21, 2022

Uh oh!

dengziming commented Nov 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Nov 22, 2022

Uh oh!

tedyu commented Nov 22, 2022

Uh oh!

tedyu commented Nov 22, 2022

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Nov 28, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tedyu commented Nov 18, 2022 •

edited

Loading

bjornjorgensen commented Nov 19, 2022 •

edited

Loading

dengziming commented Nov 22, 2022 •

edited

Loading