Skip to content

[FLINK-33331] Update netty to 4.1.100 and arrow to 13.0.0 to make flink-python passing on java 21 #23664

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
snuyanzin merged 1 commit into from
Nov 12, 2023
Merged

[FLINK-33331] Update netty to 4.1.100 and arrow to 13.0.0 to make flink-python passing on java 21 #23664

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion flink-python/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ under the License.
<packaging>jar</packaging>

<properties>
<arrow.version>5.0.0</arrow.version>
<arrow.version>13.0.0</arrow.version>
<kafka.version>3.2.3</kafka.version>
<surefire.module.config><!--
CommonTestUtils#setEnv
Expand Down
12 changes: 6 additions & 6 deletions flink-python/src/main/resources/META-INF/NOTICE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@ This project bundles the following dependencies under the Apache Software Licens
- com.fasterxml.jackson.core:jackson-core:2.14.3
- com.fasterxml.jackson.core:jackson-databind:2.14.3
- com.google.flatbuffers:flatbuffers-java:1.12.0
- io.netty:netty-buffer:4.1.70.Final
- io.netty:netty-common:4.1.70.Final
- io.netty:netty-buffer:4.1.100.Final
- io.netty:netty-common:4.1.100.Final
- joda-time:joda-time:2.5
- org.apache.arrow:arrow-format:5.0.0
- org.apache.arrow:arrow-memory-core:5.0.0
- org.apache.arrow:arrow-memory-netty:5.0.0
- org.apache.arrow:arrow-vector:5.0.0
- org.apache.arrow:arrow-format:13.0.0
- org.apache.arrow:arrow-memory-core:13.0.0
- org.apache.arrow:arrow-memory-netty:13.0.0
- org.apache.arrow:arrow-vector:13.0.0
- org.apache.beam:beam-model-fn-execution:2.43.0
- org.apache.beam:beam-model-job-management:2.43.0
- org.apache.beam:beam-model-pipeline:2.43.0
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -887,7 +887,7 @@ under the License.
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-bom</artifactId>
<version>4.1.70.Final</version>
<version>4.1.100.Final</version>
Copy link
Contributor

@TanYuxin-tyx TanYuxin-tyx Nov 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @snuyanzin Thank you for the netty version update. I have been working with netty recently and have noticed a hardcoded netty version in the flink-rpc-akka/pom.xml (

flink/flink-rpc/flink-rpc-akka/pom.xml

Line 98 in b759794

<version>3.10.6.Final</version>
). Do we need to update that version?
In addition, it appears that the original hardcoded version might not be accurate.

Copy link
Contributor Author

@snuyanzin snuyanzin Nov 7, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @TanYuxin-tyx
thanks for the question
the reason why upgrade of netty and arrow is required is failing of number of python tests while building with jdk21

I have my own pipepline setup with jdk21
example of error could be seen e.g. https://dev.azure.com/snuyanzin/flink/_build/results?buildId=2490&view=logs&j=fba17979-6d2e-591d-72f1-97cf42797c11&t=727942b6-6137-54f7-1ef9-e66e706ea068

same error was mentioned in FLINK-33331 description.

Since both netty and arrow require this removed constructor during python test they both need bumping the version.
By the way here it is the result of same pipeline with cherry-picked commit from this PR https://dev.azure.com/snuyanzin/flink/_build/results?buildId=2491&view=results
it could be seen that after that python tests with java 21 are passing (there are some other failures for other tests which are being fixed within other jira issue under FLINK-33163)

About the dependency you've mentioned: this dependency is used only for Pekko (former Akka) since current Pekko version can not run with netty 4.x... At the same time based on tests it is not an issue, looks like they don't use this removed constructor... and no need to bump the version

Copy link
Contributor Author

@snuyanzin snuyanzin Nov 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more note about flink-rpc-akka dependency

in theory it could be update in future within a separate task
however first there should be support of Netty 4 from Pekko's side which is done within apache/pekko#643
however it is not released yet...

Copy link
Contributor

@TanYuxin-tyx TanYuxin-tyx Nov 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@snuyanzin OK, make sense. Thanks for your patient explanations.

<type>pom</type>
<scope>import</scope>
</dependency>
Expand Down