[FLINK-32612] Upgrade Pulsar version to 3.0.0 #54

tisonkun · 2023-07-12T12:21:21Z

Purpose of the change

Upgrade Pulsar client and admin version to 3.0.0 for the next development cycle.

Verifying this change

Existing CI workflows should pass.

Significant changes

Dependencies have been added or upgraded - Pulsar version to 3.0.0
Public API has been changed (Public API is any class annotated with @Public(Evolving))
Serializers have been changed
New feature has been introduced
- If yes, how is this documented? (not applicable / docs / JavaDocs / not documented)

Signed-off-by: tison <wander4096@gmail.com>

tisonkun · 2023-07-12T13:25:42Z

@zentol can we disable false positive for LicenseChecker?

https://github.com/apache/flink-connector-pulsar/actions/runs/5531507804/jobs/10092293645

12:59:18,280 [org.apache.flink.tools.ci.licensecheck.LicenseChecker.main()] WARN  org.apache.flink.tools.ci.licensecheck.LicenseChecker        [] - THIS UTILITY IS ONLY CHECKING FOR COMMON LICENSING MISTAKES. A MANUAL CHECK OF THE NOTICE FILES, DEPLOYED ARTIFACTS, ETC. IS STILL NEEDED!
13:02:21,507 [org.apache.flink.tools.ci.licensecheck.LicenseChecker.main()] INFO  org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Extracted 3 modules that were deployed and 4 modules which bundle dependencies with a total of 4 dependencies
13:02:21,583 [org.apache.flink.tools.ci.licensecheck.LicenseChecker.main()] INFO  org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Found 1 NOTICE files to check
13:02:21,592 [org.apache.flink.tools.ci.licensecheck.LicenseChecker.main()] INFO  org.apache.flink.tools.ci.licensecheck.JarFileChecker        [] - Checking directory /tmp/flink-validation-deployment with a total of 4 jar files.
13:02:21,905 [org.apache.flink.tools.ci.licensecheck.LicenseChecker.main()] ERROR org.apache.flink.tools.ci.licensecheck.JarFileChecker        [] - File '/org/apache/pulsar/shade/javax/xml/bind/Messages.properties' in jar '/tmp/flink-validation-deployment/org/apache/flink/flink-sql-connector-pulsar/4.0-SNAPSHOT/flink-sql-connector-pulsar-4.0-20230712.122728-1.jar' contains match with forbidden regex 'gnu ?\R?[\s/#]*general ?\R?[\s/#]*public ?\R?[\s/#]*license'.
13:02:21,907 [org.apache.flink.tools.ci.licensecheck.LicenseChecker.main()] ERROR org.apache.flink.tools.ci.licensecheck.JarFileChecker        [] - File '/org/apache/pulsar/shade/javax/xml/bind/helpers/Messages.properties' in jar '/tmp/flink-validation-deployment/org/apache/flink/flink-sql-connector-pulsar/4.0-SNAPSHOT/flink-sql-connector-pulsar-4.0-20230712.122728-1.jar' contains match with forbidden regex 'gnu ?\R?[\s/#]*general ?\R?[\s/#]*public ?\R?[\s/#]*license'.
13:02:21,909 [org.apache.flink.tools.ci.licensecheck.LicenseChecker.main()] ERROR org.apache.flink.tools.ci.licensecheck.JarFileChecker        [] - File '/org/apache/pulsar/shade/javax/xml/bind/util/Messages.properties' in jar '/tmp/flink-validation-deployment/org/apache/flink/flink-sql-connector-pulsar/4.0-SNAPSHOT/flink-sql-connector-pulsar-4.0-20230712.122728-1.jar' contains match with forbidden regex 'gnu ?\R?[\s/#]*general ?\R?[\s/#]*public ?\R?[\s/#]*license'.
13:02:25,349 [org.apache.flink.tools.ci.licensecheck.LicenseChecker.main()] WARN  org.apache.flink.tools.ci.licensecheck.LicenseChecker        [] - Found a total of 3 severe license issues

tisonkun · 2023-07-12T13:26:18Z

The issued dependency should be fine like JDK as a dependency - GPL Classpath Exception

tisonkun · 2023-07-12T13:27:53Z

I'll try to exclude them. But generally it's a false positive.

Signed-off-by: tison <wander4096@gmail.com>

tisonkun · 2023-07-14T05:02:46Z

The log was not found. It may have been deleted based on retention settings

@MartijnVisser @zentol how can I debug this CI failure then?

EDIT - Resolved after a retry. But the CI tests are falsy passed, maybe the ci_utils can be still improved.

Signed-off-by: tison <wander4096@gmail.com>

zentol · 2023-07-14T07:37:48Z

But the CI tests are falsy passe

How so?

tisonkun · 2023-07-14T07:53:48Z

https://github.com/apache/flink-connector-pulsar/actions/runs/5550771150/jobs/10136213423

How so?

@zentol You're right that the sub workflow failed. Although the GitHub check suite display the overall state as passed, it should be a GitHub issue. Perhaps introduce a final required job can improve the experience, like -

https://github.com/apache/kvrocks/blob/0d2304300b494898b1626a38dfc5fbafe523b5a3/.github/workflows/kvrocks.yaml#L357-L379

eolivelli

+1

tisonkun · 2023-07-14T09:37:08Z

@eolivelli Still some loading issues. I don't know if there is some semantic changes in runtime.

Currently -

Caused by: java.lang.ClassNotFoundException: org.apache.pulsar.shade.io.netty.channel.group.ChannelMatchers

cc @syhily for advice :D

Signed-off-by: tison <wander4096@gmail.com>

tisonkun · 2023-07-16T07:42:53Z

Luckily all tests passed now. I'll check the details on the logs.

cbornet · 2023-07-16T15:32:56Z

LGTM

tisonkun · 2023-07-17T07:12:33Z

Somehow waiting for apache/flink#23003.

I'd prefer to use a consistent relocate pattern.

Also will create a JIRA ticket for this patch.

Hopefully we can merge this one in days.

tisonkun · 2023-07-17T10:25:08Z

It seems changeing the relocate pattern doesn't fix the issue really. I'll try to revert the changes to see if it's an flaky case.

Signed-off-by: tison <wander4096@gmail.com>

…r/pulsar/source/enumerator/cursor/stop/MessageIdStopCursor.java Co-authored-by: Yufan Sheng <syhily@gmail.com>

Signed-off-by: tison <wander4096@gmail.com>

...ava/org/apache/flink/connector/pulsar/source/enumerator/cursor/stop/MessageIdStopCursor.java

This reverts commit f29bda8.

This reverts commit 2cb31bf.

This reverts commit f2a3ad1.

This reverts commit c824082.

Signed-off-by: tison <wander4096@gmail.com>

Thanks to Yufan! Signed-off-by: tison <wander4096@gmail.com> Co-authored-by: Yufan Sheng <yufan@streamnative.io>

Signed-off-by: tison <wander4096@gmail.com>

tisonkun · 2023-07-18T01:40:39Z

...test/java/org/apache/flink/tests/util/pulsar/common/FlinkContainerWithPulsarEnvironment.java

+        configuration.set(JobManagerOptions.TOTAL_PROCESS_MEMORY, MemorySize.ofMebiBytes(2560));
+        configuration.set(JobManagerOptions.JVM_METASPACE, MemorySize.ofMebiBytes(1024));


Temporarily workaround. Hopefully we locate why JM metaspace OOM. It can be related to both the size of the final JAR ball and how we load and unload the JAR ball.

Signed-off-by: tison <wander4096@gmail.com>

tisonkun · 2023-07-18T03:15:12Z

The CNFE is a wanring that -

Previous JM shutdown
Before the next JM started, TM reported CNFE
The new started JM receive CNFE from TM and logs
After JM redistribute execution and jars, TM start to work again

tisonkun · 2023-07-18T03:39:52Z

Pending to merge..

Other improvement will be exploring later.

Signed-off-by: tison <wander4096@gmail.com>

tisonkun · 2023-07-18T07:33:04Z

Merging...

[FLINK-XXXXX] Upgrade Pulsar version to 3.0.0

c17d5d3

Signed-off-by: tison <wander4096@gmail.com>

boring-cyborg bot added the component=Connectors/Pulsar label Jul 12, 2023

tisonkun force-pushed the up-pulsar branch from ff0eb41 to c17d5d3 Compare July 12, 2023 14:59

tisonkun mentioned this pull request Jul 12, 2023

[fix][admin] Add javax.xml.bind:jaxb-api to shade apache/pulsar#20106

Merged

4 tasks

try latest flink-ci-tools

e2cfd50

Signed-off-by: tison <wander4096@gmail.com>

tisonkun mentioned this pull request Jul 14, 2023

[FLINK-32585] Filter javax.xml.bind:jaxb-api false positive for Pulsar connector apache/flink#22989

Merged

compatible with 1.18-SNAPSHOT

58485c7

Signed-off-by: tison <wander4096@gmail.com>

tisonkun requested review from MartijnVisser and zentol July 14, 2023 05:23

tisonkun mentioned this pull request Jul 14, 2023

[fix][client] Shade com.fasterxml.jackson.datatype.* to prevent ClassNotFoundException apache/pulsar#19458

Merged

5 tasks

try to add missing dep

e00a1f2

Signed-off-by: tison <wander4096@gmail.com>

tisonkun force-pushed the up-pulsar branch from dd56bb1 to e00a1f2 Compare July 14, 2023 07:27

eolivelli approved these changes Jul 14, 2023

View reviewed changes

tisonkun marked this pull request as draft July 14, 2023 09:29

tisonkun force-pushed the up-pulsar branch 2 times, most recently from a3f2e9b to 4fecf48 Compare July 16, 2023 05:50

try to relocated connector specific classes

c78689c

Signed-off-by: tison <wander4096@gmail.com>

tisonkun force-pushed the up-pulsar branch from 4fecf48 to c78689c Compare July 16, 2023 06:53

tisonkun marked this pull request as ready for review July 16, 2023 07:42

cbornet approved these changes Jul 16, 2023

View reviewed changes

tisonkun and others added 4 commits July 17, 2023 18:28

revert shade changes

f29bda8

Signed-off-by: tison <wander4096@gmail.com>

Update flink-connector-pulsar/src/main/java/org/apache/flink/connecto…

db8991a

…r/pulsar/source/enumerator/cursor/stop/MessageIdStopCursor.java Co-authored-by: Yufan Sheng <syhily@gmail.com>

fix compilation

0a6f561

Signed-off-by: tison <wander4096@gmail.com>

checkstyle

625a09b

Signed-off-by: tison <wander4096@gmail.com>

BewareMyPower reviewed Jul 17, 2023

View reviewed changes

...ava/org/apache/flink/connector/pulsar/source/enumerator/cursor/stop/MessageIdStopCursor.java Outdated Show resolved Hide resolved

tisonkun mentioned this pull request Jul 17, 2023

[FLINK-32585] Use flink-connector-pulsar shade pattern apache/flink#23003

Closed

Revert "revert shade changes"

2cb31bf

This reverts commit f29bda8.

tisonkun changed the title ~~[FLINK-XXXXX] Upgrade Pulsar version to 3.0.0~~ [FLINK-32612] Upgrade Pulsar version to 3.0.0 Jul 17, 2023

tisonkun added 4 commits July 17, 2023 22:33

Revert "Revert "revert shade changes""

f2a3ad1

This reverts commit 2cb31bf.

Revert "Revert "Revert "revert shade changes"""

c824082

This reverts commit f2a3ad1.

Revert "Revert "Revert "Revert "revert shade changes""""

9ddf2fe

This reverts commit c824082.

try to split context

f8b7f55

Signed-off-by: tison <wander4096@gmail.com>

tisonkun force-pushed the up-pulsar branch from a21a162 to f8b7f55 Compare July 17, 2023 17:33

tisonkun and others added 6 commits July 18, 2023 02:16

try fix loading

c8682b1

Thanks to Yufan! Signed-off-by: tison <wander4096@gmail.com> Co-authored-by: Yufan Sheng <yufan@streamnative.io>

spotless

6d01fda

Signed-off-by: tison <wander4096@gmail.com>

spotless

9d2f2b7

Signed-off-by: tison <wander4096@gmail.com>

spotless

100f103

Signed-off-by: tison <wander4096@gmail.com>

try increase metaspace

51be7ad

Signed-off-by: tison <wander4096@gmail.com>

trigger

176125e

Signed-off-by: tison <wander4096@gmail.com>

tisonkun commented Jul 18, 2023

View reviewed changes

cleanup code

520bf56

Signed-off-by: tison <wander4096@gmail.com>

tisonkun requested a review from syhily July 18, 2023 02:56

syhily approved these changes Jul 18, 2023

View reviewed changes

need not serde message id

96f0b7e

Signed-off-by: tison <wander4096@gmail.com>

tisonkun merged commit 279a678 into apache:main Jul 18, 2023

tisonkun deleted the up-pulsar branch July 18, 2023 07:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-32612] Upgrade Pulsar version to 3.0.0 #54

[FLINK-32612] Upgrade Pulsar version to 3.0.0 #54

tisonkun commented Jul 12, 2023 •

edited

Loading

tisonkun commented Jul 12, 2023

tisonkun commented Jul 12, 2023 •

edited

Loading

tisonkun commented Jul 12, 2023

tisonkun commented Jul 14, 2023 •

edited

Loading

zentol commented Jul 14, 2023

tisonkun commented Jul 14, 2023

eolivelli left a comment

tisonkun commented Jul 14, 2023

tisonkun commented Jul 16, 2023

cbornet commented Jul 16, 2023

tisonkun commented Jul 17, 2023

tisonkun commented Jul 17, 2023

tisonkun Jul 18, 2023

tisonkun commented Jul 18, 2023 •

edited

Loading

tisonkun commented Jul 18, 2023

tisonkun commented Jul 18, 2023

		configuration.set(JobManagerOptions.TOTAL_PROCESS_MEMORY, MemorySize.ofMebiBytes(2560));
		configuration.set(JobManagerOptions.JVM_METASPACE, MemorySize.ofMebiBytes(1024));

[FLINK-32612] Upgrade Pulsar version to 3.0.0 #54

[FLINK-32612] Upgrade Pulsar version to 3.0.0 #54

Conversation

tisonkun commented Jul 12, 2023 • edited Loading

Purpose of the change

Verifying this change

Significant changes

tisonkun commented Jul 12, 2023

tisonkun commented Jul 12, 2023 • edited Loading

tisonkun commented Jul 12, 2023

tisonkun commented Jul 14, 2023 • edited Loading

zentol commented Jul 14, 2023

tisonkun commented Jul 14, 2023

eolivelli left a comment

Choose a reason for hiding this comment

tisonkun commented Jul 14, 2023

tisonkun commented Jul 16, 2023

cbornet commented Jul 16, 2023

tisonkun commented Jul 17, 2023

tisonkun commented Jul 17, 2023

tisonkun Jul 18, 2023

Choose a reason for hiding this comment

tisonkun commented Jul 18, 2023 • edited Loading

tisonkun commented Jul 18, 2023

tisonkun commented Jul 18, 2023

tisonkun commented Jul 12, 2023 •

edited

Loading

tisonkun commented Jul 12, 2023 •

edited

Loading

tisonkun commented Jul 14, 2023 •

edited

Loading

tisonkun commented Jul 18, 2023 •

edited

Loading