-
Notifications
You must be signed in to change notification settings - Fork 58
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FLINK-32612] Upgrade Pulsar version to 3.0.0 #54
Changes from 20 commits
c17d5d3
e2cfd50
58485c7
e00a1f2
c78689c
f29bda8
db8991a
0a6f561
625a09b
2cb31bf
f2a3ad1
c824082
9ddf2fe
f8b7f55
c8682b1
6d01fda
9d2f2b7
100f103
51be7ad
176125e
520bf56
96f0b7e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
/* | ||
* Licensed to the Apache Software Foundation (ASF) under one | ||
* or more contributor license agreements. See the NOTICE file | ||
* distributed with this work for additional information | ||
* regarding copyright ownership. The ASF licenses this file | ||
* to you under the Apache License, Version 2.0 (the | ||
* "License"); you may not use this file except in compliance | ||
* with the License. You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package org.apache.flink.tests.util.pulsar.common; | ||
|
||
import org.apache.flink.connector.pulsar.testutils.PulsarTestContext; | ||
import org.apache.flink.connector.pulsar.testutils.PulsarTestContextFactory; | ||
import org.apache.flink.connector.pulsar.testutils.PulsarTestEnvironment; | ||
|
||
import java.util.function.Function; | ||
|
||
/** | ||
* This class is used for creating the Pulsar test context which will running in the Flink | ||
* containers. | ||
*/ | ||
public class PulsarContainerTestContextFactory<F, T extends PulsarTestContext<F>> | ||
extends PulsarTestContextFactory<F, T> { | ||
|
||
public PulsarContainerTestContextFactory( | ||
PulsarTestEnvironment environment, Function<PulsarTestEnvironment, T> contextFactory) { | ||
super(environment, contextFactory); | ||
} | ||
|
||
@Override | ||
public T createExternalContext(String testName) { | ||
T context = super.createExternalContext(testName); | ||
context.inContainer(); | ||
return context; | ||
} | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -18,9 +18,11 @@ | |
|
||
package org.apache.flink.connector.pulsar.sink.writer; | ||
|
||
import org.apache.flink.api.common.JobID; | ||
import org.apache.flink.api.common.operators.MailboxExecutor; | ||
import org.apache.flink.api.common.operators.ProcessingTimeService; | ||
import org.apache.flink.api.common.serialization.SerializationSchema; | ||
import org.apache.flink.api.common.typeutils.TypeSerializer; | ||
import org.apache.flink.api.connector.sink2.Sink.InitContext; | ||
import org.apache.flink.api.connector.sink2.SinkWriter; | ||
import org.apache.flink.configuration.Configuration; | ||
|
@@ -205,6 +207,25 @@ public int getAttemptNumber() { | |
return 0; | ||
} | ||
|
||
// The following three methods are for compatibility with | ||
// https://github.com/apache/flink/commit/4f5b2fb5736f5a1c098a7dc1d448a879f36f801b | ||
// . Removed the commented out `@Override` when we move to 1.18. | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we can just add the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I remember that There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That's true. But the master branch should always focus on the latest snapshot version of Flink. |
||
// @Override | ||
public boolean isObjectReuseEnabled() { | ||
return false; | ||
} | ||
|
||
// @Override | ||
public <IN> TypeSerializer<IN> createInputSerializer() { | ||
return null; | ||
} | ||
|
||
// @Override | ||
public JobID getJobId() { | ||
return null; | ||
} | ||
|
||
@Override | ||
public SinkWriterMetricGroup metricGroup() { | ||
return metricGroup; | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -52,7 +52,8 @@ under the License. | |
|
||
<properties> | ||
<flink.version>1.17.0</flink.version> | ||
<pulsar.version>2.11.0</pulsar.version> | ||
<flink-ci-tools.version>1.18-SNAPSHOT</flink-ci-tools.version> | ||
<pulsar.version>3.0.0</pulsar.version> | ||
<bouncycastle.version>1.69</bouncycastle.version> | ||
|
||
<jsr305.version>1.3.9</jsr305.version> | ||
|
@@ -457,7 +458,7 @@ under the License. | |
<dependency> | ||
<groupId>org.apache.flink</groupId> | ||
<artifactId>flink-ci-tools</artifactId> | ||
<version>${flink.version}</version> | ||
<version>${flink-ci-tools.version}</version> | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we can just change the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let me think of it. If we accept this change, #54 (comment) can be applied also. But before we cut a release, we should always pin a non-snapshot version. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The downside is that main branch and PRs may be suddenly broken due to SNAPSHOT changes, but it should not happen and we sooner or later need to fix it. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
In my opinion, we should checkout a 4.0 branch for the 1.7 Flink. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There is already one https://github.com/apache/flink-connector-pulsar/tree/v4.0. Do you mean we move to 1.18 on master and cut a new release when 1.18 release and for 1.18+ only? Although, these breaking changes existing in test code only so it should be still compatible with 1.17. But I'd prefer to do this in another round of PR. This PR focus on upgrading Pulsar version and these changes are workaround for the current failing tests. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I see. Bump the master branch with Flink 1.17 and Pulsar 3.0 support. Then we merge back to 4.0 branch and change the master branch to Flink 1.18 snapshort. Am I right? |
||
</dependency> | ||
</dependencies> | ||
</plugin> | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Temporarily workaround. Hopefully we locate why JM metaspace OOM. It can be related to both the size of the final JAR ball and how we load and unload the JAR ball.