Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update ZooKeeper to 3.6.2 and Curator to 5.1.0 #8590

Merged
merged 29 commits into from
Feb 11, 2021
Merged

Update ZooKeeper to 3.6.2 and Curator to 5.1.0 #8590

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
add680a
Update ZooKeeper to 3.6.2 and Curator to 5.1.0
eolivelli Nov 17, 2020
47cbe33
port from #8348
eolivelli Nov 17, 2020
b33b9ba
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Jan 14, 2021
60835b1
Exclude shaded ZooKeeper from spotbugs
eolivelli Jan 14, 2021
e30bf32
fix spotbugs
eolivelli Jan 14, 2021
d01491f
fix licenses and tests
eolivelli Jan 14, 2021
fe2f2db
fix more tests
eolivelli Jan 14, 2021
9e62f7c
More test deps
eolivelli Jan 15, 2021
5440699
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Jan 18, 2021
d887b9e
Upgrade Curator in Flume connector
eolivelli Jan 19, 2021
39143a7
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Jan 20, 2021
a90fe83
Fix refletion usage of ZK internals
eolivelli Jan 20, 2021
c8f571d
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Jan 21, 2021
624441f
Make AdminApiOffloadTest less flaky
eolivelli Jan 21, 2021
58a5dff
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Jan 22, 2021
a7220ec
Fix MockZooKeeper
eolivelli Jan 22, 2021
c3a9c2d
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Jan 25, 2021
d604fc3
Merge branch 'master' into fix/zk-3.6
eolivelli Jan 29, 2021
40e5275
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Feb 1, 2021
dffbfb4
remove line
eolivelli Feb 1, 2021
930f84d
Fixed use of multi() in MockZookeeper
merlimat Feb 2, 2021
c2015a8
More ZK dependency fixes
merlimat Feb 2, 2021
bf3e784
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Feb 2, 2021
822b1dc
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Feb 5, 2021
8b2c460
Merge branch 'master' into fix/zk-3.6
eolivelli Feb 8, 2021
957ea1e
Merge branch 'master' into fix/zk-3.6
eolivelli Feb 9, 2021
10487e0
Fix license check
eolivelli Feb 9, 2021
379dac7
Merge branch 'master' into fix/zk-3.6
eolivelli Feb 10, 2021
a300cac
revert typo
eolivelli Feb 10, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion managed-ledger/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,16 @@
<classifier>tests</classifier>
<scope>test</scope>
</dependency>

<dependency>
<groupId>io.dropwizard.metrics</groupId>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain why do we need this? I failed to see this dependency was used here. The same question apply to the other dependency.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because from zk 3.6 onwards the "zookeeper" Maven artifact does not import those dependencies anymore to the dependant projects.
This is because we (ZooKeeper project community) want to say that this artifact is only the "zookeeper java client" and the zookeeper java client does not need such jars.
But we still do not provide a "zookeeper server" jar

In Pulsar we are also running the ZooKeeper server (both for production and for tests) and so we need to explicitly add those two jar to the build, otherwise the ZooKeeper server won't start

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to leave a comment with the reason we're adding these, otherwise down the road it will be difficult to remember the rationale.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sijie PTAL the description was added on the new deps

<artifactId>metrics-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<scope>test</scope>
</dependency>
<!--
junit is a runtime dependency of zookeeper tests, so only
add this dependency here.
Expand Down
33 changes: 31 additions & 2 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,10 @@ flexible messaging model and an intuitive client API.</description>
<commons-compress.version>1.19</commons-compress.version>

<bookkeeper.version>4.12.1</bookkeeper.version>
<zookeeper.version>3.5.7</zookeeper.version>
<zookeeper.version>3.6.2</zookeeper.version>
<snappy.version>1.1.7</snappy.version> <!-- ZooKeeper server -->
<dropwizardmetrics.version>3.2.5</dropwizardmetrics.version> <!-- ZooKeeper server -->
<curator.version>5.1.0</curator.version>
<netty.version>4.1.51.Final</netty.version>
<netty-tc-native.version>2.0.33.Final</netty-tc-native.version>
<jetty.version>9.4.35.v20201120</jetty.version>
Expand Down Expand Up @@ -296,7 +299,21 @@ flexible messaging model and an intuitive client API.</description>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper-jute</artifactId>
<version>${zookeeper.version}</version>
</dependency>
<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-core</artifactId>
<version>${dropwizardmetrics.version}</version>
</dependency>
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<version>${snappy.version}</version>
</dependency>
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
Expand Down Expand Up @@ -342,6 +359,18 @@ flexible messaging model and an intuitive client API.</description>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-recipes</artifactId>
<version>${curator.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.zookeeper</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.bookkeeper</groupId>
Expand Down
1 change: 0 additions & 1 deletion pulsar-testclient/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,6 @@
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>${zookeeper.version}</version>
</dependency>

<dependency>
Expand Down
9 changes: 9 additions & 0 deletions pulsar-zookeeper-utils/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,17 @@
<groupId>io.grpc</groupId>
<artifactId>grpc-all</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.curator</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-recipes</artifactId>
</dependency>

<!-- `grpc-all` is excluded from `stream-storage-server` at root pom file -->
<dependency>
Expand Down
2 changes: 1 addition & 1 deletion ...ar-zookeeper-utils/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ private void runZookeeper(int maxCC) throws IOException {
serverFactory.configure(new InetSocketAddress(zkPort), maxCC);
serverFactory.startup(zks);

zkDataCleanupManager = new DatadirCleanupManager(zkDataDir, zkDataDir, 0, 1 /* hour */);
zkDataCleanupManager = new DatadirCleanupManager(zkDataDir, zkDataDir, 3, 1 /* hour */);
zkDataCleanupManager.start();
} catch (Exception e) {
LOG.error("Exception while instantiating ZooKeeper", e);
Expand Down
18 changes: 18 additions & 0 deletions pulsar-zookeeper/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,20 @@
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</dependency>

<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper-jute</artifactId>
</dependency>

<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-core</artifactId>
</dependency>
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
</dependency>

<dependency>
<groupId>${project.groupId}</groupId>
Expand Down Expand Up @@ -93,6 +107,10 @@
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</weaveDependency>
<weaveDependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper-jute</artifactId>
</weaveDependency>
</weaveDependencies>
</configuration>
<executions>
Expand Down