Skip to content
This repository has been archived by the owner on Mar 3, 2023. It is now read-only.

Update Bookkeeper to 4.13.0 and Zookeeper to 3.6.3 #3692

Merged
merged 5 commits into from
May 24, 2021

Conversation

nicknezis
Copy link
Contributor

@nicknezis nicknezis commented May 16, 2021

Update Bookkeeper to 4.13.0 and Zookeeper to 3.6.3.

In testing, the Bookkeeper shaded jar was missing necessary libraries, because of this, I moved from using the shaded jar to the standard dependencies. Our Bazel logic should resolve the transitive dependencies, but we still need to make sure the transitives are listed or exported on any java_library or java_binary tasks that use code from the transitive.

Also in updating to newer Zookeeper, their build system has changed to leverage Maven. The Jute IDL generates code. This used to be run by Ant scripts, but now is Maven. Because some of our images are older, the provided version of Maven was too old to run properly. I was able to use the older Maven to build a newer Maven binary which did then work. But I was not able to successfully integrate with our Bazel build system. Instead I generated the code from the IDL which should be platform agnostic. The tradeoff being that any new updates to Zookeeper will need an update to the generated source. If the Zookeeper team provides a package that provides the C library in an easily compiled process, we can transition to that. But for now this seemed like a good compromise.

The update to Zookeeper needed a bump to Apache Curator from 2.9.0 to 5.1.0. Curator is a library that assists with some of the more mundane and complex Zookeeper logic.

Netty was also updated from 4.1.22 to 4.1.50.

Also in the course of doing this update, I had trouble with the jarjar sonatype plugin. I switched to the Bazel rules provided by johnynek/bazel_jar_jar.

git_repository(
    name = "com_github_johnynek_bazel_jar_jar",
    commit = "171f268569384c57c19474b04aebe574d85fde0d", # Latest commit SHA as at 2019/02/13
    remote = "git://github.com/johnynek/bazel_jar_jar.git",
    shallow_since = "1594234634 -1000",
)

@nicknezis nicknezis changed the title Update to Bookkeeper and Zookeeper Update Bookkeeper to 4.13.0 and Zookeeper to 3.6.3 May 16, 2021
@nicknezis nicknezis self-assigned this May 16, 2021
@nicknezis nicknezis marked this pull request as ready for review May 16, 2021 18:07
@nicknezis nicknezis marked this pull request as draft May 16, 2021 18:26
@nicknezis
Copy link
Contributor Author

Converting to draft because I want to do some additional testing in Kubernetes. Will report back soon.

@nicknezis
Copy link
Contributor Author

This was the error and resulting fix I found that explains why I had to update Guava library.
https://stackoverflow.com/questions/45247193/nosuchmethoderror-com-google-common-base-preconditions-checkargumentzljava-lan

@nicknezis nicknezis marked this pull request as ready for review May 18, 2021 03:53
@nicknezis
Copy link
Contributor Author

I was able to successfully submit a topology to a Helm installed Heron cluster running in Kubernetes. So the DLog upload/download should be fine.


git_repository(
name = "com_github_johnynek_bazel_jar_jar",
commit = "171f268569384c57c19474b04aebe574d85fde0d", # Latest commit SHA as at 2019/02/13
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the comment with the date. +1 to you

@nwangtw
Copy link
Contributor

nwangtw commented May 19, 2021

This is great. Thanks!

@nicknezis nicknezis merged commit 22148db into master May 24, 2021
@nicknezis nicknezis deleted the nicknezis/bookkeeper-update-2 branch May 24, 2021 00:21
@nicknezis
Copy link
Contributor Author

Adding this for historical context. This merge request was created to resolve this Bookkeeper issue: apache/bookkeeper#2302

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants