KAFKA-14206: upgrade zookeeper version to 3.7.1

[showuon]

Upgrade zookeeper version to the latest stable release 3.7.1 due to some CVEs.

Apache ZooKeeper 3.8.0 is our current release, and 3.7.1 our latest stable release.

https://zookeeper.apache.org/releases.html

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[showuon]

@jsancio , should we include this upgrade into 3.3?

[ijuma]

What CVEs are this? This kind of upgrade requires quite a lot of validation, it's too late for 3.3 unless the impact is severe.

[showuon]

ZK 3.6.3 uses old Netty that has CVEs. ZK 3.7.1 upgrade the Netty version to fix it. I agree it's late for 3.3, just want to make sure we're aware of it.
CVE-2021-37136
CVE-2021-37137

[divijvaidya]

I did some analysis on what has changed and here is my summary:

ZK 3.7.1 contains CVE fixes for:

1. Jackson-databind: https://nvd.nist.gov/vuln/detail/CVE-2020-36518
2. Log4j 1.x: CVE-2022-23302/5/7: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-23302
3. Jetty: https://nvd.nist.gov/vuln/detail/cve-2021-28165
4. Reload4j: CVE-2020-9493, CVE-2022-23307

In 3.3.0-RC1 for Kafka:

1. We are picking up 4.1.78 for Netty for two sub-modules of Netty, netty-transport-native-epoll and netty-handler. The reported CVEs in Netty's other sub modules are either related to compression algorithms or in HTTP2 which ZooKeeper (or Kafka) doesn't use AFAIK. Hence, we should be ok.
2. We are picking up Jetty Server 9.4.48 which fixes the vulnerabilities fixed by new Zookeeper version.
3. We are picking up Jackson 2.13.3 which fixes the vulnerabilities fixed by new Zookeeper version.
4. We are picking up Reload4j 1.2.19 which fixes the vulnerabilities fixed by new Zookeeper version.

Since the CVEs are fixed in the versions we are directly picking the class path for Kafka, I don't think it is urgent to upgrade the zookeeper version. We can scope it for 3.4.0.

[ijuma]

Exactly right @divijvaidya, we have generally fixed transitive dependency issues directly.

[ijuma]

One more thing: ZK is going away in AK 4.0. We probably need to do one more upgrade before that. It may make sense to wait a bit and go straight to 3.8.1 (once that's released) so that we benefit from longer support from the ZK community.

[showuon]

Make sense to me. Thank you all. I'll close this PR then.

[51n15t9r]

@ijuma , @showuon - Sorry to bring this up in a closed thread.

Other than the vulnerabilities, zookeeper 3.6 is also EOL since December 2022.
Would it be useful to spend the efforts for Zookeeper 3.7.1 upgrades in the meantime?

[divijvaidya]

It may make sense to wait a bit and go straight to 3.8.1 (once that's released

Note that Zk 3.8.1 has released in Jan 2023. @ijuma, do you think it is the right time for us to move to 3.8.1 since the existing version in Kafka is EOL?

[1] https://zookeeper.apache.org/doc/r3.8.1/releasenotes.html

[ijuma]

@divijvaidya Yes, I think that would make sense. That should tide us over until the KRaft transition happens.

[ijuma]

A couple of things to consider:

1. If we upgrade the zk server to 3.8.1, what is the impact on the zk clients. That is, what's the earliest zk client version that is supported by the 3.8.x server?
2. We need to ensure there are no regressions (particularly on the stability front) when it comes to this upgrade. It would be good for someone to stress test the system a bit with the new version and check if all works well.

[divijvaidya]

Completing the conversation here, in case someone comes around reading this old thread. We have a new KIP and a PR for Zk migration to 3.8.1 which would hopefully land in 3.5.

KIP - https://cwiki.apache.org/confluence/display/KAFKA/KIP-902%3A+Upgrade+Zookeeper+to+3.8.1

PR - #13260