feat: Add Hive 4.0.0

[sbernauer]

Description

Part of stackabletech/issues#620

During this is noticed a minor problem with schemaTool.sh and created apache/hive#5419 (already merged :) )

We did not bump misc dependencies as Hive 3.1.3 is doing (see https://stackable-workspace.slack.com/archives/C031A5BEFS7/p1725275085248519 for details) to keep this within the defined time box. We can always do that later on based on a CVE scan.

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Tasks

Preview Give feedback

1. Changes are OpenShift compatible
2. All added packages (via microdnf or otherwise) have a comment on why they are added
3. Things not downloaded from Red Hat repositories should be mirrored in the Stackable repository and downloaded from there
4. All packages should have (if available) signatures/hashes verified
5. Add an entry to the CHANGELOG.md file
6. Integration tests ran successfully

TIP: Running integration tests with a new product image

The image can be built and uploaded to the kind cluster with the following commands:

bake --product <product> --image-version <stackable-image-version>
kind load docker-image <image-tagged-with-the-major-version> --name=<name-of-your-test-cluster>

See the output of bake to retrieve the image tag for <image-tagged-with-the-major-version>.

[sbernauer]

In terms of patches:
01-HIVE-26905.patch => This is just a backport of HIVE-25173, which is fixed in 4.0.0
02-HIVE-21939.patch => Fixed in 4.0.0
03-HIVE-26522.patch => This is just a backport of HIVE-22033, which is fixed in 4.0.0
04-HIVE-26743.patch => This is just a backport of HIVE-24694, which is fixed in 4.0.0
05-HIVE-26882.patch => Fixed in 4.0.0
06-HIVE-27508.patch => This is just a backport of HIVE-21584, which is fixed in 4.0.0

07-patch-updates.patch => Later :)
08-logging-dependencies.patch
09-maven-warning.patch => Does not seem to be needed
10-postgres-driver.patch

[maltesander]

Built the image locally and used operator stackabletech/hive-operator#508 . Integrationtests all work fine.

I tried the e2e demo which also worked (data in superset etc), but the Hive pod throws connection errors continuously like

2024-09-02T07:49:41,289 ERROR [Metastore-Handler-Pool: Thread-34] server.TThreadPoolServer: Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: org.apache.thrift.transport.TTransportException: Socket is closed by peer.
	at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:729) ~[hive-standalone-metastore-common-4.0.0.jar:4.0.0]
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:227) ~[libthrift-0.16.0.jar:0.16.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: org.apache.thrift.transport.TTransportException: Socket is closed by peer.
	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:184) ~[libthrift-0.16.0.jar:0.16.0]
	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) ~[libthrift-0.16.0.jar:0.16.0]
	at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:151) ~[libthrift-0.16.0.jar:0.16.0]
	at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:108) ~[libthrift-0.16.0.jar:0.16.0]
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:238) ~[libthrift-0.16.0.jar:0.16.0]
	at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:44) ~[libthrift-0.16.0.jar:0.16.0]
	at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:199) ~[libthrift-0.16.0.jar:0.16.0]
	at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:711) ~[hive-standalone-metastore-common-4.0.0.jar:4.0.0]
	at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:707) ~[hive-standalone-metastore-common-4.0.0.jar:4.0.0]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
	at javax.security.auth.Subject.doAs(Subject.java:361) ~[?:?]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876) ~[hadoop-common-3.3.6.jar:?]
	at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:707) ~[hive-standalone-metastore-common-4.0.0.jar:4.0.0]
	... 4 more

Full logs hive.log

Edit: K8s healtch check (kerberos)...

[sbernauer]

Awesome thanks!

Just for the record: This is a know problem of the k8s healthcheck not using Kerberos

[maltesander]

LGTM!. Integration tests and e2e demo works. We will merge as is(?) and probably mark this as experimental first, check the CVE scans for the new image and eventually to dependency bumps?

[sbernauer]

We will merge as is(?) and probably mark this as experimental first, check the CVE scans for the new image and eventually to dependency bumps?

Yes! :)

[maltesander]

We will merge as is(?) and probably mark this as experimental first, check the CVE scans for the new image and eventually to dependency bumps?

Yes! :)

Yeah that should have been "as discussed" and not a question :D

[maltesander]

LGTM!