HIVE-28191. Upgrade Hadoop Version to 3.4.0. #5187

slfan1989 · 2024-04-10T00:23:37Z

What changes were proposed in this pull request?

JIRA: HIVE-28191. Upgrade Hadoop Version to 3.4.0.

The 3.4.0 version of Hadoop has been released, and we aim for this version to be compatible with other communities. Hive is a crucial component among them. We will attempt to upgrade Hive's Hadoop dependency to 3.4.0.

Why are the changes needed?

HADOOP-17524 removed EventCount to facilitate the removal of dependencies on log4j 1.x. I referenced HADOOP-17524 and removed a portion of the code accordingly. We can refer to this link apache/hadoop#2909 to view the changes.

Does this PR introduce any user-facing change?

no

Is the change a dependency upgrade?

Upgrade Hadoop Version to 3.4.0.

How was this patch tested?

I tried compiling the code locally and it passed, expecting it to pass CI verification.

[INFO] Reactor Summary for Hive 4.1.0-SNAPSHOT:
[INFO] 
[INFO] Hive Storage API ................................... SUCCESS [  7.493 s]
[INFO] Hive ............................................... SUCCESS [  1.748 s]
[INFO] Hive Classifications ............................... SUCCESS [  0.356 s]
[INFO] Hive Shims Common .................................. SUCCESS [  2.004 s]
[INFO] Hive Shims 0.23 .................................... SUCCESS [  3.960 s]
[INFO] Hive Shims ......................................... SUCCESS [  1.301 s]
[INFO] Hive Standalone Metastore .......................... SUCCESS [  2.636 s]
[INFO] Hive Standalone Metastore Common Code .............. SUCCESS [ 26.648 s]
[INFO] Hive Common ........................................ SUCCESS [  7.782 s]
[INFO] Hive Service RPC ................................... SUCCESS [  1.406 s]
[INFO] Hive Serde ......................................... SUCCESS [  4.780 s]
[INFO] Hive Metastore ..................................... SUCCESS [  1.872 s]
[INFO] Hive Vector-Code-Gen Utilities ..................... SUCCESS [  0.189 s]
[INFO] Hive Parser ........................................ SUCCESS [  7.348 s]
[INFO] Hive UDF ........................................... SUCCESS [  1.538 s]
[INFO] Hive Llap Common ................................... SUCCESS [  2.677 s]
[INFO] Hive Llap Client ................................... SUCCESS [  2.632 s]
[INFO] Hive Llap Tez ...................................... SUCCESS [  2.597 s]
[INFO] Hive Metastore Server .............................. SUCCESS [ 17.629 s]
[INFO] Hive Query Language ................................ SUCCESS [02:55 min]
[INFO] Hive TestUtils ..................................... SUCCESS [  0.236 s]
[INFO] Hive Llap Server ................................... SUCCESS [  7.339 s]
[INFO] Hive HPL/SQL ....................................... SUCCESS [  3.640 s]
[INFO] Hive Service ....................................... SUCCESS [  6.208 s]
[INFO] Hive Accumulo Handler .............................. SUCCESS [  4.117 s]
[INFO] Hive JDBC .......................................... SUCCESS [ 21.490 s]
[INFO] Hive Beeline ....................................... SUCCESS [ 17.430 s]
[INFO] Hive CLI ........................................... SUCCESS [  4.026 s]
[INFO] Hive Contrib ....................................... SUCCESS [  2.230 s]
[INFO] Hive Druid Handler ................................. SUCCESS [01:15 min]
[INFO] Hive HBase Handler ................................. SUCCESS [  8.839 s]
[INFO] Hive JDBC Handler .................................. SUCCESS [  3.081 s]
[INFO] Hive HCatalog ...................................... SUCCESS [  0.772 s]
[INFO] Hive HCatalog Core ................................. SUCCESS [  4.615 s]
[INFO] Hive HCatalog Pig Adapter .......................... SUCCESS [  3.298 s]
[INFO] Hive HCatalog Server Extensions .................... SUCCESS [  2.796 s]
[INFO] Hive HCatalog Webhcat Java Client .................. SUCCESS [  3.283 s]
[INFO] Hive HCatalog Webhcat .............................. SUCCESS [  9.194 s]
[INFO] Hive Streaming ..................................... SUCCESS [  3.258 s]
[INFO] Hive Llap External Client .......................... SUCCESS [  2.198 s]
[INFO] Hive Shims Aggregator .............................. SUCCESS [  0.096 s]
[INFO] Hive Kudu Handler .................................. SUCCESS [  5.407 s]
[INFO] Hive Kafka Storage Handler ......................... SUCCESS [  4.510 s]
[INFO] Hive Packaging ..................................... SUCCESS [  01:12 h]
[INFO] Hive Metastore Tools ............................... SUCCESS [  0.057 s]
[INFO] Hive Metastore Tools common libraries .............. SUCCESS [  6.403 s]
[INFO] Hive metastore benchmarks .......................... SUCCESS [ 18.470 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:20 h
[INFO] Finished at: 2024-04-09T20:05:37+08:00
[INFO] ------------------------------------------------------------------------

sonarqubecloud · 2024-05-04T15:36:28Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

slfan1989 · 2024-05-05T04:37:33Z

I have a question. I'm trying to run unit tests locally, but I keep encountering errors like Module 'hive-standalone-metastore-common' production: java.lang.OutOfMemoryError: GC overhead limit exceeded. Is there a good way to solve this issue? My IDEA memory is already set to 12GB.

@ayushtkn Do you have any good solutions to solve this issue?

zhangbutao · 2024-05-23T14:33:07Z

http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-5187/5/tests
I find that hbase-handler related tests are failed. I think the failures are caused by the incompatibility between hbase and hadoop3.4.0. Check HIVE-27651 which i have done about the hbase-hadoop incompatibility. This needs to make sure hbase itTest can run in Hadoop3.4.0 and maybe need hbase community to provide a compatible package with hadoop3.4.0.

ava.io.IOException: Shutting down
	at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:257)
	at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:110)
	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1185)
	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1232)
	at org.apache.hadoop.hive.hbase.HBaseTestSetup.setUpFixtures(HBaseTestSetup.java:112)
	at org.apache.hadoop.hive.hbase.HBaseTestSetup.preTest(HBaseTestSetup.java:78)
	at org.apache.hadoop.hive.ql.QTestMiniClusters.setup(QTestMiniClusters.java:240)
	at org.apache.hadoop.hive.ql.QTestUtil.<init>(QTestUtil.java:221)
	at org.apache.hadoop.hive.hbase.HBaseQTestUtil.<init>(HBaseQTestUtil.java:46)
	at org.apache.hadoop.hive.cli.control.CoreHBaseCliDriver.beforeClass(CoreHBaseCliDriver.java:49)
	at org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
Caused by: java.lang.RuntimeException: Master not initialized after 200000ms
	at org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:221)
	at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:190)
	at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:407)
	at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:250)
	... 21 more

slfan1989 · 2024-05-23T21:44:16Z

http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-5187/5/tests
I find that hbase-handler related tests are failed. I think the failures are caused by the incompatibility between hbase and hadoop3.4.0. Check HIVE-27651 which i have done about the hbase-hadoop incompatibility. This needs to make sure hbase itTest can run in Hadoop3.4.0 and maybe need hbase community to provide a compatible package with hadoop3.4.0.

Thank you very much for the explanation! It is very important. I will try to make Hbase compatible with Hadoop 3.4.0.

github-actions · 2024-07-23T00:22:08Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.

linghengqian

I noticed the currently closed PR from Add GraalVM Reachability Metadata and corresponding nativeTest for Iceberg table in HiveServer2 shardingsphere#31526 side. But to be honest, isn’t https://issues.apache.org/jira/browse/HBASE-28714 related to Hadoop 3.4.0 integration on HBase side already closed? And apache/hbase:2.5.10 has been released, see https://github.com/apache/hbase/releases/tag/rel%2F2.5.10 .

zhangbutao · 2024-07-30T04:22:48Z

I noticed the currently closed PR from Add GraalVM Reachability Metadata and corresponding nativeTest for HiveServer2 shardingsphere#31526 side. But to be honest, isn’t https://issues.apache.org/jira/browse/HBASE-28714 related to Hadoop 3.4.0 integration on HBase side already closed? And apache/hbase:2.5.10 has been released, see https://github.com/apache/hbase/releases/tag/rel%2F2.5.10 .

@linghengqian Hive master branch uses Hbase2.5.6 :

hive/pom.xml

Line 146 in c26342e

<hbase.version>2.5.6-hadoop3</hbase.version>

Maybe upgrading to Hbase:2.5.10 can help to upgrade Hadoop version? @slfan1989

sonarqubecloud · 2024-07-30T05:31:12Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-09-29T00:28:05Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.

maxgio92 · 2024-10-01T14:03:44Z

Hi, I've just landed here after trying to bump Hadoop to 3.4.0 in Hive, and I ended up with the same exact patch of this PR, being able to build and make tests pass.

Do you think there's anything else missing to reach fully compatibility that is not yet covered by tests?

Thank you.

linghengqian · 2024-10-01T14:08:48Z

Hi, I've just landed here after trying to bump Hadoop to 3.4.0 in Hive, and I ended up with the same exact patch of this PR, being able to build and make tests pass.

Do you think there's anything else missing to reach fully compatibility that is not yet covered by tests?

Thank you.

Since the original author of this PR has not responded, perhaps someone needs to create a new PR containing a new version of the HBase Client. The original issue https://issues.apache.org/jira/browse/HIVE-28191 has not yet been directly claimed.
The main reason for needing to update the Hbase Client is failed integration tests. Disclaimer, I am not actually a Hive Committer, so I cannot directly update the content of the current PR.

linghengqian · 2024-10-04T04:36:58Z

Hi, I've just landed here after trying to bump Hadoop to 3.4.0 in Hive, and I ended up with the same exact patch of this PR, being able to build and make tests pass.
Do you think there's anything else missing to reach fully compatibility that is not yet covered by tests?
Thank you.

Since the original author of this PR has not responded, perhaps someone needs to create a new PR containing a new version of the HBase Client. The original issue https://issues.apache.org/jira/browse/HIVE-28191 has not yet been directly claimed.

The main reason for needing to update the Hbase Client is failed integration tests. Disclaimer, I am not actually a Hive Committer, so I cannot directly update the content of the current PR.

I have opened https://issues.apache.org/jira/browse/HIVE-28552 to discuss the unit test bug for the master branch.

slfan1989 · 2024-10-10T09:02:28Z

@zhangbutao @maxgio92 @linghengqian Sorry for the late reply. I think we can work together to solve this issue. Since Hadoop 3.4.1 is about to be released, I hope we can integrate 3.4.0 into Hive first. Let’s discuss it further in #5481.

HIVE-28191. Upgrade Hadoop Version to 3.4.0.

28c59ab

asf-ci-hive added tests pending tests unstable and removed tests pending labels Apr 10, 2024

slfan1989 marked this pull request as draft April 10, 2024 05:10

Merge branch 'apache:master' into HIVE-28191

1fb7cab

asf-ci-hive added tests pending tests unstable and removed tests unstable tests pending labels May 1, 2024

Merge branch 'apache:master' into HIVE-28191

a4332f8

asf-ci-hive added tests pending tests unstable and removed tests unstable tests pending labels May 2, 2024

HIVE-28191. Upgrade Hadoop Version to 3.4.0.

62581c8

asf-ci-hive added tests pending tests failed and removed tests unstable tests pending labels May 3, 2024

HIVE-28191. Fix CheckStyle.

971461b

github-actions bot requested a review from miklosgergely May 4, 2024 14:42

asf-ci-hive added tests pending and removed tests failed labels May 4, 2024

asf-ci-hive added tests unstable and removed tests pending labels May 4, 2024

linghengqian mentioned this pull request Jul 22, 2024

How to use shardingsphere-proxy to proxy the hive db apache/shardingsphere#32223

Closed

github-actions bot added the stale label Jul 23, 2024

github-actions bot closed this Jul 30, 2024

linghengqian reviewed Jul 30, 2024

View reviewed changes

zhangbutao reopened this Jul 30, 2024

asf-ci-hive added tests pending and removed tests unstable labels Jul 30, 2024

linghengqian mentioned this pull request Jul 30, 2024

Add GraalVM Reachability Metadata and corresponding nativeTest for Iceberg table in HiveServer2 apache/shardingsphere#31526

Merged

6 tasks

asf-ci-hive added tests unstable and removed tests pending labels Jul 30, 2024

github-actions bot removed the stale label Jul 31, 2024

linghengqian mentioned this pull request Jul 31, 2024

Hive Connection Error with ZooKeeper dbeaver/dbeaver#22777

Closed

github-actions bot added the stale label Sep 29, 2024

maxgio92 mentioned this pull request Oct 1, 2024

feat(hive): add apache hive package wolfi-dev/os#29412

Closed

3 tasks

github-actions bot removed the stale label Oct 2, 2024

linghengqian mentioned this pull request Oct 3, 2024

HIVE-28191: Upgrade Hadoop and Hbase Client version #5481

Closed

slfan1989 closed this Oct 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HIVE-28191. Upgrade Hadoop Version to 3.4.0. #5187

HIVE-28191. Upgrade Hadoop Version to 3.4.0. #5187

slfan1989 commented Apr 10, 2024 •

edited

Loading

sonarqubecloud bot commented May 4, 2024

slfan1989 commented May 5, 2024

zhangbutao commented May 23, 2024

slfan1989 commented May 23, 2024

github-actions bot commented Jul 23, 2024

linghengqian left a comment

zhangbutao commented Jul 30, 2024

sonarqubecloud bot commented Jul 30, 2024

github-actions bot commented Sep 29, 2024

maxgio92 commented Oct 1, 2024

linghengqian commented Oct 1, 2024

linghengqian commented Oct 4, 2024

slfan1989 commented Oct 10, 2024 •

edited

Loading

HIVE-28191. Upgrade Hadoop Version to 3.4.0. #5187

HIVE-28191. Upgrade Hadoop Version to 3.4.0. #5187

Conversation

slfan1989 commented Apr 10, 2024 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

Is the change a dependency upgrade?

How was this patch tested?

sonarqubecloud bot commented May 4, 2024

Quality Gate passed

slfan1989 commented May 5, 2024

zhangbutao commented May 23, 2024

slfan1989 commented May 23, 2024

github-actions bot commented Jul 23, 2024

linghengqian left a comment

Choose a reason for hiding this comment

zhangbutao commented Jul 30, 2024

sonarqubecloud bot commented Jul 30, 2024

Quality Gate passed

github-actions bot commented Sep 29, 2024

maxgio92 commented Oct 1, 2024

linghengqian commented Oct 1, 2024

linghengqian commented Oct 4, 2024

slfan1989 commented Oct 10, 2024 • edited Loading

slfan1989 commented Apr 10, 2024 •

edited

Loading

slfan1989 commented Oct 10, 2024 •

edited

Loading