HBASE-26907 Update Hadoop3 versions for JEP 223 compliance #4305

ndimiduk · 2022-03-31T16:48:01Z

No description provided.

ndimiduk · 2022-03-31T16:51:36Z

See investigation and discussion on the JIRA.

Maybe we want to drop 2.3 from the matrix in the book?

Our personality makes no distinction in Hadoop3 versions for JDK8 vs. JDK11+. The latter requires 3.2+. We probably should fix that.

ndimiduk · 2022-03-31T18:32:22Z

Well that's interesting. From hadoop-3.3.2:

[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4305/yetus-general-check/src/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java:[832,29] package org.apache.htrace.core does not exist
[INFO] 1 error
[INFO] -------------------------------------------------------------

busbey · 2022-03-31T18:45:45Z

we have our hbase-thirdparty no-op stand-in for htrace. We can add a profile for including that when building against some Hadoops?

Apache9

Just a simple nit.

And on the compatibile matrix, IIRC we have a consensus that we will only have ref guide on master branch so it should contain all the release lines instead of only non-EOL release lines. So let's still keep the 2.3.x release line. Let me find the thread on mailing list and try to move forward on this topic.

dev-support/hbase-personality.sh

joshelser

LGTM (agree with Duo's suggestion to bump to the latest 3.3.x)

ndimiduk · 2022-04-13T15:30:52Z

And on the compatibile matrix, IIRC we have a consensus that we will only have ref guide on master branch so it should contain all the release lines instead of only non-EOL release lines. So let's still keep the 2.3.x release line. Let me find the thread on mailing list and try to move forward on this topic.

@Apache9 okay ; I'll take your word for it, I can restore the 2.3 entries. Obviously we don't have all released mentioned, so at what point do we drop them off?

ndimiduk · 2022-04-13T20:54:36Z

I opened HBASE-26954 for the Hadoop 3.3.2 compilation issue.

ndimiduk · 2022-04-13T21:07:47Z

I've bumped our Hadoop 3.2 dependency to 3.2.3, per @Apache9 's suggestion. I've retained the HBase 2.3 rows in the compatibility matrix. I've stepped back our promises about Hadoop 3.3.2, given HBASE-26954.

ndimiduk · 2022-04-13T21:08:15Z

Let me see which branches are impacted by HBASE-26954.

ndimiduk · 2022-04-13T21:25:25Z

HBASE-26954 seems to only affect master branch.

Apache9 · 2022-04-14T02:18:29Z

And on the compatibile matrix, IIRC we have a consensus that we will only have ref guide on master branch so it should contain all the release lines instead of only non-EOL release lines. So let's still keep the 2.3.x release line. Let me find the thread on mailing list and try to move forward on this topic.

@Apache9 okay ; I'll take your word for it, I can restore the 2.3 entries. Obviously we don't have all released mentioned, so at what point do we drop them off?

This has not been well discussed yet, I mean when to drop the EOL versions. For me, I prefer we split the table to two pieces, for EOL versions, after some time(half a year or a whole year?), we move it to the 'archived' pieces, which means we will not update them any more.

Apache9 · 2022-05-22T02:50:52Z

Any updates here?

ndimiduk · 2022-05-22T09:42:24Z

Progress is reported over on the linked ticket where I describe the test failure. There's a ThreadGroup object that goes out of scope during DataNode operation on these newer Hadoop versions. I think that the explicitly managed ThreadGoup serves no purpose, but I'm still looking for a commit that changed the behavior.

I believe is indicates a real and serious bug in the DataNode, which will cause it to become unresponsive after an idle period.

ndimiduk · 2022-08-15T12:49:47Z

I think @saintstack fixed the issue i was seeing in HDFS-16586 / apache/hadoop#4338 . Let me see how widely that patch has propagated.

Apache9 · 2022-08-15T12:58:43Z

Ah, I've replied on the jira, in HBASE-27148 we have already moved up the hadoop 3.2.3.

I also noticed the problem described in HDFS-16586 but I'm afraid we can not only support the hadoop versions which have this patch since it is only a test issue. I added a piece of workaround code in HBTU so we can still use the versions which do not have HDFS-16586 in.

hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtil.java

Line 579 in 9215066

private static final class FsDatasetAsyncDiskServiceFixer extends Thread {

So I think we could close this PR now?

ndimiduk · 2022-08-15T13:07:04Z

Rebased and squashed current patched onto latest master.

ndimiduk · 2022-08-15T13:09:59Z

Ah, I've replied on the jira, in HBASE-27148 we have already moved up the hadoop 3.2.3.

@Apache9 Ah okay, I missed HBASE-27148.

ndimiduk · 2022-08-15T13:12:03Z

It looks like we still have hadoop-3.1.2 in the quick-check list for branch-2.4, I guess we can bump that version. Also, it appears that I have added a column regarding branch-2.5 in this patch.

Let me push an updated based the knowledge of HBASE-27148.

Apache9 · 2022-08-15T13:17:26Z

It looks like we still have hadoop-3.1.2 in the quick-check list for branch-2.4, I guess we can bump that version. Also, it appears that I have added a column regarding branch-2.5 in this patch.

Let me push an updated based the knowledge of HBASE-27148.

Sounds good.

And please see #4692 , where I plan to change the hadoop check versions for our active branches, which also implies what hadoop versions we still wantto support. I think for 2.4.x, we should not remove the support of a hadoop version, but for 2.5+, we can drop the EOL versions and also the versions which have critical CVEs.

- drop 3.1.x - ensure 3.2 >= 3.2.3 - add 3.3.3, 3.3.4

Apache-HBase · 2022-08-15T13:21:56Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/console in case of problems.

Apache-HBase · 2022-08-15T13:22:02Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/console in case of problems.

Apache-HBase · 2022-08-15T13:22:12Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/console in case of problems.

Apache-HBase · 2022-08-15T13:22:49Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 42s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
		_ Patch Compile Tests _
		_ Other Tests _
		1m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4305
Optional Tests
uname	Linux de954c16e7c7 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `9215066`
Max. process+thread count	29 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-08-15T13:23:14Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 57s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
		_ Patch Compile Tests _
		_ Other Tests _
		2m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4305
Optional Tests
uname	Linux 312e578bb58a 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `9215066`
Max. process+thread count	37 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ndimiduk · 2022-08-15T13:25:41Z

Hmm okay. @Apache9 how about I close HBASE-26907 this PR and let you resolve them on HBASE-27294. Do you might grabbing the update to the book once that ticket is resolved? Thanks.

Apache9 · 2022-08-15T13:37:24Z

Hmm okay. @Apache9 how about I close HBASE-26907 this PR and let you resolve them on HBASE-27294. Do you might grabbing the update to the book once that ticket is resolved? Thanks.

For the book update let's open a separated issue? As part of the branch-2.5 release process.

Apache-HBase · 2022-08-15T13:40:00Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 39s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	shelldocs	0m 0s	Shelldocs was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 46s	master passed
+0 🆗	refguide	3m 22s	branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚	spotless	1m 6s	branch has no errors when running spotless:check.
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 21s	the patch passed
+1 💚	shellcheck	0m 1s	There were no new shellcheck issues.
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+0 🆗	refguide	2m 46s	patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚	spotless	0m 50s	patch has no errors when running spotless:check.
		_ Other Tests _
+1 💚	asflicense	0m 13s	The patch does not generate ASF License warnings.
		18m 43s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4305
Optional Tests	dupname asflicense spotless shellcheck shelldocs refguide
uname	Linux 695c599fd0f6 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `9215066`
refguide	https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4305/4/yetus-general-check/output/branch-site/book.html
refguide	https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4305/4/yetus-general-check/output/patch-site/book.html
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/4/console
versions	git=2.17.1 maven=3.6.3 shellcheck=0.4.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2022-08-15T13:40:36Z

Ah, maybe I misunderstood your comment. You also mean we open a new issue to land the book update? That's fine by me. I could help implementing the PR or reviewing the PR, no problem. Thanks.

Apache-HBase · 2022-08-15T13:45:08Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/console in case of problems.

Apache-HBase · 2022-08-15T13:45:12Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/console in case of problems.

Apache-HBase · 2022-08-15T13:45:29Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/console in case of problems.

Apache-HBase · 2022-08-15T13:46:01Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 41s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
		_ Patch Compile Tests _
		_ Other Tests _
		1m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4305
Optional Tests
uname	Linux 24c3d085eac6 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `9215066`
Max. process+thread count	29 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-08-15T13:46:05Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 42s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
		_ Patch Compile Tests _
		_ Other Tests _
		1m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4305
Optional Tests
uname	Linux 538fb0037040 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `9215066`
Max. process+thread count	39 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-08-15T14:02:37Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 29s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+0 🆗	shelldocs	0m 1s	Shelldocs was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 17s	master passed
+0 🆗	refguide	3m 6s	branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚	spotless	1m 14s	branch has no errors when running spotless:check.
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 12s	the patch passed
+1 💚	shellcheck	0m 1s	There were no new shellcheck issues.
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+0 🆗	refguide	2m 54s	patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚	spotless	1m 1s	patch has no errors when running spotless:check.
		_ Other Tests _
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		17m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4305
Optional Tests	dupname asflicense spotless shellcheck shelldocs refguide
uname	Linux 51327528fcf9 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `9215066`
refguide	https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4305/5/yetus-general-check/output/branch-site/book.html
refguide	https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4305/5/yetus-general-check/output/patch-site/book.html
Max. process+thread count	70 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4305/5/console
versions	git=2.17.1 maven=3.6.3 shellcheck=0.4.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ndimiduk · 2022-08-16T12:26:06Z

Superceded HBASE-27148, HBASE-27294.

ndimiduk requested review from busbey, joshelser, apurtell and Apache9 March 31, 2022 16:48

This comment was marked as outdated.

Sign in to view

apurtell approved these changes Apr 2, 2022

View reviewed changes

Apache9 reviewed Apr 4, 2022

View reviewed changes

dev-support/hbase-personality.sh Outdated Show resolved Hide resolved

joshelser approved these changes Apr 4, 2022

View reviewed changes

Apache9 mentioned this pull request Apr 12, 2022

Bump hadoop-common from 3.1.2 to 3.1.4 #4330

Closed

ndimiduk force-pushed the 26907-hadoop3-JEP223-compliance branch from 8a71d7c to 5f8345d Compare April 13, 2022 21:06

This comment was marked as outdated.

Sign in to view

Apache9 approved these changes Apr 14, 2022

View reviewed changes

This comment was marked as outdated.

Sign in to view

ndimiduk mentioned this pull request Apr 28, 2022

YETUS-1060. github status should use htmlreport apache/yetus#261

Merged

HBASE-26907 Update Hadoop3 versions for JEP 223 compliance

a215a0a

ndimiduk force-pushed the 26907-hadoop3-JEP223-compliance branch from 0b9fb7d to a215a0a Compare August 15, 2022 13:06

update after HBASE-27148

16cca4b

- drop 3.1.x - ensure 3.2 >= 3.2.3 - add 3.3.3, 3.3.4

ndimiduk closed this Aug 16, 2022

ndimiduk deleted the 26907-hadoop3-JEP223-compliance branch August 16, 2022 12:26

HBASE-26907 Update Hadoop3 versions for JEP 223 compliance #4305

HBASE-26907 Update Hadoop3 versions for JEP 223 compliance #4305

Conversation

ndimiduk commented Mar 31, 2022

ndimiduk commented Mar 31, 2022

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

ndimiduk commented Mar 31, 2022

busbey commented Mar 31, 2022

This comment was marked as outdated.

This comment was marked as outdated.

Apache9 left a comment

Choose a reason for hiding this comment

joshelser left a comment

Choose a reason for hiding this comment

ndimiduk commented Apr 13, 2022

ndimiduk commented Apr 13, 2022

ndimiduk commented Apr 13, 2022

ndimiduk commented Apr 13, 2022

ndimiduk commented Apr 13, 2022

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Apache9 commented Apr 14, 2022

This comment was marked as outdated.

Apache9 commented May 22, 2022

ndimiduk commented May 22, 2022

ndimiduk commented Aug 15, 2022

Apache9 commented Aug 15, 2022

ndimiduk commented Aug 15, 2022

ndimiduk commented Aug 15, 2022

ndimiduk commented Aug 15, 2022

Apache9 commented Aug 15, 2022

Apache-HBase commented Aug 15, 2022

Apache-HBase commented Aug 15, 2022

Apache-HBase commented Aug 15, 2022

Apache-HBase commented Aug 15, 2022

Apache-HBase commented Aug 15, 2022

ndimiduk commented Aug 15, 2022

Apache9 commented Aug 15, 2022

Apache-HBase commented Aug 15, 2022

Apache9 commented Aug 15, 2022

Apache-HBase commented Aug 15, 2022

Apache-HBase commented Aug 15, 2022

Apache-HBase commented Aug 15, 2022

Apache-HBase commented Aug 15, 2022

Apache-HBase commented Aug 15, 2022

Apache-HBase commented Aug 15, 2022

ndimiduk commented Aug 16, 2022