[HBASE-26534] Update HBase version to 2.4, make Hadoop 3 and Spark 3 default versions. #88

LucaCanali · 2021-12-02T15:35:53Z

This proposes to bump up the default versions used by HBase-connectors to recent versions, in particular for HBase, Spark, and Hadoop:
HBase is upgraded to 2.4.9, Spark to 3.1.2, Hadoop to 3.2.0.
This proposes also to make Spark 3 and Hadoop 3 default version (as opposed to Spark 2 and Hadoop 2).
Additional minor changes to the code are proposed/needed to compile the connector with HBase 2.4.9.

Apache-HBase · 2021-12-02T15:51:16Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 54s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	spotbugs	0m 0s	spotbugs executables are not available.
+0 🆗	markdownlint	0m 0s	markdownlint was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ master Compile Tests _
+0 🆗	mvndep	0m 23s	Maven dependency ordering for branch
+1 💚	mvninstall	1m 13s	master passed
+1 💚	compile	0m 47s	master passed
+1 💚	javadoc	0m 48s	master passed
+1 💚	scaladoc	0m 0s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 19s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 50s	the patch passed
+1 💚	compile	0m 46s	the patch passed
+1 💚	javac	0m 46s	the patch passed
+1 💚	scalac	0m 46s	the patch passed
-1 ❌	whitespace	0m 0s	The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	javadoc	0m 46s	the patch passed
+1 💚	scaladoc	0m 0s	the patch passed
		_ Other Tests _
+1 💚	unit	7m 12s	root in the patch passed.
		14m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/1/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR	#88
Optional Tests	dupname javac javadoc unit spotbugs findbugs compile xml markdownlint scalac scaladoc
uname	Linux b4309709c03d 5.4.0-1054-aws #57~18.04.1-Ubuntu SMP Thu Jul 15 03:21:36 UTC 2021 x86_64 GNU/Linux
Build tool	hb_maven
Personality	dev-support/jenkins/hbase-personality.sh
git revision	master / `9a839b2`
Default Java	Oracle Corporation-1.8.0_282-b08
whitespace	https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/1/artifact/yetus-precommit-check/output/whitespace-eol.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/1/testReport/
Max. process+thread count	892 (vs. ulimit of 12500)
modules	C: kafka/hbase-kafka-proxy spark spark/hbase-spark . U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/1/console
versions	git=2.20.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-02T18:40:18Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 57s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	spotbugs	0m 0s	spotbugs executables are not available.
+0 🆗	markdownlint	0m 0s	markdownlint was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ master Compile Tests _
+0 🆗	mvndep	0m 23s	Maven dependency ordering for branch
+1 💚	mvninstall	1m 28s	master passed
+1 💚	compile	0m 51s	master passed
+1 💚	javadoc	0m 50s	master passed
+1 💚	scaladoc	0m 0s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 20s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 52s	the patch passed
+1 💚	compile	0m 49s	the patch passed
+1 💚	javac	0m 49s	the patch passed
+1 💚	scalac	0m 49s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	javadoc	0m 48s	the patch passed
+1 💚	scaladoc	0m 0s	the patch passed
		_ Other Tests _
+1 💚	unit	7m 24s	root in the patch passed.
		15m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/2/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR	#88
Optional Tests	dupname javac javadoc unit spotbugs findbugs compile xml markdownlint scalac scaladoc
uname	Linux f4ba024dd4d1 5.4.0-1047-aws #49~18.04.1-Ubuntu SMP Wed Apr 28 23:08:58 UTC 2021 x86_64 GNU/Linux
Build tool	hb_maven
Personality	dev-support/jenkins/hbase-personality.sh
git revision	master / `9a839b2`
Default Java	Oracle Corporation-1.8.0_282-b08
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/2/testReport/
Max. process+thread count	916 (vs. ulimit of 12500)
modules	C: kafka/hbase-kafka-proxy spark spark/hbase-spark . U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/2/console
versions	git=2.20.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

The purpose of this is to match our hbase-server version with the jar provided on EMR. While our HBase is listed as 2.2.6, the library code we have from Amazon is for 2.4.1+. There was a breaking change in 2.4.1 and this mismatch prevents bulkLoadThinRows() from working.

ianawilson · 2021-12-28T22:43:37Z

This is great. I ran into an issue with running this on AWS EMR and which provides HBase 2.4.1+ library code, and this had exactly the updates I needed.

I'd love to see this merged and packaged in a release.

ianawilson · 2021-12-28T22:58:33Z

One thing to think about with this:

HBase 2.4.1 introduced (via this PR apache/hbase#2800) a breaking change for bulkLoadThinRows() by moving some of the static helpers from HStore to StoreUtils (this was precisely the problem that lead me to this PR).

This PR enables you to use this with HBase 2.4.1 and later, but will prevent you from using it with earlier HBase versions. I know you currently allow the HBase version to be configured via the system properties, but as-is this change won't work with the HBase library prior to 2.4.1.

LucaCanali · 2022-01-11T09:29:59Z

Thanks @ianawilson for looking at this.
My thought and proposal with this is that if HBase 2.4.1 could afford a breaking change, that could reflect on the HBase connector too. Indeed this PR this requires HBase client 2.4.x to work. However, this is only for the HBase client version, it does not affect the server version, as many people are still using HBase versions older than 2.4.

ianawilson

@LucaCanali Makes sense to me!

I'm not a contributor to this repo, so I don't know how much weight my approval carries, but I've been using this branch successfully in a couple of ad hoc Spark jobs with HBase client 2.4, and it's been great.

ianawilson · 2022-01-12T17:32:13Z

@LucaCanali FYI, looks like there's a PR with a similar intent and slightly different approach #89

taklwu · 2022-01-12T19:30:29Z

@LucaCanali do you mind merging my changes from #89 to here? I will review yours instead, and that could help to move upgrade the forward .

taklwu

did you run the unit test on your local environment ? may I ask if you're using linux not mac?

basically, I had some changes in TestJavaHBaseContext.java and other *Suite.scala , please see them in my PR #89 , that helps to prevent unit test failure on my mac. but it may be only me having these issues. if you can, please help to merge those changes in unit tests as well and IMO it's helping to run it smooth without conflicts between each tests.

pom.xml

spark/README.md

spark/hbase-spark/pom.xml

spark/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseContext.scala

spark/pom.xml

Apache-HBase · 2022-01-13T10:46:04Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 15s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	spotbugs	0m 1s	spotbugs executables are not available.
+0 🆗	markdownlint	0m 1s	markdownlint was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ master Compile Tests _
+0 🆗	mvndep	0m 23s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 55s	master passed
+1 💚	compile	0m 50s	master passed
+1 💚	javadoc	0m 48s	master passed
+1 💚	scaladoc	0m 0s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 21s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 53s	the patch passed
+1 💚	compile	0m 48s	the patch passed
+1 💚	javac	0m 48s	the patch passed
+1 💚	scalac	0m 48s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	javadoc	0m 48s	the patch passed
+1 💚	scaladoc	0m 0s	the patch passed
		_ Other Tests _
+1 💚	unit	7m 18s	root in the patch passed.
		17m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/3/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR	#88
Optional Tests	dupname javac javadoc unit spotbugs findbugs compile xml markdownlint scalac scaladoc
uname	Linux 1b6d27673a83 5.4.0-1047-aws #49~18.04.1-Ubuntu SMP Wed Apr 28 23:08:58 UTC 2021 x86_64 GNU/Linux
Build tool	hb_maven
Personality	dev-support/jenkins/hbase-personality.sh
git revision	master / `9a839b2`
Default Java	Oracle Corporation-1.8.0_282-b08
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/3/testReport/
Max. process+thread count	917 (vs. ulimit of 12500)
modules	C: kafka/hbase-kafka-proxy spark spark/hbase-spark . U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/3/console
versions	git=2.20.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

LucaCanali · 2022-01-13T10:46:33Z

Thanks @taklwu for comments and review.
I run my tests on Linux and they are all OK, however I have added your proposed changes to TestJavaHBaseContext.java too.

Apache-HBase · 2022-01-13T11:03:23Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 44s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+0 🆗	spotbugs	0m 0s	spotbugs executables are not available.
+0 🆗	markdownlint	0m 0s	markdownlint was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ master Compile Tests _
+0 🆗	mvndep	0m 28s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 47s	master passed
+1 💚	compile	0m 48s	master passed
+1 💚	javadoc	0m 48s	master passed
+1 💚	scaladoc	0m 0s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 48s	the patch passed
+1 💚	compile	0m 47s	the patch passed
+1 💚	javac	0m 47s	the patch passed
+1 💚	scalac	0m 47s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 3s	The patch has no ill-formed XML file.
+1 💚	javadoc	0m 45s	the patch passed
+1 💚	scaladoc	0m 0s	the patch passed
		_ Other Tests _
+1 💚	unit	7m 37s	root in the patch passed.
		16m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/4/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR	#88
Optional Tests	dupname javac javadoc unit spotbugs findbugs compile xml markdownlint scalac scaladoc
uname	Linux 46e8c32c7378 5.4.0-1054-aws #57~18.04.1-Ubuntu SMP Thu Jul 15 03:21:36 UTC 2021 x86_64 GNU/Linux
Build tool	hb_maven
Personality	dev-support/jenkins/hbase-personality.sh
git revision	master / `9a839b2`
Default Java	Oracle Corporation-1.8.0_282-b08
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/4/testReport/
Max. process+thread count	903 (vs. ulimit of 12500)
modules	C: kafka/hbase-kafka-proxy spark spark/hbase-spark . U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/4/console
versions	git=2.20.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

taklwu

LGTM ! and thanks for merging the changes

ianawilson · 2022-01-13T20:02:08Z

This is great, thank you both @LucaCanali @taklwu! One question I've got: Is it possible for this update to be released and published to maven central? It seems like there are quite a few updates on this repository that haven't been released there yet.

Update HBase version to 2.4.8, make Hadoop 3, and Spark 3 default

3c77c87

Fix extrawhitespaces in README.md

cecb422

ianawilson mentioned this pull request Dec 29, 2021

HBase 2.4 sproutsocial/hbase-connectors#10

Closed

ianawilson approved these changes Jan 12, 2022

View reviewed changes

taklwu reviewed Jan 12, 2022

View reviewed changes

taklwu requested review from petersomogyi and meszibalu January 12, 2022 20:06

LucaCanali changed the title ~~[HBASE-26534] Update HBase version to 2.4.8, make Hadoop 3 and Spark 3 default versions.~~ [HBASE-26534] Update HBase version to 2.4, make Hadoop 3 and Spark 3 default versions. Jan 13, 2022

LucaCanali added 2 commits January 13, 2022 11:27

Review comments

07f860c

Modified test following review

ae28445

taklwu approved these changes Jan 13, 2022

View reviewed changes

taklwu merged commit 7578a33 into apache:master Jan 13, 2022

taklwu mentioned this pull request Jan 13, 2022

HBASE-24356 Move hbase-connectors to hbase-thirdparty-4.0.1 #89

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HBASE-26534] Update HBase version to 2.4, make Hadoop 3 and Spark 3 default versions. #88

[HBASE-26534] Update HBase version to 2.4, make Hadoop 3 and Spark 3 default versions. #88

LucaCanali commented Dec 2, 2021 •

edited

Loading

Apache-HBase commented Dec 2, 2021

Apache-HBase commented Dec 2, 2021

ianawilson commented Dec 28, 2021

ianawilson commented Dec 28, 2021

LucaCanali commented Jan 11, 2022

ianawilson left a comment

ianawilson commented Jan 12, 2022

taklwu commented Jan 12, 2022 •

edited

Loading

taklwu left a comment

Apache-HBase commented Jan 13, 2022

LucaCanali commented Jan 13, 2022

Apache-HBase commented Jan 13, 2022

taklwu left a comment

ianawilson commented Jan 13, 2022

[HBASE-26534] Update HBase version to 2.4, make Hadoop 3 and Spark 3 default versions. #88

[HBASE-26534] Update HBase version to 2.4, make Hadoop 3 and Spark 3 default versions. #88

Conversation

LucaCanali commented Dec 2, 2021 • edited Loading

Apache-HBase commented Dec 2, 2021

Apache-HBase commented Dec 2, 2021

ianawilson commented Dec 28, 2021

ianawilson commented Dec 28, 2021

LucaCanali commented Jan 11, 2022

ianawilson left a comment

Choose a reason for hiding this comment

ianawilson commented Jan 12, 2022

taklwu commented Jan 12, 2022 • edited Loading

taklwu left a comment

Choose a reason for hiding this comment

Apache-HBase commented Jan 13, 2022

LucaCanali commented Jan 13, 2022

Apache-HBase commented Jan 13, 2022

taklwu left a comment

Choose a reason for hiding this comment

ianawilson commented Jan 13, 2022

LucaCanali commented Dec 2, 2021 •

edited

Loading

taklwu commented Jan 12, 2022 •

edited

Loading