HBASE-27872 xerial's snappy-java requires GLIBC >= 2.32 #5245

apurtell · 2023-05-18T23:52:51Z

We need to add a native library load check with a helpful error message if xerial snappy fails to initialize due to a too old glibc or similar reason, and disable the unit test if the native library fails to load.

apurtell · 2023-05-18T23:57:37Z

After the proposed changes:

2023-05-18T16:49:25,730 ERROR [Time-limited test {}] xerial.SnappyCodec(58): The Snappy native libraries could not be loaded
java.lang.UnsatisfiedLinkError: /home/apurtell/src/hbase/hbase-compression/hbase-compression-snappy/target/snappy-1.1.9-1a39c2aa-088d-44df-a86b-7bc7d8682d47-libsnappyjava.so:
/lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /home/apurtell/src/hbase/hbase-compression/hbase-compression-snappy/target/snappy-1.1.9-1a39c2aa-088d-44df-a86b-7bc7d8682d47-libsnappyjava.so)
    at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_332]
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950) ~[?:1.8.0_332]
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832) ~[?:1.8.0_332]
    at java.lang.Runtime.load0(Runtime.java:811) ~[?:1.8.0_332]
    at java.lang.System.load(System.java:1088) ~[?:1.8.0_332]
    at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:180) ~[snappy-java-1.1.9.1.jar:1.1.9.1]
    at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:155) ~[snappy-java-1.1.9.1.jar:1.1.9.1]
    at org.xerial.snappy.Snappy.init(Snappy.java:70) ~[snappy-java-1.1.9.1.jar:1.1.9.1]
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:47) ~[snappy-java-1.1.9.1.jar:1.1.9.1]
    at org.apache.hadoop.hbase.io.compress.xerial.SnappyCodec.<clinit>(SnappyCodec.java:55) ~[classes/:?]
    at org.apache.hadoop.hbase.io.compress.xerial.TestWALCompressionSnappy.setUpBeforeClass(TestWALCompressionSnappy.java:53) ~[test-classes/:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_332]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_332]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_332]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_332]
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) ~[junit-4.13.2.jar:4.13.2]
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.13.2.jar:4.13.2]
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) ~[junit-4.13.2.jar:4.13.2]
    at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33) ~[junit-4.13.2.jar:4.13.2]
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) ~[junit-4.13.2.jar:4.13.2]
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) ~[junit-4.13.2.jar:4.13.2]
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) ~[junit-4.13.2.jar:4.13.2]
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) ~[junit-4.13.2.jar:4.13.2]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_332]
    at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_332]                                                                                                                         
2023-05-18T16:49:25,736 WARN  [Time-limited test {}] xerial.TestWALCompressionSnappy(54): Snappy codec cannot be loaded. Test will not execute.

… failed to load

Apache-HBase · 2023-05-19T00:23:17Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 58s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 5s	master passed
+1 💚	compile	0m 13s	master passed
+1 💚	shadedjars	4m 23s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 12s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 49s	the patch passed
+1 💚	compile	0m 13s	the patch passed
+1 💚	javac	0m 13s	the patch passed
+1 💚	shadedjars	4m 22s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 11s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 29s	hbase-compression-snappy in the patch passed.
		17m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5245/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5245
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 05b0d36f7096 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `71d7996`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5245/1/testReport/
Max. process+thread count	241 (vs. ulimit of 30000)
modules	C: hbase-compression/hbase-compression-snappy U: hbase-compression/hbase-compression-snappy
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5245/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-05-19T00:24:41Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 24s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 43s	master passed
+1 💚	compile	0m 11s	master passed
+1 💚	shadedjars	4m 48s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 12s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 25s	the patch passed
+1 💚	compile	0m 11s	the patch passed
+1 💚	javac	0m 11s	the patch passed
+1 💚	shadedjars	4m 43s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 10s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 28s	hbase-compression-snappy in the patch passed.
		19m 18s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5245/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5245
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 7117cb35d475 5.4.0-1101-aws #109~18.04.1-Ubuntu SMP Mon Apr 24 20:40:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `71d7996`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5245/1/testReport/
Max. process+thread count	261 (vs. ulimit of 30000)
modules	C: hbase-compression/hbase-compression-snappy U: hbase-compression/hbase-compression-snappy
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5245/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-05-19T00:36:27Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 17s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 33s	master passed
+1 💚	compile	0m 17s	master passed
+1 💚	checkstyle	0m 10s	master passed
+1 💚	spotless	0m 43s	branch has no errors when running spotless:check.
+1 💚	spotbugs	0m 22s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 23s	the patch passed
+1 💚	compile	0m 15s	the patch passed
+1 💚	javac	0m 15s	the patch passed
+1 💚	checkstyle	0m 8s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	12m 23s	Patch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	0m 26s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 10s	The patch does not generate ASF License warnings.
		31m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5245/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5245
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 9526a7cbc398 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `71d7996`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	84 (vs. ulimit of 30000)
modules	C: hbase-compression/hbase-compression-snappy U: hbase-compression/hbase-compression-snappy
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5245/1/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

...ppy/src/test/java/org/apache/hadoop/hbase/io/compress/xerial/TestHFileCompressionSnappy.java

Apache-HBase · 2023-05-19T22:08:11Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 27s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 15s	master passed
+1 💚	compile	0m 11s	master passed
+1 💚	shadedjars	4m 41s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 11s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 51s	the patch passed
+1 💚	compile	0m 10s	the patch passed
+1 💚	javac	0m 10s	the patch passed
+1 💚	shadedjars	4m 39s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 9s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 29s	hbase-compression-snappy in the patch passed.
		17m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5245/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5245
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux a6c79b7f6fee 5.4.0-1099-aws #107~18.04.1-Ubuntu SMP Fri Mar 17 16:49:05 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `71d7996`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5245/2/testReport/
Max. process+thread count	242 (vs. ulimit of 30000)
modules	C: hbase-compression/hbase-compression-snappy U: hbase-compression/hbase-compression-snappy
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5245/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-05-19T22:09:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 25s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 49s	master passed
+1 💚	compile	0m 11s	master passed
+1 💚	shadedjars	4m 44s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 12s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 39s	the patch passed
+1 💚	compile	0m 11s	the patch passed
+1 💚	javac	0m 11s	the patch passed
+1 💚	shadedjars	4m 55s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 10s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 29s	hbase-compression-snappy in the patch passed.
		19m 36s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5245/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5245
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 6e2002e6b171 5.4.0-1101-aws #109~18.04.1-Ubuntu SMP Mon Apr 24 20:40:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `71d7996`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5245/2/testReport/
Max. process+thread count	253 (vs. ulimit of 30000)
modules	C: hbase-compression/hbase-compression-snappy U: hbase-compression/hbase-compression-snappy
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5245/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-05-19T22:20:59Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 44s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 39s	master passed
+1 💚	compile	0m 17s	master passed
+1 💚	checkstyle	0m 9s	master passed
+1 💚	spotless	0m 42s	branch has no errors when running spotless:check.
+1 💚	spotbugs	0m 21s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 22s	the patch passed
+1 💚	compile	0m 16s	the patch passed
+1 💚	javac	0m 16s	the patch passed
+1 💚	checkstyle	0m 9s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	12m 25s	Patch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	0m 25s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 11s	The patch does not generate ASF License warnings.
		30m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5245/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5245
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux f82964fb906c 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `71d7996`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	83 (vs. ulimit of 30000)
modules	C: hbase-compression/hbase-compression-snappy U: hbase-compression/hbase-compression-snappy
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5245/2/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

We need to add a native library load check with a helpful error message if xerial snappy fails to initialize due to a too old glibc or similar reason, and disable the unit test if the native library fails to load. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org>

HBASE-27872 xerial's snappy-java requires GLIBC >= 2.32

b09062f

We need to add a native library load check with a helpful error message if xerial snappy fails to initialize due to a too old glibc or similar reason, and disable the unit test if the native library fails to load.

apurtell requested review from sunhelly and virajjasani May 18, 2023 23:52

Also prevent the instantiation of SnappyCodec if the native libraries…

de5c6ae

… failed to load

Apache9 reviewed May 19, 2023

View reviewed changes

...ppy/src/test/java/org/apache/hadoop/hbase/io/compress/xerial/TestHFileCompressionSnappy.java Outdated Show resolved Hide resolved

Address review feedback

c9e05c7

Apache9 approved these changes May 20, 2023

View reviewed changes

virajjasani approved these changes May 22, 2023

View reviewed changes

apurtell merged commit f347867 into apache:master May 26, 2023

apurtell deleted the HBASE-27872 branch May 26, 2023 00:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-27872 xerial's snappy-java requires GLIBC >= 2.32 #5245

HBASE-27872 xerial's snappy-java requires GLIBC >= 2.32 #5245

apurtell commented May 18, 2023

apurtell commented May 18, 2023

Apache-HBase commented May 19, 2023

Apache-HBase commented May 19, 2023

Apache-HBase commented May 19, 2023

Apache-HBase commented May 19, 2023

Apache-HBase commented May 19, 2023

Apache-HBase commented May 19, 2023

HBASE-27872 xerial's snappy-java requires GLIBC >= 2.32 #5245

HBASE-27872 xerial's snappy-java requires GLIBC >= 2.32 #5245

Conversation

apurtell commented May 18, 2023

apurtell commented May 18, 2023

Apache-HBase commented May 19, 2023

Apache-HBase commented May 19, 2023

Apache-HBase commented May 19, 2023

Apache-HBase commented May 19, 2023

Apache-HBase commented May 19, 2023

Apache-HBase commented May 19, 2023