HBASE-27746 Check if the file system supports storage policy before invoking setStoragePolicy() #5189

jojochuang · 2023-04-19T01:19:17Z

Simply check and ignore silently if the underlying FS does not support storage policy.

the hasPathCapability() API is Hadoop 3.2 and above. So do not cherrypick this change to HBase 2.x or lower.

…nvoking setStoragePolicy() Change-Id: I1fcc90ec23f3b23a6876f094436320d9eb8480ee

Apache-HBase · 2023-04-19T02:09:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 24s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 3s	master passed
+1 💚	compile	0m 34s	master passed
+1 💚	checkstyle	0m 15s	master passed
+1 💚	spotless	0m 47s	branch has no errors when running spotless:check.
+1 💚	spotbugs	0m 33s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 37s	the patch passed
+1 💚	compile	0m 32s	the patch passed
+1 💚	javac	0m 32s	the patch passed
-0 ⚠️	checkstyle	0m 12s	hbase-common: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	14m 16s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
-1 ❌	spotless	0m 13s	patch has 30 errors when running spotless:check, run spotless:apply to fix.
+1 💚	spotbugs	0m 37s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 9s	The patch does not generate ASF License warnings.
		34m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5189/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5189
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 078dc94c8c37 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `94a8f31`
Default Java	Eclipse Adoptium-11.0.17+8
checkstyle	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5189/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-common.txt
spotless	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5189/1/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-common U: hbase-common
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5189/1/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-04-19T05:40:28Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 39s	master passed
+1 💚	compile	0m 17s	master passed
+1 💚	shadedjars	4m 56s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 18s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 7s	the patch passed
+1 💚	compile	0m 21s	the patch passed
+1 💚	javac	0m 21s	the patch passed
+1 💚	shadedjars	4m 56s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 16s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 35s	hbase-common in the patch passed.
-1 ❌	unit	217m 27s	hbase-server in the patch failed.
		245m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5189/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5189
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux f9ac1086ae6b 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `94a8f31`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5189/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5189/1/testReport/
Max. process+thread count	2539 (vs. ulimit of 30000)
modules	C: hbase-common U: hbase-common
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5189/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2023-04-19T14:00:15Z

Please run mvn spotless to fix the style issue?

Change-Id: I7cab20073e24b97dd6b3090723f30941febd40b4

Change-Id: I4639f791febc29d7f89ee5905e043b5dcb1071dc

jojochuang · 2023-04-19T21:28:36Z

Thanks for the tips! Updated checkstyle and also fixed the test.

jojochuang · 2023-04-19T21:29:18Z

hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java

    FileSystem testFs = new AlwaysFailSetStoragePolicyFileSystem();
+    testFs.initialize(new URI("hdfs://localhost/"), conf);


This line is required otherwise uri is null pointer and will fail hasPathCapability() API.

Can we not use the 'hdfs' scheme here? It is not HDFS, actually...

Apache-HBase · 2023-04-19T22:11:43Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 57s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 8s	Maven dependency ordering for branch
+1 💚	mvninstall	3m 22s	master passed
+1 💚	compile	2m 56s	master passed
+1 💚	checkstyle	0m 51s	master passed
+1 💚	spotless	0m 44s	branch has no errors when running spotless:check.
+1 💚	spotbugs	2m 0s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 10s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 18s	the patch passed
+1 💚	compile	2m 53s	the patch passed
+1 💚	javac	2m 53s	the patch passed
+1 💚	checkstyle	0m 48s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	12m 18s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 14s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 20s	The patch does not generate ASF License warnings.
		41m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5189/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5189
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux bf57ef62419f 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `398c5ef`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	85 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5189/2/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-04-20T01:19:21Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 24s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for branch
+1 💚	mvninstall	3m 38s	master passed
+1 💚	compile	0m 56s	master passed
+1 💚	shadedjars	4m 38s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 36s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 15s	the patch passed
+1 💚	compile	0m 57s	the patch passed
+1 💚	javac	0m 57s	the patch passed
+1 💚	shadedjars	4m 39s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 34s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 24s	hbase-common in the patch passed.
+1 💚	unit	201m 53s	hbase-server in the patch passed.
		228m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5189/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5189
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 099ffbccbdc0 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `398c5ef`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5189/2/testReport/
Max. process+thread count	2335 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5189/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-04-20T01:24:46Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 21s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 59s	master passed
+1 💚	compile	0m 57s	master passed
+1 💚	shadedjars	4m 19s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 39s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 47s	the patch passed
+1 💚	compile	0m 56s	the patch passed
+1 💚	javac	0m 56s	the patch passed
+1 💚	shadedjars	4m 17s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 38s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 56s	hbase-common in the patch passed.
-1 ❌	unit	209m 26s	hbase-server in the patch failed.
		234m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5189/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5189
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux cd015a95cec3 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `398c5ef`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5189/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5189/2/testReport/
Max. process+thread count	2635 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5189/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

jojochuang · 2023-04-20T17:07:56Z

The failed test doesn't look related. @taklwu can you help review this one?

Apache9 · 2023-04-23T14:22:30Z

The only concern is about the scheme we use when testing, all others are OK.

Apache9 · 2023-06-16T15:25:34Z

Any updates here @jojochuang ?

jojochuang · 2023-06-16T16:55:14Z

Well actually the test would fail if it's not "hdfs"

[ERROR] TestFSUtils.testSetStoragePolicyDefault:412->verifyNoHDFSApiInvocationForDefaultPolicy:422 » IllegalArgument Invalid URI for NameNode address (check fs.defaultFS): failfs://localhost/ is not of scheme 'hdfs'.

jojochuang · 2024-08-20T21:20:50Z

Looks like this one got missed out. I'll merge it now.

NihalJain · 2024-08-21T14:22:11Z

Hi @jojochuang we should put this in branch-3 as well, right? Do you plan to backport?

jojochuang · 2024-08-21T19:00:45Z

Yes thanks for reminder. I'll take this up.

…nvoking setStoragePolicy() (apache#5189) (cherry picked from commit 0646151)

…nvoking setStoragePolicy() (apache#5189) (cherry picked from commit 0646151) (cherry picked from commit 35b11eb)

…nvoking setStoragePolicy() (apache#5189) (cherry picked from commit 0646151) (cherry picked from commit 35b11eb) (cherry picked from commit d93c0c0)

…nvoking setStoragePolicy() (#5189) (#6170) (cherry picked from commit 0646151)

HBASE-27746 Check if the file system supports storage policy before i…

71b32bc

…nvoking setStoragePolicy() Change-Id: I1fcc90ec23f3b23a6876f094436320d9eb8480ee

jojochuang added 2 commits April 19, 2023 13:04

Checkstyle

bf7c689

Change-Id: I7cab20073e24b97dd6b3090723f30941febd40b4

Fix test

00bc6a1

Change-Id: I4639f791febc29d7f89ee5905e043b5dcb1071dc

jojochuang commented Apr 19, 2023

View reviewed changes

jojochuang requested a review from taklwu April 20, 2023 17:07

Apache9 approved these changes Apr 23, 2023

View reviewed changes

sunhelly approved these changes Apr 24, 2023

View reviewed changes

jojochuang merged commit 0646151 into apache:master Aug 20, 2024

jojochuang added a commit to jojochuang/hbase that referenced this pull request Aug 21, 2024

HBASE-27746 Check if the file system supports storage policy before i…

35b11eb

…nvoking setStoragePolicy() (apache#5189) (cherry picked from commit 0646151)

jojochuang added a commit to jojochuang/hbase that referenced this pull request Aug 21, 2024

HBASE-27746 Check if the file system supports storage policy before i…

d93c0c0

…nvoking setStoragePolicy() (apache#5189) (cherry picked from commit 0646151) (cherry picked from commit 35b11eb)

jojochuang added a commit that referenced this pull request Aug 22, 2024

HBASE-27746 Check if the file system supports storage policy before i…

4e2842e

…nvoking setStoragePolicy() (#5189) (#6170) (cherry picked from commit 0646151)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-27746 Check if the file system supports storage policy before invoking setStoragePolicy() #5189

HBASE-27746 Check if the file system supports storage policy before invoking setStoragePolicy() #5189

jojochuang commented Apr 19, 2023 •

edited

Loading

Apache-HBase commented Apr 19, 2023

Apache-HBase commented Apr 19, 2023

Apache9 commented Apr 19, 2023

jojochuang commented Apr 19, 2023

jojochuang Apr 19, 2023

Apache9 Apr 21, 2023

Apache-HBase commented Apr 19, 2023

Apache-HBase commented Apr 20, 2023

Apache-HBase commented Apr 20, 2023

jojochuang commented Apr 20, 2023

Apache9 commented Apr 23, 2023

Apache9 commented Jun 16, 2023

jojochuang commented Jun 16, 2023

jojochuang commented Aug 20, 2024

NihalJain commented Aug 21, 2024

jojochuang commented Aug 21, 2024

		FileSystem testFs = new AlwaysFailSetStoragePolicyFileSystem();
		testFs.initialize(new URI("hdfs://localhost/"), conf);

HBASE-27746 Check if the file system supports storage policy before invoking setStoragePolicy() #5189

HBASE-27746 Check if the file system supports storage policy before invoking setStoragePolicy() #5189

Conversation

jojochuang commented Apr 19, 2023 • edited Loading

Apache-HBase commented Apr 19, 2023

Apache-HBase commented Apr 19, 2023

Apache9 commented Apr 19, 2023

jojochuang commented Apr 19, 2023

jojochuang Apr 19, 2023

Choose a reason for hiding this comment

Apache9 Apr 21, 2023

Choose a reason for hiding this comment

Apache-HBase commented Apr 19, 2023

Apache-HBase commented Apr 20, 2023

Apache-HBase commented Apr 20, 2023

jojochuang commented Apr 20, 2023

Apache9 commented Apr 23, 2023

Apache9 commented Jun 16, 2023

jojochuang commented Jun 16, 2023

jojochuang commented Aug 20, 2024

NihalJain commented Aug 21, 2024

jojochuang commented Aug 21, 2024

jojochuang commented Apr 19, 2023 •

edited

Loading