HBASE-26718 HFileArchiver can remove referenced StoreFiles from the archive #4274

d-c-manning · 2022-03-25T06:35:39Z

Of particular note:

The tests as they exist today are not validating a rename into archive, but instead validating a rename from archive into the timestamped backup copy. With this change, the tests are actually validating the proper renames.
I add a short-circuit return for the case where the source file does not exist, as there is no reason to delete a file from archive if we don't even have a source file to move.
I retained the logic for sidelining the archived file and moving the source file, if they both exist. One could argue that we could ignore the source file instead... I am really hoping this is the "one in a trillion" case that the comment mentions. So I left the logic as-is, in case it is silently being relied on somewhere, or in case for some reason the archive file truly is detritus. I did upgrade the log from debug to error as this does not seem like any condition we want to rely on.

d-c-manning · 2022-03-25T06:37:01Z

hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java

@@ -185,11 +215,30 @@ private void testArchiveStoreFilesDifferentFileSystems(String walDir, String exp
    when(mockedFile.getPath()).thenReturn(filePath);
    when(mockedFileSystem.rename(any(),any())).thenReturn(true);
    archivingFunction.apply(conf, mockedFileSystem, mockedRegion, tableDir, family, list);
-    ArgumentCaptor<Path> pathCaptor = ArgumentCaptor.forClass(Path.class);
-    verify(mockedFileSystem, times(2)).rename(pathCaptor.capture(), any());


note that we used to validate that:

we had two rename calls (one to sideline the timestamped backup, and one to actually do the archive)

we only validated the first call - that the source file was in the archive folder... which is not the important part we want to validate here.

Now we do some improved validations depending on the scenario.

Apache-HBase · 2022-03-25T07:15:43Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 0s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 40s	master passed
+1 💚	compile	2m 39s	master passed
+1 💚	checkstyle	0m 39s	master passed
+1 💚	spotbugs	1m 31s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 28s	the patch passed
+1 💚	compile	2m 34s	the patch passed
+1 💚	javac	2m 34s	the patch passed
+1 💚	checkstyle	0m 37s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	12m 56s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	1m 33s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 11s	The patch does not generate ASF License warnings.
		35m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4274/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4274
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux f88854f5863d 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4ac600b`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	60 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4274/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-03-25T10:51:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 53s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 42s	master passed
+1 💚	compile	1m 3s	master passed
+1 💚	shadedjars	5m 25s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 42s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 36s	the patch passed
+1 💚	compile	0m 54s	the patch passed
+1 💚	javac	0m 54s	the patch passed
+1 💚	shadedjars	5m 18s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 35s	the patch passed
		_ Other Tests _
+1 💚	unit	225m 18s	hbase-server in the patch passed.
		251m 4s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4274/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4274
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 7b8c4c18c7eb 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4ac600b`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4274/1/testReport/
Max. process+thread count	2550 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4274/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java

Apache-HBase · 2022-03-26T05:11:02Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 41s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 0s	master passed
+1 💚	compile	2m 39s	master passed
+1 💚	checkstyle	0m 42s	master passed
+1 💚	spotbugs	1m 36s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 29s	the patch passed
+1 💚	compile	2m 37s	the patch passed
+1 💚	javac	2m 37s	the patch passed
+1 💚	checkstyle	0m 40s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	13m 37s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	1m 37s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 11s	The patch does not generate ASF License warnings.
		35m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4274/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4274
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 188444a904a4 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4ac600b`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	60 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4274/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-03-26T07:48:35Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 12s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 46s	master passed
+1 💚	compile	0m 46s	master passed
+1 💚	shadedjars	3m 38s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 28s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 35s	the patch passed
+1 💚	compile	0m 46s	the patch passed
+1 💚	javac	0m 46s	the patch passed
+1 💚	shadedjars	3m 39s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	the patch passed
		_ Other Tests _
+1 💚	unit	175m 48s	hbase-server in the patch passed.
		193m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4274/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4274
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 8e7fad7a9095 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4ac600b`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4274/2/testReport/
Max. process+thread count	3347 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4274/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-03-26T08:01:21Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 38s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 40s	master passed
+1 💚	compile	0m 42s	master passed
+1 💚	shadedjars	4m 18s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 30s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 29s	the patch passed
+1 💚	compile	0m 43s	the patch passed
+1 💚	javac	0m 43s	the patch passed
+1 💚	shadedjars	4m 18s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	the patch passed
		_ Other Tests _
+1 💚	unit	187m 7s	hbase-server in the patch passed.
		205m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4274/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4274
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux adf43f5814d8 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4ac600b`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4274/2/testReport/
Max. process+thread count	2791 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4274/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…rchive (#4274) Signed-off-by: Andrew Purtell <apurtell@apache.org>

…rchive (apache#4274) Signed-off-by: Andrew Purtell <apurtell@apache.org> (cherry picked from commit da2c32c) Change-Id: I1c16751230e9472f61fc0583ee3c447f29896e2d

fix for HBASE-26718

b1ef853

d-c-manning commented Mar 25, 2022

View reviewed changes

apurtell requested changes Mar 25, 2022

View reviewed changes

hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java Show resolved Hide resolved

throw exception on mismatched size

1b31271

apurtell approved these changes Mar 28, 2022

View reviewed changes

apurtell changed the title ~~fix for HBASE-26718~~ HBASE-26718 HFileArchiver can remove referenced StoreFiles from the archive Mar 28, 2022

apurtell merged commit a4db023 into apache:master Mar 28, 2022

asfgit pushed a commit that referenced this pull request Mar 28, 2022

HBASE-26718 HFileArchiver can remove referenced StoreFiles from the a…

0dbcf8f

…rchive (#4274) Signed-off-by: Andrew Purtell <apurtell@apache.org>

asfgit pushed a commit that referenced this pull request Mar 28, 2022

HBASE-26718 HFileArchiver can remove referenced StoreFiles from the a…

da2c32c

…rchive (#4274) Signed-off-by: Andrew Purtell <apurtell@apache.org>

asfgit pushed a commit that referenced this pull request Mar 28, 2022

HBASE-26718 HFileArchiver can remove referenced StoreFiles from the a…

a28e7ac

…rchive (#4274) Signed-off-by: Andrew Purtell <apurtell@apache.org>

d-c-manning deleted the HBASE-26718 branch March 28, 2022 19:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-26718 HFileArchiver can remove referenced StoreFiles from the archive #4274

HBASE-26718 HFileArchiver can remove referenced StoreFiles from the archive #4274

d-c-manning commented Mar 25, 2022

d-c-manning Mar 25, 2022 •

edited

Loading

Apache-HBase commented Mar 25, 2022

Apache-HBase commented Mar 25, 2022

Apache-HBase commented Mar 26, 2022

Apache-HBase commented Mar 26, 2022

Apache-HBase commented Mar 26, 2022

HBASE-26718 HFileArchiver can remove referenced StoreFiles from the archive #4274

HBASE-26718 HFileArchiver can remove referenced StoreFiles from the archive #4274

Conversation

d-c-manning commented Mar 25, 2022

d-c-manning Mar 25, 2022 • edited Loading

Choose a reason for hiding this comment

Apache-HBase commented Mar 25, 2022

Apache-HBase commented Mar 25, 2022

Apache-HBase commented Mar 26, 2022

Apache-HBase commented Mar 26, 2022

Apache-HBase commented Mar 26, 2022

d-c-manning Mar 25, 2022 •

edited

Loading