HBASE-28460 Full backup restore failed on empty HFiles #5782

DieterDP-ng · 2024-03-27T15:04:29Z

No description provided.

Apache-HBase · 2024-03-27T15:46:19Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 37s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 58s	master passed
+1 💚	compile	0m 27s	master passed
+1 💚	checkstyle	0m 12s	master passed
+1 💚	spotless	0m 45s	branch has no errors when running spotless:check.
+1 💚	spotbugs	0m 32s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 10s	the patch passed
+1 💚	compile	0m 29s	the patch passed
+1 💚	javac	0m 29s	the patch passed
+1 💚	checkstyle	0m 13s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	5m 56s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	0m 50s	patch has no errors when running spotless:check.
+1 💚	spotbugs	0m 45s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 12s	The patch does not generate ASF License warnings.
		25m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5782
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 8cf984148b3f 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `298c550`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-03-27T15:49:55Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 25s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 10s	master passed
+1 💚	compile	0m 16s	master passed
+1 💚	shadedjars	5m 35s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 13s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 54s	the patch passed
+1 💚	compile	0m 16s	the patch passed
+1 💚	javac	0m 16s	the patch passed
+1 💚	shadedjars	5m 34s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 12s	the patch passed
		_ Other Tests _
+1 💚	unit	9m 48s	hbase-backup in the patch passed.
		29m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#5782
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 12d25b1bf672 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `298c550`
Default Java	Eclipse Adoptium-17.0.10+7
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/testReport/
Max. process+thread count	3877 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-03-27T15:54:07Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 53s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 43s	master passed
+1 💚	compile	0m 21s	master passed
+1 💚	shadedjars	6m 13s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 17s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 16s	the patch passed
+1 💚	compile	0m 21s	the patch passed
+1 💚	javac	0m 21s	the patch passed
+1 💚	shadedjars	5m 29s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 13s	the patch passed
		_ Other Tests _
+1 💚	unit	11m 37s	hbase-backup in the patch passed.
		33m 34s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5782
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 35f0111cf1c0 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `298c550`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/testReport/
Max. process+thread count	3476 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-03-27T15:55:03Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 34s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 13s	master passed
+1 💚	compile	0m 19s	master passed
+1 💚	shadedjars	7m 3s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 16s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 13s	the patch passed
+1 💚	compile	0m 20s	the patch passed
+1 💚	javac	0m 20s	the patch passed
+1 💚	shadedjars	6m 7s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 13s	the patch passed
		_ Other Tests _
+1 💚	unit	12m 1s	hbase-backup in the patch passed.
		34m 25s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5782
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux d3b46a6b2794 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `298c550`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/testReport/
Max. process+thread count	3756 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

bbeaudreault

While you are here, can you fix the LOG name in RestoreTool to match the classname? Right now it's set as BackupUtils which is misleading.

As a big aside, I wonder if we should simplify FULL restore to just use the existing region boundaries in the snapshot manifest

hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestFullRestore.java

bbeaudreault · 2024-04-01T18:32:51Z

hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/MapReduceRestoreJob.java

-            throw new IOException("Can not restore from backup directory " + dirs
-              + " (check Hadoop and HBase logs). Bulk loader returns null");
-          }
+          loader.bulkLoad(newTableNames[i], bulkOutputPath);


Is this a safe change? It solves for 1 case where empty hfiles could fail the restore, but I wonder if there are other cases where the bulkload might unexpectedly not restore any files.

I don't have enough knowledge about the internals to give an informed opinion I'm afraid. The test suite finds no problem with the change, but that's no guarantee of course.

But I think that if there would be cases where not loading anything in the bulkload indicated an error, it would have to be detected at other locations rather than here. Restoring no files is a valid usecase, and so the only thing I can think of is that there would be an issue when those HFiles are written, or that something happens to those files before they are restored.

The former should already be detected as an issue I'd hope.

The latter could be something that damages or deletes the HFiles. The only (valid) to detect that would be to track how many (or even which specific HFiles) should be loaded. So metadata that would be recorded when creating the backup, and verified when restoring. But - the assumption that this could happen, would have a lot more repercussions outside of backup&restore in my opinion.

Apache-HBase · 2024-04-02T09:45:23Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 59s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 2s	master passed
+1 💚	compile	0m 26s	master passed
+1 💚	checkstyle	0m 11s	master passed
+1 💚	spotless	0m 44s	branch has no errors when running spotless:check.
+1 💚	spotbugs	0m 31s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 44s	the patch passed
+1 💚	compile	0m 24s	the patch passed
+1 💚	javac	0m 24s	the patch passed
+1 💚	checkstyle	0m 11s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	4m 54s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	0m 38s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 12s	The patch does not generate ASF License warnings.
		24m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5782
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 6634d1785539 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2941d6e`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-04-02T09:50:25Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 27s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 49s	master passed
+1 💚	compile	0m 16s	master passed
+1 💚	shadedjars	5m 41s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 12s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 30s	the patch passed
+1 💚	compile	0m 15s	the patch passed
+1 💚	javac	0m 15s	the patch passed
+1 💚	shadedjars	5m 38s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 11s	the patch passed
		_ Other Tests _
+1 💚	unit	10m 36s	hbase-backup in the patch passed.
		29m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5782
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux d885ffc4bd10 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2941d6e`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/testReport/
Max. process+thread count	3535 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-04-02T09:51:22Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 25s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 54s	master passed
+1 💚	compile	0m 17s	master passed
+1 💚	shadedjars	5m 33s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 11s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 53s	the patch passed
+1 💚	compile	0m 15s	the patch passed
+1 💚	javac	0m 15s	the patch passed
+1 💚	shadedjars	5m 33s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 11s	the patch passed
		_ Other Tests _
+1 💚	unit	9m 17s	hbase-backup in the patch passed.
		30m 25s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#5782
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 311353d117e7 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2941d6e`
Default Java	Eclipse Adoptium-17.0.10+7
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/testReport/
Max. process+thread count	3819 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-04-02T09:52:45Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 33s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 3s	master passed
+1 💚	compile	0m 19s	master passed
+1 💚	shadedjars	5m 8s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 16s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 43s	the patch passed
+1 💚	compile	0m 19s	the patch passed
+1 💚	javac	0m 19s	the patch passed
+1 💚	shadedjars	5m 6s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 15s	the patch passed
		_ Other Tests _
+1 💚	unit	10m 31s	hbase-backup in the patch passed.
		31m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5782
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 1a1470ba9288 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2941d6e`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/testReport/
Max. process+thread count	3815 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5782/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>

…les (apache#5782) Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>

HBASE-28460 Full backup restore failed on empty HFiles

7f22704

bbeaudreault reviewed Apr 1, 2024

View reviewed changes

Process review comments

55df619

DieterDP-ng requested a review from bbeaudreault April 2, 2024 09:34

bbeaudreault approved these changes Apr 2, 2024

View reviewed changes

bbeaudreault merged commit 90639d7 into apache:master Apr 2, 2024
1 check passed

bbeaudreault pushed a commit that referenced this pull request Apr 2, 2024

HBASE-28460 Full backup restore failed on empty HFiles (#5782)

2f05ee3

Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>

bbeaudreault pushed a commit that referenced this pull request Apr 2, 2024

HBASE-28460 Full backup restore failed on empty HFiles (#5782)

691fb65

Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>

bbeaudreault pushed a commit that referenced this pull request Apr 2, 2024

HBASE-28460 Full backup restore failed on empty HFiles (#5782)

e8d24fa

Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>

bbeaudreault pushed a commit to HubSpot/hbase that referenced this pull request Apr 2, 2024

HubSpot Backport: HBASE-28460 Full backup restore failed on empty HFi…

a15bdbd

…les (apache#5782) Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-28460 Full backup restore failed on empty HFiles #5782

HBASE-28460 Full backup restore failed on empty HFiles #5782

DieterDP-ng commented Mar 27, 2024

Apache-HBase commented Mar 27, 2024

Apache-HBase commented Mar 27, 2024

Apache-HBase commented Mar 27, 2024

Apache-HBase commented Mar 27, 2024

bbeaudreault left a comment

bbeaudreault Apr 1, 2024

DieterDP-ng Apr 2, 2024

Apache-HBase commented Apr 2, 2024

Apache-HBase commented Apr 2, 2024

Apache-HBase commented Apr 2, 2024

Apache-HBase commented Apr 2, 2024

HBASE-28460 Full backup restore failed on empty HFiles #5782

HBASE-28460 Full backup restore failed on empty HFiles #5782

Conversation

DieterDP-ng commented Mar 27, 2024

Apache-HBase commented Mar 27, 2024

Apache-HBase commented Mar 27, 2024

Apache-HBase commented Mar 27, 2024

Apache-HBase commented Mar 27, 2024

bbeaudreault left a comment

Choose a reason for hiding this comment

bbeaudreault Apr 1, 2024

Choose a reason for hiding this comment

DieterDP-ng Apr 2, 2024

Choose a reason for hiding this comment

Apache-HBase commented Apr 2, 2024

Apache-HBase commented Apr 2, 2024

Apache-HBase commented Apr 2, 2024

Apache-HBase commented Apr 2, 2024