[HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup #7246

ankitsol · 2025-08-26T11:14:20Z

Utilize Backed-up Bulkloaded Files in Incremental Backup

Bulkload Files are backed up in ContinuousBackupReplicationEndpoint if bulkload replication is enabled.
When user trigger incremental backup, it takes backup (copy) of bulkload files too along with regular Hfiles of tables to be backed up. Through this change, in incremental backup, backed up bulkload hfiles are copied, instead of these files from source cluster

Jira: https://issues.apache.org/jira/browse/HBASE-29520

Apache-HBase · 2025-09-04T11:57:19Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ HBASE-28957 Compile Tests _
+1 💚	mvninstall	3m 22s		HBASE-28957 passed
+1 💚	compile	0m 33s		HBASE-28957 passed
-0 ⚠️	checkstyle	0m 9s	/buildtool-branch-checkstyle-hbase-backup.txt	The patch fails to run checkstyle in hbase-backup
+1 💚	spotbugs	0m 31s		HBASE-28957 passed
+1 💚	spotless	0m 51s		branch has no errors when running spotless:check.
-0 ⚠️	patch	0m 58s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 4s		the patch passed
+1 💚	compile	0m 32s		the patch passed
+1 💚	javac	0m 32s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 9s	/buildtool-patch-checkstyle-hbase-backup.txt	The patch fails to run checkstyle in hbase-backup
+1 💚	spotbugs	0m 35s		the patch passed
+1 💚	hadoopcheck	12m 10s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 46s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 10s		The patch does not generate ASF License warnings.
		31m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/9/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7246
JIRA Issue	HBASE-29520
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux fa5c0a4b126b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-28957 / `12940d5`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	84 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/9/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-09-04T12:06:10Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 36s		Docker mode activated.
-0 ⚠️	yetus	0m 4s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ HBASE-28957 Compile Tests _
+1 💚	mvninstall	3m 17s		HBASE-28957 passed
+1 💚	compile	0m 20s		HBASE-28957 passed
+1 💚	javadoc	0m 15s		HBASE-28957 passed
+1 💚	shadedjars	6m 8s		branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	patch	6m 16s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 9s		the patch passed
+1 💚	compile	0m 19s		the patch passed
+1 💚	javac	0m 19s		the patch passed
+1 💚	javadoc	0m 13s		the patch passed
+1 💚	shadedjars	6m 3s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	18m 35s		hbase-backup in the patch passed.
		39m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/9/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7246
JIRA Issue	HBASE-29520
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux b0e61d5ca550 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-28957 / `12940d5`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/9/testReport/
Max. process+thread count	3759 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7246/9/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

taklwu

I only have a minor comment for the warning message , otherwise lgtm

taklwu · 2025-09-04T16:33:59Z

...e-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java

+          LOG.debug("Backup bulkload file found {}", fullBulkLoadBackupPath);
+          p = fullBulkLoadBackupPath;
+        } else {
+          LOG.warn("Backup bulkload file not found {}", fullBulkLoadBackupPath);


I assumed we're hitting the following
1/ it does not have any bulkload previously,
2/ something is wrong?

so, should we ask user to check if this is expected in the warning message?

In non-continuous incremental backup approach, bulkload files are copied directly from the source cluster to the backup location.

In continuous backup approach, these files are instead copied from the bulkload backup location. I’ve added a warning here because if the required bulkload backup files are missing, they would be copied from the source cluster (as a fallback mechanism).

Source cluster files are only deleted after a successful full or incremental backup (in both non-continuous and continuous)

Ideally, with continuous backups, once a bulkload file has been backed up by the replication endpoint, it could be safely deleted from the source cluster. However, doing so would significantly complicate the checkpoint logic, since it would then depend on both WAL flushes and bulkload backups.

In continuous backup approach, these files are instead copied from the bulkload backup location. I’ve added a warning here because if the required bulkload backup files are missing, they would be copied from the source cluster (as a fallback mechanism).

[nit] right, my concerns are the action after the WARN message, and current message does not clearly tell what the user should do when he see this, do you think we should add some suggestion in this message ?

we can improve it as a follow up JIRA or in the documentation, if user see this message , should they care about it? if so, what manual operation should they perform to get rid of it ?

taklwu · 2025-09-04T16:48:47Z

...e-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java


+      // For continuous backup: bulkload files are copied from backup directory defined by
+      // CONF_CONTINUOUS_BACKUP_WAL_DIR instead of source cluster.
+      String backupRootDir = conf.get(CONF_CONTINUOUS_BACKUP_WAL_DIR);


[nit] somehow, hbase.backup.continuous.wal.dir is become the root now for both WALs and Bulkloads, shouldn't we call it as hbase.backup.continuous.dir/hbase.backup.continuous.root.dir

Yes, makes sense. I will create a new JIRA to update this config from everywhere it is referenced

#7246) Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org>

[HBASE-29520] Utilize Backed-up Bulkloaded Files in Incremental Backup

0802f3a

ankitsol changed the base branch from master to HBASE-28957 August 26, 2025 11:15