HBASE-29310 Handle Bulk Load Operations in Continuous Backup #7150

ankitsol · 2025-07-10T15:08:53Z

This PR addresses these 2 improvements:

Logging Suggestion during successful bulkload operation:
After a bulk load operation completes, log a message suggesting the user perform a full or incremental backup.

PITR Enhancements:
During PITR, check if any bulk load operation occurred after the last successful backup.
If no such backup exists, inform the user and fail the process.

JIRA: https://issues.apache.org/jira/browse/HBASE-29310

Copilot

Pull Request Overview

This PR enhances bulk load tooling and point-in-time restore (PITR) to handle bulk-load operations in continuous backup workflows.

Adds a post-bulkload suggestion to take a backup
Extends PITR validation to detect bulk-loaded files since the last backup and adds a --force option
Updates model and storage of bulk-load timestamps and adapts tests and CLI to support the new flow

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
BulkLoadHFilesTool.java	Prints a suggestion to back up after successful bulk load
TestPointInTimeRestore.java	Exposed overloaded `buildPITRArgs` with `force` flag and added force restore tests
TestIncrementalBackupWithContinuous.java	Made test counters dynamic, updated `performBulkLoad` and `loadTable` signatures
BulkLoad.java	Introduced `timestamp` field and getter
BackupSystemTable.java	Captured cell timestamp and passed it into the `BulkLoad` constructor
BackupAdminImpl.java	Propagated `isForce` through PITR validation, added `checkBulkLoadAfterBackup`
PointInTimeRestoreRequest.java	Added `force` parameter to the request model
PointInTimeRestoreDriver.java	Parsed `-f/--force` CLI flag and wired it into the request builder
BackupRestoreConstants.java	Defined constants for the force-restore option

Comments suppressed due to low confidence (1)

hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupAdminImpl.java:633

The Javadoc for validatePitr does not mention the new isForce parameter. Please update the method’s Javadoc to describe this argument.

  private void validatePitr(long endTime, TableName[] sTableArray, TableName[] tTableArray,

Copilot · 2025-07-11T18:08:59Z

hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java

+      System.out.println("Bulk load completed successfully.");
+      System.out.println("IMPORTANT: Please take a backup of the table immediately if this table "


Use the framework’s logger (e.g., LOG.info) instead of System.out.println for consistency with the rest of HBase’s logging.

Suggested change

System.out.println("Bulk load completed successfully.");

System.out.println("IMPORTANT: Please take a backup of the table immediately if this table "

LOG.info("Bulk load completed successfully.");

LOG.info("IMPORTANT: Please take a backup of the table immediately if this table "

this is not related, you can ignore it.

hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/BulkLoad.java

Copilot · 2025-07-11T18:08:59Z

hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/PointInTimeRestoreDriver.java


+    boolean force = cmd.hasOption(OPTION_FORCE_RESTORE);
+    if (force) {
+      LOG.debug("Found force option (-{}) in restore command, "


[nitpick] This debug-level message may not be visible in default logs; consider logging at INFO or printing a user-facing warning so users know they’ve forced a restore.

Suggested change

LOG.debug("Found force option (-{}) in restore command, "

LOG.info("Found force option (-{}) in restore command, "

or maybe WARN level?

taklwu

LGTM, please explain to us why we would like the force option to be implemented.

taklwu · 2025-07-11T18:12:16Z

hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/BackupRestoreConstants.java

+  String OPTION_FORCE_RESTORE = "f";
+  String LONG_OPTION_FORCE_RESTORE = "force";
+  String OPTION_FORCE_RESTORE_DESC =


nit: can you explain why should we support this use case if missing backup could cause data loss ?

taklwu · 2025-07-11T18:12:46Z

hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/PointInTimeRestoreDriver.java


+    boolean force = cmd.hasOption(OPTION_FORCE_RESTORE);
+    if (force) {
+      LOG.debug("Found force option (-{}) in restore command, "


or maybe WARN level?

taklwu · 2025-07-11T18:13:37Z

hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java

+      System.out.println("Bulk load completed successfully.");
+      System.out.println("IMPORTANT: Please take a backup of the table immediately if this table "


this is not related, you can ignore it.

taklwu · 2025-07-11T18:16:05Z

also, please try to fix the checkstyle and javac if possible.

kgeisz

LGTM, as long as we're okay with the user being able to perform a backup without performing a bulk load first.

kgeisz

LGTM overall. I just have one comment.

kgeisz · 2025-07-21T22:57:25Z

hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/AbstractPitrRestoreHandler.java

+          throw new IOException("Bulk load operation detected after last successful backup for "
+            + "table: " + sTableName);


Do you think this error message should be a little more descriptive? It is saying why there is an error, but it doesn't necessarily tell the user what they should be doing instead to prevent this error.

My understanding is the user is supposed to perform a full or incremental backup after doing a bulkload. To me, this function is detecting that a bulkload has occurred since the last backup, and it is correctly throwing an error. However, the message isn't telling the user they should do another backup after bulkloading in order to get around this error.

taklwu · 2025-07-22T00:44:51Z

let's wait till the tests complete and then we can merge

taklwu

nit: there is a used variable found by the javac check

Apache-HBase · 2025-07-23T04:57:28Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	buf	0m 1s		buf was not available.
+0 🆗	buf	0m 1s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ HBASE-28957 Compile Tests _
+0 🆗	mvndep	0m 10s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 18s		HBASE-28957 passed
+1 💚	compile	4m 22s		HBASE-28957 passed
-0 ⚠️	checkstyle	0m 10s	/buildtool-branch-checkstyle-hbase-backup.txt	The patch fails to run checkstyle in hbase-backup
+1 💚	spotbugs	4m 29s		HBASE-28957 passed
+1 💚	spotless	0m 48s		branch has no errors when running spotless:check.
-0 ⚠️	patch	1m 8s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 6s		the patch passed
+1 💚	compile	4m 21s		the patch passed
+1 💚	cc	4m 21s		the patch passed
+1 💚	javac	0m 37s		hbase-protocol-shaded in the patch passed.
+1 💚	javac	3m 12s		hbase-server in the patch passed.
+1 💚	javac	0m 32s		hbase-backup generated 0 new + 117 unchanged - 1 fixed = 117 total (was 118)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 9s	/buildtool-patch-checkstyle-hbase-backup.txt	The patch fails to run checkstyle in hbase-backup
+1 💚	spotbugs	4m 52s		the patch passed
+1 💚	hadoopcheck	12m 17s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	hbaseprotoc	1m 37s		the patch passed
+1 💚	spotless	0m 45s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 27s		The patch does not generate ASF License warnings.
		51m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7150/11/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7150
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless cc buflint bufcompat hbaseprotoc
uname	Linux d71e27de5eee 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-28957 / `694511b`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	85 (vs. ulimit of 30000)
modules	C: hbase-protocol-shaded hbase-server hbase-backup U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7150/11/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-07-23T08:54:48Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 37s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ HBASE-28957 Compile Tests _
+0 🆗	mvndep	0m 34s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 12s		HBASE-28957 passed
+1 💚	compile	1m 49s		HBASE-28957 passed
+1 💚	javadoc	0m 51s		HBASE-28957 passed
+1 💚	shadedjars	6m 1s		branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	patch	6m 26s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 3s		the patch passed
+1 💚	compile	1m 49s		the patch passed
+1 💚	javac	1m 49s		the patch passed
+1 💚	javadoc	0m 49s		the patch passed
+1 💚	shadedjars	5m 57s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	0m 35s		hbase-protocol-shaded in the patch passed.
-1 ❌	unit	232m 50s	/patch-unit-hbase-server.txt	hbase-server in the patch failed.
-1 ❌	unit	24m 51s	/patch-unit-hbase-backup.txt	hbase-backup in the patch failed.
		288m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7150/11/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7150
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 59cbeba0001b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-28957 / `694511b`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7150/11/testReport/
Max. process+thread count	5203 (vs. ulimit of 30000)
modules	C: hbase-protocol-shaded hbase-server hbase-backup U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7150/11/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org> Reviewed by: Kevin Geiszler <kevin.j.geiszler@gmail.com>

…7150) Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org> Reviewed by: Kevin Geiszler <kevin.j.geiszler@gmail.com>

…pache#7150)" This reverts commit 5ac2a73.

Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org> Reviewed by: Kevin Geiszler <kevin.j.geiszler@gmail.com>

…pache#7150)" This reverts commit 5ac2a73.

…7150)" (#7290) This reverts commit 5ac2a73.

Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org> Reviewed by: Kevin Geiszler <kevin.j.geiszler@gmail.com>

…7150)" (#7290) This reverts commit 5ac2a73.

HBASE-29310 Handle Bulk Load Operations in Continuous Backup

9ee1908

This comment has been minimized.

Sign in to view

taklwu requested a review from Copilot July 11, 2025 18:07

Copilot AI reviewed Jul 11, 2025

View reviewed changes

taklwu reviewed Jul 11, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

kgeisz approved these changes Jul 14, 2025

View reviewed changes

taklwu mentioned this pull request Jul 14, 2025

HBASE-29449 Update backup describe command for continuous backup #7045

Merged

asolomonC added 4 commits July 20, 2025 01:30

Removed force flag

0dbf4cb

Minor fix

48ffcff

Merge remote-tracking branch 'hbase/HBASE-28957' into as.HBASE-29310

67c84f1

Spotless fix

8046cc2

This comment has been minimized.

Sign in to view

Minor fix

1796c61

This comment has been minimized.

Sign in to view

kgeisz approved these changes Jul 21, 2025

View reviewed changes

taklwu mentioned this pull request Jul 22, 2025

HBASE-29459 Capture bulkload files only till IncrCommittedWalTs during Incremental Backup #7166

Merged

taklwu approved these changes Jul 22, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

Checkstyle fix

f642f2d

This comment has been minimized.

Sign in to view

Merge remote-tracking branch 'hbase/HBASE-28957' into as.HBASE-29310

f5ef04b

This comment has been minimized.

Sign in to view

taklwu reviewed Jul 22, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

Removed unused variables

694511b

taklwu merged commit b2f2c2f into apache:HBASE-28957 Jul 23, 2025
1 check failed

anmolnar pushed a commit that referenced this pull request Jul 28, 2025

HBASE-29310 Handle Bulk Load Operations in Continuous Backup (#7150)

3555f21

Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org> Reviewed by: Kevin Geiszler <kevin.j.geiszler@gmail.com>

anmolnar added a commit to anmolnar/hbase that referenced this pull request Sep 10, 2025

Revert "HBASE-29310 Handle Bulk Load Operations in Continuous Backup (a…

34bb292

…pache#7150)" This reverts commit 5ac2a73.

anmolnar added a commit to anmolnar/hbase that referenced this pull request Sep 10, 2025

Revert "HBASE-29310 Handle Bulk Load Operations in Continuous Backup (a…

2b0c59e

…pache#7150)" This reverts commit 5ac2a73.

anmolnar pushed a commit that referenced this pull request Sep 11, 2025

HBASE-29310 Handle Bulk Load Operations in Continuous Backup (#7150)

4383da3

Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org> Reviewed by: Kevin Geiszler <kevin.j.geiszler@gmail.com>

anmolnar added a commit to anmolnar/hbase that referenced this pull request Sep 11, 2025

Revert "HBASE-29310 Handle Bulk Load Operations in Continuous Backup (a…

331ff9a

…pache#7150)" This reverts commit 5ac2a73.

anmolnar added a commit that referenced this pull request Sep 11, 2025

Revert "HBASE-29310 Handle Bulk Load Operations in Continuous Backup (#…

f5d18fc

…7150)" (#7290) This reverts commit 5ac2a73.

anmolnar pushed a commit that referenced this pull request Nov 6, 2025

HBASE-29310 Handle Bulk Load Operations in Continuous Backup (#7150)

3c5c999

Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org> Reviewed by: Kevin Geiszler <kevin.j.geiszler@gmail.com>

anmolnar added a commit that referenced this pull request Nov 6, 2025

Revert "HBASE-29310 Handle Bulk Load Operations in Continuous Backup (#…

9bd36d0

…7150)" (#7290) This reverts commit 5ac2a73.

		System.out.println("Bulk load completed successfully.");
		System.out.println("IMPORTANT: Please take a backup of the table immediately if this table "

	LOG.debug("Found force option (-{}) in restore command, "
	LOG.info("Found force option (-{}) in restore command, "

		throw new IOException("Bulk load operation detected after last successful backup for "
		+ "table: " + sTableName);

HBASE-29310 Handle Bulk Load Operations in Continuous Backup #7150

HBASE-29310 Handle Bulk Load Operations in Continuous Backup #7150

Uh oh!

Conversation

ankitsol commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

taklwu Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

taklwu Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

taklwu left a comment

Choose a reason for hiding this comment

Uh oh!

taklwu Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

taklwu Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

taklwu Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

taklwu commented Jul 11, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

kgeisz left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

kgeisz left a comment

Choose a reason for hiding this comment

Uh oh!

kgeisz Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

taklwu commented Jul 22, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

taklwu left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

Apache-HBase commented Jul 23, 2025

Uh oh!

Apache-HBase commented Jul 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

ankitsol commented Jul 10, 2025 •

edited

Loading