HBASE-28705 backup log cleaner cleans required walls when using multiple backuproots #6040
Conversation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Ping @bbeaudreault @rmdmattingly PTAL at this? I'm not familiar with the backup code and I think this is a blocker for the 2.6.1 release? |
| if (existingValue == null || existingValue > entry.getValue()) { | ||
| serversToLatestUnusedLogTimestamp.put(address, entry.getValue()); | ||
| } |
There was a problem hiding this comment.
If we want to store the latest, or largest, timestamp in the serversToLatestUnusedLogTimestamp then I think we'd want to flip existingValue > entry.getValue() to existingValue < entry.getValue() if this conditional indicates that entry.getValue() should be added to the map
There was a problem hiding this comment.
Re-reading this code after 2 months, I realize it deserves some additional documentation.
But I think the comparison is good as-is:
- In the first loop (line 88), we track the newest/mostRecent backup per root
- In the second loop (line 95), we check the lowest timestamp (= oldest/leastRecent) per address, using the backup per root
So we effectively check per address which timestamp is included in all backups. Only wals older (= lower TS) than that boundary can be deleted.
| protected static boolean canDeleteFile(Map<Address, Long> addressToLastBackupMap, Path path) { | ||
| protected static boolean canDeleteFile(Map<Address, Long> addressToLatestUnusedLogTs, Path path) { |
There was a problem hiding this comment.
Total nitpick, but we unified on "newest" in that other PR, in case we'd like to use newest here as well for the sake of consistency.
There was a problem hiding this comment.
A "newest unused log timestamp" also feels weird. I'll just rename it to a "boundary" to avoid the newest/oldest confusion.
|
This one looks pretty close. Shall we try to get this in for 2.6.1? |
… backuproots Extend & refactor the test for BackupLogCleaner.
DieterDP-ng
force-pushed
the
HBASE-28705-BackupLogCleaner-cleans-required-WALs-when-using-multiple-backuproots
branch
from
aa0e531 to
64a0156
Compare
|
Rebased on master with additonal javadoc and some renames for clarity. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
… backuproots The BackupLogCleaner is responsible for avoiding the deletion of WAL/logs that still need to be included in a future backup. The logic to decide which files can be deleted did not work correctly when multiple backup roots are used. Each backup root has a different chain of backups (full, incremental1, incremental2, ...). So, if any chain requires a log, it should be preserved. This was not the case. The result was that logs could be incorrectly deleted, resulting in data loss in backups.
DieterDP-ng
force-pushed
the
HBASE-28705-BackupLogCleaner-cleans-required-WALs-when-using-multiple-backuproots
branch
from
64a0156 to
7d76842
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
| Map<TableName, Long> tableNameBackupInfoMap = new HashMap<>(); | ||
| for (BackupInfo backupInfo : backups) { | ||
| // This map tracks, for every backup root, the most recent created backup (= highest timestamp) | ||
| Map<String, BackupInfo> newestBackupPerRootDir = new HashMap<>(); |
There was a problem hiding this comment.
Can there be multiple BackupInfo objects with the same root? (what is a "root" anyway? a FileSystem + Path?) Should this be a Map<String, List<BackupInfo>>?
There was a problem hiding this comment.
A backup root corresponds to the path you specify when doing hbase backup create <type> <backup_path>. Within a single root, you'll have a series of backups. Having multiple roots allows you to manage backups independently (eg: if you delete one root, backups in other roots are unaffected). Incremental backups are always built on top of previous backups from the same root.
Eg, (F = full backup, I = incremental backup):
Time --------------------------------------------->
Root1: F I I I F I I I
Root2: F F I I I F
So to answer your questions:
- There are multiple
BackupInfos per root: 1 per backup created in that root. A root is aPath. - In this piece of logic, we only need the most recent (= newest) backup, so
Map<String, BackupInfo>, where the root is represented as aString.
| * I.e. WALs with a lower (= older) or equal timestamp are no longer needed for future incremental | ||
| * backups. | ||
| */ | ||
| private Map<Address, Long> serverToPreservationBoundaryTs(List<BackupInfo> backups) |
There was a problem hiding this comment.
should this method have a precondition asserting that all BackupInfos in the list have an identical table set?
Otherwise I think this could cause data loss in an incremental backup. Let's say we have one-table and another-table on our cluster, and we back up each one independently. Let's say we first took a backup for one-table, then a backup for another-table. Then we passed the corresponding BackupInfos into this method, this would yield preservation boundaries which would remove WALs necessary for the next incremental backup of one-table
There was a problem hiding this comment.
I don't see an issue. Trying to follow your example:
- Imagine 3 servers S1, S2, S3
- table T1 has regions on S1 and S2, table T2 has regions on S2 and S3
- we have 2 backup roots R1 that backs up T1, and R2 that backs up T2
At time:
- t=0, we backup T1 in R1 => backup B1
- t=0, we backup T2 in R2 => backup B2
- t=10, we backup T1 in R1 => backup B3
- t=20, we backup T2 in R2. => backup B4
Following the logic in this method:
- newestBackupPerRootDir will contain: (R1: B3, R2: B4)
- boundaries will contain: (S1: 10, S2: 10, S3: 20)
So for T1, all WALs up to t=10 can be deleted, for T2, WALs will be preserved from t=10 or t=20, depending whether other tables are present.
There was a problem hiding this comment.
Thank you for the explanation, both this and below. I think I follow you. Let me rephrase.
newestBackupPerRootDir is a "race to the top" in terms of identifying the most recent timestamp across backups, while boundaries is a "race to the bottom" in terms of identifying the least recent backup present on each server. By selecting the "oldest of the newest" you determine the minimum timestamp that must be preserved for each server. Any WALs older than this minimum timestamp can be safely discarded.
There was a problem hiding this comment.
Cool, I think the missing piece for me was that a backup root was specific to a given table
we have 2 backup roots R1 that backs up T1, and R2 that backs up T2
There was a problem hiding this comment.
Cool, I think the missing piece for me was that a backup root was specific to a given table
It isn't. Each backup root contains one or more tables, i.e. a set of tables, these table sets may or may not have tables in common.
There was a problem hiding this comment.
Hmm, okay so reframing your example in a single root:
Imagine
3 servers S1, S2, S3
table T1 has regions on S1 and S2, table T2 has regions on S2 and S3
we have 1 backup root R1 that backs up T1 and T2
At time:
t=0, we backup T1 in R1 => backup B1
t=0, we backup T2 in R1 => backup B2
t=10, we backup T1 in R1 => backup B3
t=20, we backup T2 in R1 => backup B4
Following the logic in this method:
newestBackupPerRootDir will contain: (R1: B4)
We would calculate the preservation boundaries here:
// This map tracks, for every address, the least recent (= oldest / lowest timestamp) inclusion
// in any backup. In other words, it is the timestamp boundary up to which all backups roots
// have included the WAL in their backup.
Map<Address, Long> boundaries = new HashMap<>();
for (BackupInfo backupInfo : newestBackupPerRootDir.values()) {
for (TableName table : backupInfo.getTables()) {
for (Map.Entry<String, Long> entry : backupInfo.getTableSetTimestampMap().get(table)
.entrySet()) {
Address address = Address.fromString(entry.getKey());
Long storedTs = boundaries.get(address);
if (storedTs == null || entry.getValue() < storedTs) {
boundaries.put(address, entry.getValue());
}
}
}
}Since the newest backup in R1 is B4, serverToPreservationBoundaryTs will contain (S1: 20, S2: 20, S3: 20)
In this situation, we must not delete WALs from S1 or S2 between times 10 and 20 because a subsequent incremental backup of T1 will require those WALs.
In the BackupLogCleaner, though, we will end up calling canDeleteFile and hitting this code with that serverToPreservationBoundaryTs (S1: 20, S2: 20, S3: 20) renamed as addressToBoundaryTs:
if (!addressToBoundaryTs.containsKey(walServerAddress)) { // for S1 & S2, this will be false
// irrelevant ...
}
Long backupBoundary = addressToBoundaryTs.get(walServerAddress); // for S1 & S2, this will be 20
if (backupBoundary >= walTimestamp) { // For WALs between times 10 and 20, this will be true
if (LOG.isDebugEnabled()) {
LOG.debug(
"Backup found for server: {}. All backups from {} are newer than file, so deleting: {}",
walServerAddress.getHostName(), backupBoundary, path);
}
// so we will erroneously delete WALs <20, despite the next inc backup of T1 requiring them
return true;
}I guess this depends on how ancestry is defined? If we consider all backups in a root to be ancestors regardless of their table set, then maybe it is okay to delete these WALs. But, if not, then I don't see how it is okay?
There was a problem hiding this comment.
newestBackupPerRootDir will contain: (R1: B4)
Agreed
Since the newest backup in R1 is B4, serverToPreservationBoundaryTs will contain (S1: 20, S2: 20, S3: 20)
I think this is wrong.
To make things more concrete, I'm going to assume all backups were full backups, and B1 == B2 (since it has the same timestamp). I.e. the backups were:
- B1: at timestamp 0, containing tables T1 & T2
- B3: at timestamp 10, containing T1
- B4: at timestamp 20, containing T2
At this point, the data in backupInfo.getTableSetTimestampMap() will be:
T1:
S1: 10
S2: 10
T2:
S2: 20
S3: 20
and serverToPreservationBoundaryTs will be (S1: 10, S2: 10, S3: 20).
The reason that the BackupInfo of B4 contains the log timestamps of tables not included in B4 is due to how the backup client updates these:
// From: FullTableBackupClient
// Updates only the rows for the tables included in B4
backupManager.writeRegionServerLogTimestamp(backupInfo.getTables(), newTimestamps);
// Reads the rows for all tables once included in a backup in this backup root
Map<TableName, Map<String, Long>> newTableSetTimestampMap = backupManager.readLogTimestampMap();
There was a problem hiding this comment.
Sorry for being a pain here, but I'm not sure I agree. When building serverToPreservationBoundaryTs we loop through:
Map<Address, Long> boundaries = new HashMap<>();
for (BackupInfo backupInfo : newestBackupPerRootDir.values()) {
// build boundaries via tableSetTimestampMap, like you said
}
// logging omitted
return boundaries;and we agree that newestBackupPerRootDir.values() will only contain B4. So our boundaries would end up only being based on the timestamps from B4? How does the second newest backup, and beyond, in the root come into play?
There was a problem hiding this comment.
How does the second newest backup, and beyond, in the root come into play?
Answering my own questions a bit here, I think I needed to look deeper into how the BackupInfo gets constructed. When the tableSetTimestampMap is populated, it will contain entries for every table in the root, not just every table in the given backup. So while B4 might only pertain to T2, its corresponding BackupInfo will be aware of the timestamps for T1. That's a confusing design imo, but if true then I think this logic is sound
There was a problem hiding this comment.
It's indeed like that. Doing it that way is a good way to avoid some special cases where backups in the same root backup do not contain the same tables, but the docs surrounding those BackupInfo concepts is indeed lacking.
There was a problem hiding this comment.
@DieterDP-ng thank you for your explanations. I think I can now participate effectively in the discussion.
hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/master/TestBackupLogCleaner.java
Outdated
Show resolved
Hide resolved
| * I.e. WALs with a lower (= older) or equal timestamp are no longer needed for future incremental | ||
| * backups. | ||
| */ | ||
| private Map<Address, Long> serverToPreservationBoundaryTs(List<BackupInfo> backups) |
There was a problem hiding this comment.
Thank you for the explanation, both this and below. I think I follow you. Let me rephrase.
newestBackupPerRootDir is a "race to the top" in terms of identifying the most recent timestamp across backups, while boundaries is a "race to the bottom" in terms of identifying the least recent backup present on each server. By selecting the "oldest of the newest" you determine the minimum timestamp that must be preserved for each server. Any WALs older than this minimum timestamp can be safely discarded.
hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/master/TestBackupLogCleaner.java
Outdated
Show resolved
Hide resolved
hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/master/TestBackupLogCleaner.java
Outdated
Show resolved
Hide resolved
hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/master/TestBackupLogCleaner.java
Outdated
Show resolved
Hide resolved
| @@ -61,8 +65,10 @@ public class TestBackupLogCleaner extends TestBackupBase { | |||
|
|
|||
| @Test | |||
| public void testBackupLogCleaner() throws Exception { | |||
There was a problem hiding this comment.
This test is not quite what you described in your earlier comment, and I'm pretty sure that it doesn't test the criteria in the same way either. It's a valid test case, but I think it's missing some cases that we would like to see covered.
In this test, we have a single region server, which serves as S2 from your above example.
- At T0 you take a full backup B1 of all four tables at R1.
boundaryfor our server is the timestamp of oldest of the wals of all the tables. - Data is inserted into tables 1 and 2.
- At T1 you take incremental backup B1 of tables 1, 2, 3 to R1.
boundaryfor our server doesn't change because the oldest wal is still from B1 and table 4. - At T2 you take a full backup B2 of tables 1, 2, 3 to R2.
boundarydoes not change because the oldest wal is still from B1 and table 4.
Based on my understanding, this test never sees a point when a wal can be discarded because wals from table4 are never rolled and so the wals identified for table4 in B1 are always the "oldest of the newest" for this server.
I think that we'd like to see a test that does what you described in your initial comment: observes the "rising tide" of wal timestamp on the server as backups across roots progress.
There was a problem hiding this comment.
I've majorly refactored & extended the testcase, and found a bug along the way.
@rmdmattingly @ndimiduk please take a look.
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
There was a problem hiding this comment.
I'll give @rmdmattingly a chance to comment, but this is good by me. Thanks a lot for this cleanup @DieterDP-ng , this is a real improvement in the system.
… backuproots (apache#6040) The BackupLogCleaner is responsible for avoiding the deletion of WAL/logs that still need to be included in a future backup. The logic to decide which files can be deleted did not work correctly when multiple backup roots are used. Each backup root has a different chain of backups (full, incremental1, incremental2, ...). So, if any chain requires a log, it should be preserved. This was not the case. The result was that logs could be incorrectly deleted, resulting in data loss in backups. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Ray Mattingly <rmattingly@apache.org >
… backuproots (apache#6040) The BackupLogCleaner is responsible for avoiding the deletion of WAL/logs that still need to be included in a future backup. The logic to decide which files can be deleted did not work correctly when multiple backup roots are used. Each backup root has a different chain of backups (full, incremental1, incremental2, ...). So, if any chain requires a log, it should be preserved. This was not the case. The result was that logs could be incorrectly deleted, resulting in data loss in backups. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Ray Mattingly <rmattingly@apache.org >
… backuproots (apache#6040) The BackupLogCleaner is responsible for avoiding the deletion of WAL/logs that still need to be included in a future backup. The logic to decide which files can be deleted did not work correctly when multiple backup roots are used. Each backup root has a different chain of backups (full, incremental1, incremental2, ...). So, if any chain requires a log, it should be preserved. This was not the case. The result was that logs could be incorrectly deleted, resulting in data loss in backups. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Ray Mattingly <rmattingly@apache.org >
… backuproots (#6040) The BackupLogCleaner is responsible for avoiding the deletion of WAL/logs that still need to be included in a future backup. The logic to decide which files can be deleted did not work correctly when multiple backup roots are used. Each backup root has a different chain of backups (full, incremental1, incremental2, ...). So, if any chain requires a log, it should be preserved. This was not the case. The result was that logs could be incorrectly deleted, resulting in data loss in backups. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Ray Mattingly <rmattingly@apache.org >
… backuproots (#6040) The BackupLogCleaner is responsible for avoiding the deletion of WAL/logs that still need to be included in a future backup. The logic to decide which files can be deleted did not work correctly when multiple backup roots are used. Each backup root has a different chain of backups (full, incremental1, incremental2, ...). So, if any chain requires a log, it should be preserved. This was not the case. The result was that logs could be incorrectly deleted, resulting in data loss in backups. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Ray Mattingly <rmattingly@apache.org > Co-authored-by: DieterDP <90392398+DieterDP-ng@users.noreply.github.com>
… backuproots (#6040) The BackupLogCleaner is responsible for avoiding the deletion of WAL/logs that still need to be included in a future backup. The logic to decide which files can be deleted did not work correctly when multiple backup roots are used. Each backup root has a different chain of backups (full, incremental1, incremental2, ...). So, if any chain requires a log, it should be preserved. This was not the case. The result was that logs could be incorrectly deleted, resulting in data loss in backups. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Ray Mattingly <rmattingly@apache.org > Co-authored-by: DieterDP <90392398+DieterDP-ng@users.noreply.github.com>
https://issues.apache.org/jira/projects/HBASE/issues/HBASE-28705