HBASE-28680 BackupLogCleaner should clean up archived HMaster logs #6006
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rmdmattingly
commented
Jun 20, 2024
Comment on lines
-126
to
-127
| String fn = file.getPath().getName(); | ||
| if (fn.startsWith(WALProcedureStore.LOG_PREFIX)) { |
There was a problem hiding this comment.
Logic was already in place for marking fresh HMaster WALs as deletable — I believe we just forgot to consider archived ones too
Comment on lines
-132
to
130
| try { | ||
| Address walServerAddress = | ||
| Address.fromString(BackupUtils.parseHostNameFromLogFile(file.getPath())); | ||
| long walTimestamp = AbstractFSWALProvider.getTimestamp(file.getPath().getName()); | ||
|
|
||
| if ( | ||
| !addressToLastBackupMap.containsKey(walServerAddress) | ||
| || addressToLastBackupMap.get(walServerAddress) >= walTimestamp | ||
| ) { | ||
| filteredFiles.add(file); | ||
| } | ||
| } catch (Exception ex) { | ||
| LOG.warn( | ||
| "Error occurred while filtering file: {} with error: {}. Ignoring cleanup of this log", | ||
| file.getPath(), ex.getMessage()); | ||
| } |
There was a problem hiding this comment.
I moved this logic to a separate method to improve testability
Comment on lines
+168
to
+188
| try { | ||
| String hostname = BackupUtils.parseHostNameFromLogFile(path); | ||
| if (hostname == null) { | ||
| LOG.warn( | ||
| "Cannot parse hostname from RegionServer WAL file: {}. Ignoring cleanup of this log", | ||
| path); | ||
| return false; | ||
| } | ||
| Address walServerAddress = Address.fromString(hostname); | ||
| long walTimestamp = AbstractFSWALProvider.getTimestamp(path.getName()); | ||
|
|
||
| if ( | ||
| !addressToLastBackupMap.containsKey(walServerAddress) | ||
| || addressToLastBackupMap.get(walServerAddress) >= walTimestamp | ||
| ) { | ||
| return true; | ||
| } | ||
| } catch (Exception ex) { | ||
| LOG.warn("Error occurred while filtering file: {}. Ignoring cleanup of this log", path, ex); | ||
| return false; | ||
| } |
There was a problem hiding this comment.
This logic is largely the same as what was removed above, with two key differences:
- Of course, it will now return archived HMaster WALs as deletable too
- Better handling of the possible
nullfromBackupUtils#parseHostNameFromLogFile. It was difficult to reason about this error without a patch in our fork because, from the user's perspective, we were previously just hitting an NPE when constructing an Address
Comment on lines
+146
to
+152
| Path normalPath = | ||
| new Path("/hbase/oldWALs/hmaster%2C60000%2C1716224062663.1716247552189$masterlocalwal$"); | ||
| assertTrue(BackupLogCleaner.canDeleteFile(Collections.emptyMap(), normalPath)); | ||
|
|
||
| Path masterPath = new Path( | ||
| "/hbase/MasterData/oldWALs/hmaster%2C60000%2C1716224062663.1716247552189$masterlocalwal$"); | ||
| assertTrue(BackupLogCleaner.canDeleteFile(Collections.emptyMap(), masterPath)); |
There was a problem hiding this comment.
I believe it is necessary to check both paths for backwards compatibility — I'm not totally sure when the directory changed, but the book tells me that it was likely around 2.3. Anyway, there's no additional business logic necessary to flag both paths, but I thought it was worth explicitly testing both
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
rmdmattingly
changed the title
|
Pretty sure the build failure is just noise. I'm going to force push to rebuild. |
rmdmattingly
force-pushed
the
HBASE-28680
branch
from
332297c
to
2edc36b
Compare
rmdmattingly
force-pushed
the
HBASE-28680
branch
from
2edc36b
to
3f00def
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
ndimiduk
approved these changes
Jun 21, 2024
bbeaudreault
approved these changes
Jun 21, 2024
ndimiduk
pushed a commit
to ndimiduk/hbase
that referenced
this pull request
Jun 21, 2024
…ely (apache#6006) We have been trying to setup daily incremental backups for hundreds of clusters at my day job. Recently we discovered that old WALs were piling up across many clusters inline with when we began running incremental backups. This led to the realization that the BackupLogCleaner will always skip archived HMaster WALs. This is a problem because, if a cleaner is skipping a given file, then the CleanerChore will never delete it. This seems like a misunderstanding of what it means to "skip" a WAL in a BaseLogCleanerDelegate, and, instead, we should always return these HMaster WALs as deletable from the perspective of the BackupLogCleaner. We could subject them to the same scrutiny as RegionServer WALs: are they older than the most recent successful backup? But, if I understand correctly, HMaster WALs do not contain any data relevant to table backups, so that would be unnecessary. Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
to ndimiduk/hbase
that referenced
this pull request
Jun 21, 2024
…ely (apache#6006) We have been trying to setup daily incremental backups for hundreds of clusters at my day job. Recently we discovered that old WALs were piling up across many clusters inline with when we began running incremental backups. This led to the realization that the BackupLogCleaner will always skip archived HMaster WALs. This is a problem because, if a cleaner is skipping a given file, then the CleanerChore will never delete it. This seems like a misunderstanding of what it means to "skip" a WAL in a BaseLogCleanerDelegate, and, instead, we should always return these HMaster WALs as deletable from the perspective of the BackupLogCleaner. We could subject them to the same scrutiny as RegionServer WALs: are they older than the most recent successful backup? But, if I understand correctly, HMaster WALs do not contain any data relevant to table backups, so that would be unnecessary. Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
to ndimiduk/hbase
that referenced
this pull request
Jun 21, 2024
…ely (apache#6006) We have been trying to setup daily incremental backups for hundreds of clusters at my day job. Recently we discovered that old WALs were piling up across many clusters inline with when we began running incremental backups. This led to the realization that the BackupLogCleaner will always skip archived HMaster WALs. This is a problem because, if a cleaner is skipping a given file, then the CleanerChore will never delete it. This seems like a misunderstanding of what it means to "skip" a WAL in a BaseLogCleanerDelegate, and, instead, we should always return these HMaster WALs as deletable from the perspective of the BackupLogCleaner. We could subject them to the same scrutiny as RegionServer WALs: are they older than the most recent successful backup? But, if I understand correctly, HMaster WALs do not contain any data relevant to table backups, so that would be unnecessary. Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
that referenced
this pull request
Jun 21, 2024
…ely (#6006) We have been trying to setup daily incremental backups for hundreds of clusters at my day job. Recently we discovered that old WALs were piling up across many clusters inline with when we began running incremental backups. This led to the realization that the BackupLogCleaner will always skip archived HMaster WALs. This is a problem because, if a cleaner is skipping a given file, then the CleanerChore will never delete it. This seems like a misunderstanding of what it means to "skip" a WAL in a BaseLogCleanerDelegate, and, instead, we should always return these HMaster WALs as deletable from the perspective of the BackupLogCleaner. We could subject them to the same scrutiny as RegionServer WALs: are they older than the most recent successful backup? But, if I understand correctly, HMaster WALs do not contain any data relevant to table backups, so that would be unnecessary. Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
that referenced
this pull request
Jun 21, 2024
…ely (#6006) We have been trying to setup daily incremental backups for hundreds of clusters at my day job. Recently we discovered that old WALs were piling up across many clusters inline with when we began running incremental backups. This led to the realization that the BackupLogCleaner will always skip archived HMaster WALs. This is a problem because, if a cleaner is skipping a given file, then the CleanerChore will never delete it. This seems like a misunderstanding of what it means to "skip" a WAL in a BaseLogCleanerDelegate, and, instead, we should always return these HMaster WALs as deletable from the perspective of the BackupLogCleaner. We could subject them to the same scrutiny as RegionServer WALs: are they older than the most recent successful backup? But, if I understand correctly, HMaster WALs do not contain any data relevant to table backups, so that would be unnecessary. Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
that referenced
this pull request
Jun 21, 2024
…ely (#6006) We have been trying to setup daily incremental backups for hundreds of clusters at my day job. Recently we discovered that old WALs were piling up across many clusters inline with when we began running incremental backups. This led to the realization that the BackupLogCleaner will always skip archived HMaster WALs. This is a problem because, if a cleaner is skipping a given file, then the CleanerChore will never delete it. This seems like a misunderstanding of what it means to "skip" a WAL in a BaseLogCleanerDelegate, and, instead, we should always return these HMaster WALs as deletable from the perspective of the BackupLogCleaner. We could subject them to the same scrutiny as RegionServer WALs: are they older than the most recent successful backup? But, if I understand correctly, HMaster WALs do not contain any data relevant to table backups, so that would be unnecessary. Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
rmdmattingly
added a commit
to HubSpot/hbase
that referenced
this pull request
Jun 24, 2024
…ely (apache#6006) We have been trying to setup daily incremental backups for hundreds of clusters at my day job. Recently we discovered that old WALs were piling up across many clusters inline with when we began running incremental backups. This led to the realization that the BackupLogCleaner will always skip archived HMaster WALs. This is a problem because, if a cleaner is skipping a given file, then the CleanerChore will never delete it. This seems like a misunderstanding of what it means to "skip" a WAL in a BaseLogCleanerDelegate, and, instead, we should always return these HMaster WALs as deletable from the perspective of the BackupLogCleaner. We could subject them to the same scrutiny as RegionServer WALs: are they older than the most recent successful backup? But, if I understand correctly, HMaster WALs do not contain any data relevant to table backups, so that would be unnecessary. Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
rmdmattingly
added a commit
to HubSpot/hbase
that referenced
this pull request
Jun 24, 2024
…ely (apache#6006) We have been trying to setup daily incremental backups for hundreds of clusters at my day job. Recently we discovered that old WALs were piling up across many clusters inline with when we began running incremental backups. This led to the realization that the BackupLogCleaner will always skip archived HMaster WALs. This is a problem because, if a cleaner is skipping a given file, then the CleanerChore will never delete it. This seems like a misunderstanding of what it means to "skip" a WAL in a BaseLogCleanerDelegate, and, instead, we should always return these HMaster WALs as deletable from the perspective of the BackupLogCleaner. We could subject them to the same scrutiny as RegionServer WALs: are they older than the most recent successful backup? But, if I understand correctly, HMaster WALs do not contain any data relevant to table backups, so that would be unnecessary. Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
rmdmattingly
added a commit
to HubSpot/hbase
that referenced
this pull request
Jul 8, 2024
… pile up indefinitely (apache#6006) (#101) We have been trying to setup daily incremental backups for hundreds of clusters at my day job. Recently we discovered that old WALs were piling up across many clusters inline with when we began running incremental backups. This led to the realization that the BackupLogCleaner will always skip archived HMaster WALs. This is a problem because, if a cleaner is skipping a given file, then the CleanerChore will never delete it. This seems like a misunderstanding of what it means to "skip" a WAL in a BaseLogCleanerDelegate, and, instead, we should always return these HMaster WALs as deletable from the perspective of the BackupLogCleaner. We could subject them to the same scrutiny as RegionServer WALs: are they older than the most recent successful backup? But, if I understand correctly, HMaster WALs do not contain any data relevant to table backups, so that would be unnecessary. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
rmdmattingly
added a commit
to HubSpot/hbase
that referenced
this pull request
Jul 8, 2024
… pile up indefinitely (apache#6006) (#100) We have been trying to setup daily incremental backups for hundreds of clusters at my day job. Recently we discovered that old WALs were piling up across many clusters inline with when we began running incremental backups. This led to the realization that the BackupLogCleaner will always skip archived HMaster WALs. This is a problem because, if a cleaner is skipping a given file, then the CleanerChore will never delete it. This seems like a misunderstanding of what it means to "skip" a WAL in a BaseLogCleanerDelegate, and, instead, we should always return these HMaster WALs as deletable from the perspective of the BackupLogCleaner. We could subject them to the same scrutiny as RegionServer WALs: are they older than the most recent successful backup? But, if I understand correctly, HMaster WALs do not contain any data relevant to table backups, so that would be unnecessary. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See https://issues.apache.org/jira/browse/HBASE-28680
We have been trying to setup daily incremental backups for hundreds of clusters at my day job. Recently we discovered that old WALs were piling up across many clusters inline with when we began running incremental backups.
This led to the realization that the BackupLogCleaner will always skip archived HMaster WALs. This is a problem because, if a cleaner is skipping a given file, then the CleanerChore will never delete it.
At first this seemed like a misunderstanding of what it means to "skip" a WAL in a BaseLogCleanerDelegate, but when working through this code I now believe it is more of an ordinary bug that forgot to take archived HMaster WALs into account
@ndimiduk @charlesconnell @hgromer