HBASE-28687 BackupSystemTable#checkSystemTable should ensure system tables are enabled #6018
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
Hmm. Test failures could be related. Looking into it edit: they pass on my machine 😓 |
…ables are enabled
rmdmattingly
force-pushed
the
HBASE-28687
branch
from
7aef574 to
f920d12
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
This same test is failing across multiple PRs: #6017 |
|
💔 -1 overall
This message was automatically generated. |
|
Okay, I can reproduce the failures if I run all backups tests locally rather than just the one. I bet this has something to do with all of the inheritance going on in these unit tests |
bbeaudreault
approved these changes
Jun 21, 2024
There was a problem hiding this comment.
IMO enabling them makes sense since kicking off a backup is an implicitly request to use the backup state. It's also consistent with the other states handled here.
The only other thing I'd say is, do we need to handle exceptions? For example if we check that the table is disabled and it is, so we go to enable and it's already enabled. I've seen this problem in other areas where we handle table lifecycle automatically. Basically if multiple callers conflict, it throws an exception that could easily have been handled.
If this is already protected by the backup lock, then may not be necessary.
rmdmattingly
commented
Jun 23, 2024
Comment on lines
+121
to
+151
| @Before | ||
| public void ensurePreviousBackupTestsAreCleanedUp() throws Exception { | ||
| // Every operation here may not be necessary for any given test, | ||
| // some often being no-ops. the goal is to help ensure atomicity | ||
| // of that tests that implement TestBackupBase | ||
| try (BackupAdmin backupAdmin = getBackupAdmin()) { | ||
| backupManager.finishBackupSession(); | ||
| backupAdmin.listBackupSets().forEach(backupSet -> { | ||
| try { | ||
| backupAdmin.deleteBackupSet(backupSet.getName()); | ||
| } catch (IOException ignored) { | ||
| } | ||
| }); | ||
| } catch (Exception ignored) { | ||
| } | ||
| Arrays.stream(TEST_UTIL.getAdmin().listTableNames()) | ||
| .filter(tableName -> !tableName.isSystemTable()).forEach(tableName -> { | ||
| try { | ||
| TEST_UTIL.truncateTable(tableName); | ||
| } catch (IOException ignored) { | ||
| } | ||
| }); | ||
| TEST_UTIL.getMiniHBaseCluster().getRegionServerThreads().forEach(rst -> { | ||
| try { | ||
| LogRoller walRoller = rst.getRegionServer().getWalRoller(); | ||
| walRoller.requestRollAll(); | ||
| walRoller.waitUntilWalRollFinished(); | ||
| } catch (Exception ignored) { | ||
| } | ||
| }); | ||
| } |
There was a problem hiding this comment.
After some playing around, this seemed like a pretty good way to get TestBackupMerge, and others, to consistently pass even when ran in bulk. Another option would be to get rid of the TestBackupMerge inheritance, but that would be a really annoying change to implement, and would probably be significantly slower in the build pipeline
There was a problem hiding this comment.
Sure, let's try it.
For what it's worth, I've had success with refactoring inheritance out of these junit4 tests by moving shared logic into an ExternalResource implementation. It doesn't let you reuse @Test implementations, but it does make implementing before/after logic more composable.
|
Agreed with the feedback, I added handling of TableNotDisabled exceptions |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
ndimiduk
approved these changes
Jun 26, 2024
Comment on lines
+121
to
+151
| @Before | ||
| public void ensurePreviousBackupTestsAreCleanedUp() throws Exception { | ||
| // Every operation here may not be necessary for any given test, | ||
| // some often being no-ops. the goal is to help ensure atomicity | ||
| // of that tests that implement TestBackupBase | ||
| try (BackupAdmin backupAdmin = getBackupAdmin()) { | ||
| backupManager.finishBackupSession(); | ||
| backupAdmin.listBackupSets().forEach(backupSet -> { | ||
| try { | ||
| backupAdmin.deleteBackupSet(backupSet.getName()); | ||
| } catch (IOException ignored) { | ||
| } | ||
| }); | ||
| } catch (Exception ignored) { | ||
| } | ||
| Arrays.stream(TEST_UTIL.getAdmin().listTableNames()) | ||
| .filter(tableName -> !tableName.isSystemTable()).forEach(tableName -> { | ||
| try { | ||
| TEST_UTIL.truncateTable(tableName); | ||
| } catch (IOException ignored) { | ||
| } | ||
| }); | ||
| TEST_UTIL.getMiniHBaseCluster().getRegionServerThreads().forEach(rst -> { | ||
| try { | ||
| LogRoller walRoller = rst.getRegionServer().getWalRoller(); | ||
| walRoller.requestRollAll(); | ||
| walRoller.waitUntilWalRollFinished(); | ||
| } catch (Exception ignored) { | ||
| } | ||
| }); | ||
| } |
There was a problem hiding this comment.
Sure, let's try it.
For what it's worth, I've had success with refactoring inheritance out of these junit4 tests by moving shared logic into an ExternalResource implementation. It doesn't let you reuse @Test implementations, but it does make implementing before/after logic more composable.
ndimiduk
pushed a commit
to ndimiduk/hbase
that referenced
this pull request
Jun 26, 2024
…ables are enabled (apache#6018) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
to ndimiduk/hbase
that referenced
this pull request
Jun 26, 2024
…ables are enabled (apache#6018) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
to ndimiduk/hbase
that referenced
this pull request
Jun 26, 2024
…ables are enabled (apache#6018) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
that referenced
this pull request
Jun 27, 2024
…ables are enabled (#6018) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
that referenced
this pull request
Jun 27, 2024
…ables are enabled (#6018) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
that referenced
this pull request
Jun 27, 2024
…ables are enabled (#6018) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
rmdmattingly
added a commit
to HubSpot/hbase
that referenced
this pull request
Jun 28, 2024
…ables are enabled (apache#6018) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
rmdmattingly
added a commit
to HubSpot/hbase
that referenced
this pull request
Jun 28, 2024
…ables are enabled (apache#6018) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
rmdmattingly
added a commit
to HubSpot/hbase
that referenced
this pull request
Jul 8, 2024
…ld ensure system tables are enabled (apache#6018) (#102) Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
rmdmattingly
added a commit
to HubSpot/hbase
that referenced
this pull request
Jul 8, 2024
…ld ensure system tables are enabled (apache#6018) (#103) Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See https://issues.apache.org/jira/browse/HBASE-28687
If the backup system tables become disabled, then we enter a state which the backup client will not recover from. Without manual intervention, every subsequent backup attempt will fail on BackupSystemTable's calls to waitForSystemTable.
This checkSystemTable method already ensures that the tables exist — it should also ensure that the tables are enabled before we await that condition.
Alternatively, we could fast-fail if the tables are disabled rather than awaiting an enabled state that will never occur.
cc @ndimiduk @charlesconnell @hgromer