Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-22942 move Snapshot verification to procedure #662

Closed
wants to merge 1 commit into from

Conversation

sanjeetnishad95
Copy link
Contributor

Problem : Right now we do snapshot verification prior to queueing the restore / clone request from the client. That means the initial call from the client has to block until we're done. On large manifests (~single digit millions) this easily takes longer than the default timeout of 20 minutes.

Solution: Instead we should handle verification as one of the steps in the relevant procedure (hbase 2+) or table handler (hbase 1) so that the master can do it in the background and report status.

@sanjeetnishad95
Copy link
Contributor Author

Hi @busbey , I have moved the SnapshotReferenceUtil.verifySnapshot() from SnapshotManager to prepareRestore() step of RestoreSnapshotProcedure. Please review the PR and let me know if this is what was intended?
Thanks.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
💙 reexec 0m 33s Docker mode activated.
_ Prechecks _
💚 dupname 0m 0s No case conflicting files found.
💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
💚 @author 0m 0s The patch does not contain any @author tags.
💛 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
💚 mvninstall 5m 22s master passed
💚 compile 0m 56s master passed
💚 checkstyle 1m 19s master passed
💚 shadedjars 4m 35s branch has no errors when building our shaded downstream artifacts.
💚 javadoc 0m 36s master passed
💙 spotbugs 4m 10s Used deprecated FindBugs config; considering switching to SpotBugs.
💚 findbugs 4m 9s master passed
_ Patch Compile Tests _
💚 mvninstall 4m 56s the patch passed
💚 compile 0m 58s the patch passed
💚 javac 0m 58s the patch passed
💔 checkstyle 1m 16s hbase-server: The patch generated 1 new + 20 unchanged - 0 fixed = 21 total (was 20)
💚 whitespace 0m 0s The patch has no whitespace issues.
💚 shadedjars 4m 49s patch has no errors when building our shaded downstream artifacts.
💚 hadoopcheck 16m 55s Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚 javadoc 0m 35s the patch passed
💚 findbugs 4m 7s the patch passed
_ Other Tests _
💔 unit 157m 15s hbase-server in the patch failed.
💚 asflicense 0m 35s The patch does not generate ASF License warnings.
215m 2s
Reason Tests
Failed junit tests hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple
hadoop.hbase.client.TestRestoreSnapshotFromClientSimple
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/out/Dockerfile
GITHUB PR #662
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 6429fa7337aa 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build tool maven
Personality /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-662/out/precommit/personality/provided.sh
git revision master / ea24ea7
Default Java 1.8.0_181
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/out/diff-checkstyle-hbase-server.txt
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/testReport/
Max. process+thread count 4567 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/console
versions git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered by Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@saintstack saintstack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens to the procedure if the verify fails? Will the Procedure close out cleanly? Or will we it be perpetually rescheduled? Thanks.

@sanjeetnishad95
Copy link
Contributor Author

Hi @saintstack ,If the procedure fails as a result of verifySnapshot, it will throw corruptedSnapshotException or IOException which will be taken care as a part of the procedure framework. The procedure will close out cleanly.

@busbey
Copy link
Contributor

busbey commented Sep 28, 2019

Please check on the unit test failures since they're supposed to cover the code that changed.

@@ -351,6 +352,12 @@ private void prepareRestore(final MasterProcedureEnv env) throws IOException {
mfs.getFileSystem(),
SnapshotDescriptionUtils.getCompletedSnapshotDir(snapshot, mfs.getRootDir()),
snapshot);

// Verify snapshot validity
SnapshotReferenceUtil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is currently in a block for only checking non-system tables. Needs to happen for all tables.

Also should probably be the first thing we do in prepareRestore, since it used to proceed the step.

Please update the status monitor about doing the verification as well.

@saintstack
Copy link
Contributor

@sanjeetnishad95 See @busbey comments above? Any chance of addressing them? Thanks.

@sanjeetnishad95
Copy link
Contributor Author

yes @saintstack , I will address them ASAP.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 3m 31s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 💚 mvninstall 4m 31s master passed
+1 💚 checkstyle 1m 21s master passed
+1 💚 spotbugs 2m 32s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 14s the patch passed
-0 ⚠️ checkstyle 1m 16s hbase-server: The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 13m 34s Patch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚 spotbugs 2m 35s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 14s The patch does not generate ASF License warnings.
41m 57s
Subsystem Report/Notes
Docker Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #662
Optional Tests dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname Linux e7d2dcbeb9c4 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / e1d7dc8
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count 84 (vs. ulimit of 12500)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/console
versions git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 38s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 4m 17s master passed
+1 💚 compile 1m 4s master passed
+1 💚 shadedjars 5m 48s branch has no errors when building our shaded downstream artifacts.
-0 ⚠️ javadoc 0m 42s hbase-server in master failed.
_ Patch Compile Tests _
+1 💚 mvninstall 4m 5s the patch passed
+1 💚 compile 1m 3s the patch passed
+1 💚 javac 1m 3s the patch passed
+1 💚 shadedjars 5m 44s patch has no errors when building our shaded downstream artifacts.
-0 ⚠️ javadoc 0m 41s hbase-server in the patch failed.
_ Other Tests _
-1 ❌ unit 139m 19s hbase-server in the patch failed.
165m 20s
Subsystem Report/Notes
Docker Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #662
Optional Tests javac javadoc unit shadedjars compile
uname Linux 8e8876e9906d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / e1d7dc8
Default Java 2020-01-14
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/testReport/
Max. process+thread count 4098 (vs. ulimit of 12500)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/console
versions git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 26s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 4m 8s master passed
+1 💚 compile 1m 6s master passed
+1 💚 shadedjars 6m 18s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 49s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 4m 40s the patch passed
+1 💚 compile 1m 12s the patch passed
+1 💚 javac 1m 12s the patch passed
+1 💚 shadedjars 7m 33s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 43s the patch passed
_ Other Tests _
-1 ❌ unit 265m 9s hbase-server in the patch failed.
293m 44s
Subsystem Report/Notes
Docker Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #662
Optional Tests javac javadoc unit shadedjars compile
uname Linux ceb98a23fea8 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / e1d7dc8
Default Java 1.8.0_232
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/testReport/
Max. process+thread count 2862 (vs. ulimit of 12500)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/1/console
versions git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 30s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 💚 mvninstall 3m 45s master passed
+1 💚 checkstyle 1m 4s master passed
+1 💚 spotbugs 2m 1s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 21s the patch passed
-0 ⚠️ checkstyle 1m 5s hbase-server: The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 10m 59s Patch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚 spotbugs 2m 7s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 15s The patch does not generate ASF License warnings.
32m 10s
Subsystem Report/Notes
Docker Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #662
Optional Tests dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname Linux e139c5add47b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 2b894dc
checkstyle https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count 94 (vs. ulimit of 12500)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/console
versions git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 31s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 3m 45s master passed
+1 💚 compile 0m 54s master passed
+1 💚 shadedjars 5m 36s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 37s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 23s the patch passed
+1 💚 compile 0m 55s the patch passed
+1 💚 javac 0m 55s the patch passed
+1 💚 shadedjars 5m 33s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 38s the patch passed
_ Other Tests _
-1 ❌ unit 169m 30s hbase-server in the patch failed.
193m 21s
Subsystem Report/Notes
Docker Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #662
Optional Tests javac javadoc unit shadedjars compile
uname Linux e4d02cd1fed1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 2b894dc
Default Java 1.8.0_232
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/testReport/
Max. process+thread count 5480 (vs. ulimit of 12500)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/console
versions git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 56s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 6m 15s master passed
+1 💚 compile 1m 23s master passed
+1 💚 shadedjars 7m 40s branch has no errors when building our shaded downstream artifacts.
-0 ⚠️ javadoc 0m 52s hbase-server in master failed.
_ Patch Compile Tests _
+1 💚 mvninstall 5m 21s the patch passed
+1 💚 compile 1m 29s the patch passed
+1 💚 javac 1m 29s the patch passed
+1 💚 shadedjars 7m 44s patch has no errors when building our shaded downstream artifacts.
-0 ⚠️ javadoc 0m 58s hbase-server in the patch failed.
_ Other Tests _
-1 ❌ unit 209m 22s hbase-server in the patch failed.
244m 46s
Subsystem Report/Notes
Docker Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #662
Optional Tests javac javadoc unit shadedjars compile
uname Linux 210f721a6603 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 2b894dc
Default Java 2020-01-14
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadoc https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/testReport/
Max. process+thread count 3143 (vs. ulimit of 12500)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-662/2/console
versions git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@saintstack
Copy link
Contributor

Closing. No progress. Make a new PR if I have this wrong. Thanks.

@saintstack saintstack closed this Jun 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants