Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HDDS-1798. Propagate failure in writeStateMachineData to Ratis. Contributed by Supratim Deka #1113

Merged
merged 4 commits into from
Aug 5, 2019
Merged

HDDS-1798. Propagate failure in writeStateMachineData to Ratis. Contributed by Supratim Deka #1113

merged 4 commits into from
Aug 5, 2019

Conversation

supratimdeka
Copy link
Contributor

https://issues.apache.org/jira/browse/HDDS-1798

unit test is pending. Will add the infra required for testing this change as part of https://issues.apache.org/jira/browse/HDDS-1818

writeStateMachineData() currently returns a future to Ratis. The scope of this jira is to signal any errors or failures encountered as part of the operation - WriteChunk / handleWriteChunk() to Ratis via this future. As of now, the future does not track failures.

This change is required to complete the failure handling done in HDDS-1603.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 75 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 513 trunk passed
+1 compile 272 trunk passed
+1 checkstyle 70 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 948 branch has no errors when building and testing our client artifacts.
+1 javadoc 170 trunk passed
0 spotbugs 349 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 555 trunk passed
_ Patch Compile Tests _
+1 mvninstall 470 the patch passed
+1 compile 277 the patch passed
+1 javac 277 the patch passed
+1 checkstyle 80 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 741 patch has no errors when building and testing our client artifacts.
+1 javadoc 174 the patch passed
+1 findbugs 579 the patch passed
_ Other Tests _
+1 unit 347 hadoop-hdds in the patch passed.
-1 unit 2362 hadoop-ozone in the patch failed.
+1 asflicense 43 The patch does not generate ASF License warnings.
7830
Reason Tests
Failed junit tests hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory
hadoop.ozone.container.server.TestSecureContainerServer
hadoop.ozone.client.rpc.TestWatchForCommit
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
hadoop.ozone.om.TestSecureOzoneManager
hadoop.ozone.container.ozoneimpl.TestSecureOzoneContainer
hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient
hadoop.ozone.client.rpc.TestContainerStateMachineFailures
hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
hadoop.ozone.om.TestOzoneManagerHA
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures
hadoop.ozone.client.rpc.TestFailureHandlingByClient
hadoop.ozone.client.rpc.TestContainerStateMachine
hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.ozone.client.rpc.TestOzoneRpcClient
Subsystem Report/Notes
Docker Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1113/1/artifact/out/Dockerfile
GITHUB PR #1113
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux f4c8d4834b55 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 4e66cb9
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1113/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1113/1/testReport/
Max. process+thread count 5057 (vs. ulimit of 5500)
modules C: hadoop-hdds/container-service U: hadoop-hdds/container-service
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1113/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

mukul1987
mukul1987 reviewed Jul 22, 2019
View reviewed changes
Copy link
Contributor

@mukul1987 mukul1987 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes look good to me. Lets create a followup for a unit test for this.

...a/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java Show resolved Hide resolved
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 42 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 568 trunk passed
+1 compile 348 trunk passed
+1 checkstyle 63 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 807 branch has no errors when building and testing our client artifacts.
+1 javadoc 160 trunk passed
0 spotbugs 414 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 605 trunk passed
_ Patch Compile Tests _
+1 mvninstall 561 the patch passed
+1 compile 347 the patch passed
+1 javac 347 the patch passed
+1 checkstyle 70 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 621 patch has no errors when building and testing our client artifacts.
+1 javadoc 148 the patch passed
+1 findbugs 633 the patch passed
_ Other Tests _
-1 unit 293 hadoop-hdds in the patch failed.
-1 unit 1569 hadoop-ozone in the patch failed.
+1 asflicense 36 The patch does not generate ASF License warnings.
7011
Reason Tests
Failed junit tests hadoop.ozone.om.TestScmSafeMode
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1113/2/artifact/out/Dockerfile
GITHUB PR #1113
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 04b6a6feb7b7 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / aebac6d
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1113/2/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1113/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1113/2/testReport/
Max. process+thread count 4835 (vs. ulimit of 5500)
modules C: hadoop-hdds/container-service U: hadoop-hdds/container-service
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1113/2/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus
hadoop-yetus reviewed Aug 2, 2019
View reviewed changes
...gration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestContainerStateMachine.java Outdated
"Requested operation not allowed as ContainerState is UNHEALTHY"));
}
key.close();

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace:end of line

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 106 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
0 mvndep 68 Maven dependency ordering for branch
+1 mvninstall 628 trunk passed
+1 compile 382 trunk passed
+1 checkstyle 73 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 940 branch has no errors when building and testing our client artifacts.
+1 javadoc 191 trunk passed
0 spotbugs 482 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 726 trunk passed
_ Patch Compile Tests _
0 mvndep 33 Maven dependency ordering for patch
+1 mvninstall 634 the patch passed
+1 compile 427 the patch passed
+1 javac 427 the patch passed
+1 checkstyle 90 the patch passed
+1 mvnsite 0 the patch passed
-1 whitespace 0 The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 shadedclient 804 patch has no errors when building and testing our client artifacts.
+1 javadoc 171 the patch passed
+1 findbugs 667 the patch passed
_ Other Tests _
+1 unit 359 hadoop-hdds in the patch passed.
-1 unit 2101 hadoop-ozone in the patch failed.
+1 asflicense 43 The patch does not generate ASF License warnings.
8609
Reason Tests
Failed junit tests hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
hadoop.ozone.om.TestSecureOzoneManager
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.om.TestScmSafeMode
hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures
hadoop.ozone.TestMiniOzoneCluster
hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.ozone.client.rpc.TestOzoneRpcClient
Subsystem Report/Notes
Docker Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1113/3/artifact/out/Dockerfile
GITHUB PR #1113
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 18c1db4e2dff 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / d086d05
Default Java 1.8.0_212
whitespace https://builds.apache.org/job/hadoop-multibranch/job/PR-1113/3/artifact/out/whitespace-eol.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1113/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1113/3/testReport/
Max. process+thread count 5328 (vs. ulimit of 5500)
modules C: hadoop-hdds/container-service hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1113/3/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@bshashikant
Copy link
Contributor

/retest

@bshashikant bshashikant merged commit f8ea6e1 into apache:trunk Aug 5, 2019
@bshashikant
Copy link
Contributor

Thanks @supratimdeka for working on this and @mukul1987 for review. I have committed this change to trunk.

amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019
@supratimdeka
HDDS-1798. Propagate failure in writeStateMachineData to Ratis. Contr…
d4f9ccb 
…ibuted by Supratim Deka (apache#1113)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hadoop-yetus hadoop-yetus hadoop-yetus left review comments

@bshashikant bshashikant bshashikant left review comments

@mukul1987 mukul1987 mukul1987 left review comments

Assignees
No one assigned
Labels
ozone
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@supratimdeka @hadoop-yetus @bshashikant @mukul1987