HDDS-2169. Avoid buffer copies while submitting client requests in Ratis #1517

szetszwo · 2019-09-24T23:06:44Z

https://issues.apache.org/jira/browse/HDDS-2169

hadoop-yetus · 2019-09-24T23:48:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	98	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	11	Maven dependency ordering for branch
-1	mvninstall	29	hadoop-hdds in trunk failed.
-1	mvninstall	24	hadoop-ozone in trunk failed.
-1	compile	18	hadoop-hdds in trunk failed.
-1	compile	13	hadoop-ozone in trunk failed.
+1	checkstyle	49	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	919	branch has no errors when building and testing our client artifacts.
-1	javadoc	18	hadoop-hdds in trunk failed.
-1	javadoc	16	hadoop-ozone in trunk failed.
0	spotbugs	1000	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	27	hadoop-hdds in trunk failed.
-1	findbugs	16	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
0	mvndep	15	Maven dependency ordering for patch
-1	mvninstall	35	hadoop-hdds in the patch failed.
-1	mvninstall	29	hadoop-ozone in the patch failed.
-1	compile	21	hadoop-hdds in the patch failed.
-1	compile	16	hadoop-ozone in the patch failed.
-1	javac	21	hadoop-hdds in the patch failed.
-1	javac	16	hadoop-ozone in the patch failed.
-0	checkstyle	25	hadoop-hdds: The patch generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	767	patch has no errors when building and testing our client artifacts.
-1	javadoc	18	hadoop-hdds in the patch failed.
-1	javadoc	17	hadoop-ozone in the patch failed.
-1	findbugs	28	hadoop-hdds in the patch failed.
-1	findbugs	16	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	24	hadoop-hdds in the patch failed.
-1	unit	20	hadoop-ozone in the patch failed.
+1	asflicense	28	The patch does not generate ASF License warnings.
		2450

Subsystem	Report/Notes
Docker	Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/Dockerfile
GITHUB PR	#1517
JIRA Issue	HDDS-2169
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 655817f52c57 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `6917754`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/patch-compile-hadoop-ozone.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/diff-checkstyle-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/testReport/
Max. process+thread count	306 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-hdds/client hadoop-hdds/container-service U: hadoop-hdds
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/1/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-09-27T05:30:23Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	11	#1517 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#1517
JIRA Issue	HDDS-2169
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/2/console
versions	git=2.17.1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-09-27T06:13:25Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	11	#1517 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#1517
JIRA Issue	HDDS-2169
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/3/console
versions	git=2.17.1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-09-27T09:43:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	96	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	20	Maven dependency ordering for branch
-1	mvninstall	30	hadoop-hdds in trunk failed.
-1	mvninstall	25	hadoop-ozone in trunk failed.
-1	compile	18	hadoop-hdds in trunk failed.
-1	compile	13	hadoop-ozone in trunk failed.
+1	checkstyle	57	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	918	branch has no errors when building and testing our client artifacts.
-1	javadoc	20	hadoop-hdds in trunk failed.
-1	javadoc	16	hadoop-ozone in trunk failed.
0	spotbugs	1009	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	32	hadoop-hdds in trunk failed.
-1	findbugs	18	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
0	mvndep	14	Maven dependency ordering for patch
-1	mvninstall	33	hadoop-hdds in the patch failed.
-1	mvninstall	26	hadoop-ozone in the patch failed.
-1	compile	21	hadoop-hdds in the patch failed.
-1	compile	15	hadoop-ozone in the patch failed.
-1	javac	21	hadoop-hdds in the patch failed.
-1	javac	15	hadoop-ozone in the patch failed.
-0	checkstyle	24	hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	800	patch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in the patch failed.
-1	javadoc	16	hadoop-ozone in the patch failed.
-1	findbugs	28	hadoop-hdds in the patch failed.
-1	findbugs	17	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	24	hadoop-hdds in the patch failed.
-1	unit	20	hadoop-ozone in the patch failed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		2494

Subsystem	Report/Notes
Docker	Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/Dockerfile
GITHUB PR	#1517
JIRA Issue	HDDS-2169
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 569081994f2e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `8a9ede5`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/patch-compile-hadoop-ozone.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/diff-checkstyle-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/testReport/
Max. process+thread count	308 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-hdds/client hadoop-hdds/container-service U: hadoop-hdds
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/4/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-09-27T15:40:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	78	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	25	Maven dependency ordering for branch
-1	mvninstall	41	hadoop-hdds in trunk failed.
-1	mvninstall	26	hadoop-ozone in trunk failed.
-1	compile	19	hadoop-hdds in trunk failed.
-1	compile	12	hadoop-ozone in trunk failed.
+1	checkstyle	61	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	947	branch has no errors when building and testing our client artifacts.
-1	javadoc	20	hadoop-hdds in trunk failed.
-1	javadoc	16	hadoop-ozone in trunk failed.
0	spotbugs	1036	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	32	hadoop-hdds in trunk failed.
-1	findbugs	16	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
0	mvndep	13	Maven dependency ordering for patch
-1	mvninstall	33	hadoop-hdds in the patch failed.
-1	mvninstall	26	hadoop-ozone in the patch failed.
-1	compile	20	hadoop-hdds in the patch failed.
-1	compile	15	hadoop-ozone in the patch failed.
-1	javac	20	hadoop-hdds in the patch failed.
-1	javac	15	hadoop-ozone in the patch failed.
+1	checkstyle	50	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	778	patch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in the patch failed.
-1	javadoc	16	hadoop-ozone in the patch failed.
-1	findbugs	28	hadoop-hdds in the patch failed.
-1	findbugs	16	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	24	hadoop-hdds in the patch failed.
-1	unit	20	hadoop-ozone in the patch failed.
+1	asflicense	30	The patch does not generate ASF License warnings.
		2492

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/Dockerfile
GITHUB PR	#1517
JIRA Issue	HDDS-2169
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 62979f013a8d 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `ce58c05`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/testReport/
Max. process+thread count	306 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-hdds/client hadoop-hdds/container-service U: hadoop-hdds
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/5/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

bshashikant · 2019-09-30T06:54:48Z

/retest

bshashikant · 2019-09-30T16:32:03Z

Thanks @szetszwo for working on this. With the patch, while running the tests in TestDataValidateWithUnsafeByteOperations, the below issue is observed.

2019-09-30 21:58:02,745 [grpc-default-executor-2] ERROR segmented.SegmentedRaftLogWorker (SegmentedRaftLogWorker.java:<init>(449)) - e4ab8454-30fe-420c-a1cf-40d223cb4898@group-D0335C23E8DA-SegmentedRaftLogWorker: writeStateMachineData failed for index 1, entry=(t:1, i:1), STATEMACHINELOGENTRY, client-6C45A0D09519, cid=8 java.lang.IndexOutOfBoundsException: End index: 135008824 >= 207 at org.apache.ratis.thirdparty.com.google.protobuf.ByteString.checkRange(ByteString.java:1233) at org.apache.ratis.thirdparty.com.google.protobuf.ByteString$LiteralByteString.substring(ByteString.java:1288) at org.apache.hadoop.hdds.ratis.ContainerCommandRequestMessage.toProto(ContainerCommandRequestMessage.java:66) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.getContainerCommandRequestProto(ContainerStateMachine.java:375) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.writeStateMachineData(ContainerStateMachine.java:494) at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker$WriteLog.<init>(SegmentedRaftLogWorker.java:447) at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker.writeLogEntry(SegmentedRaftLogWorker.java:397) at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.appendEntryImpl(SegmentedRaftLog.java:411) at org.apache.ratis.server.raftlog.RaftLog.lambda$appendEntry$10(RaftLog.java:359) at org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:77) at org.apache.ratis.server.raftlog.RaftLog.appendEntry(RaftLog.java:359) at org.apache.ratis.server.raftlog.RaftLog.appendImpl(RaftLog.java:183) at org.apache.ratis.server.raftlog.RaftLog.lambda$append$2(RaftLog.java:159) at org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:68) at org.apache.ratis.server.raftlog.RaftLog.append(RaftLog.java:159) at org.apache.ratis.server.impl.ServerState.appendLog(ServerState.java:282) at org.apache.ratis.server.impl.RaftServerImpl.appendTransaction(RaftServerImpl.java:505) at org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:576) at org.apache.ratis.server.impl.RaftServerProxy.lambda$submitClientRequestAsync$7(RaftServerProxy.java:333) at org.apache.ratis.server.impl.RaftServerProxy.lambda$null$5(RaftServerProxy.java:328) at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:109) at org.apache.ratis.server.impl.RaftServerProxy.lambda$submitRequest$6(RaftServerProxy.java:328) at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) at org.apache.ratis.server.impl.RaftServerProxy.submitRequest(RaftServerProxy.java:327) at org.apache.ratis.server.impl.RaftServerProxy.submitClientRequestAsync(RaftServerProxy.java:333) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:220) at org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:326) at org.apache.ratis.util.SlidingWindow$Server.processRequestsFromHead(SlidingWindow.java:429) at org.apache.ratis.util.SlidingWindow$Server.receivedRequest(SlidingWindow.java:421) at org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:345) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:240) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:168) at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:248) at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:263) at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:686) at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

We need to change to code in ContainerStateMachine to parse the data correctly in writeStateMachineData. Can you please check?

hadoop-yetus · 2019-09-30T17:37:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	2244	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
0	mvndep	36	Maven dependency ordering for branch
-1	mvninstall	35	hadoop-hdds in trunk failed.
-1	mvninstall	38	hadoop-ozone in trunk failed.
-1	compile	19	hadoop-hdds in trunk failed.
-1	compile	13	hadoop-ozone in trunk failed.
+1	checkstyle	59	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	940	branch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in trunk failed.
-1	javadoc	17	hadoop-ozone in trunk failed.
0	spotbugs	1028	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	30	hadoop-hdds in trunk failed.
-1	findbugs	17	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
0	mvndep	15	Maven dependency ordering for patch
-1	mvninstall	32	hadoop-hdds in the patch failed.
-1	mvninstall	34	hadoop-ozone in the patch failed.
-1	compile	22	hadoop-hdds in the patch failed.
-1	compile	15	hadoop-ozone in the patch failed.
-1	javac	22	hadoop-hdds in the patch failed.
-1	javac	15	hadoop-ozone in the patch failed.
-0	checkstyle	25	hadoop-hdds: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	789	patch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in the patch failed.
-1	javadoc	16	hadoop-ozone in the patch failed.
-1	findbugs	29	hadoop-hdds in the patch failed.
-1	findbugs	17	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	25	hadoop-hdds in the patch failed.
-1	unit	22	hadoop-ozone in the patch failed.
+1	asflicense	30	The patch does not generate ASF License warnings.
		4686

Subsystem	Report/Notes
Docker	Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/Dockerfile
JIRA Issue	HDDS-2169
JIRA Patch URL	https://issues.apache.org/jira/secure/attachment/12981115/o2169_20190923.patch
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 11b7ced0c5c4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `98ca07e`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/patch-compile-hadoop-ozone.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/diff-checkstyle-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/testReport/
Max. process+thread count	306 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-hdds/client hadoop-hdds/container-service U: hadoop-hdds
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

szetszwo · 2019-10-01T02:50:48Z

Thanks @szetszwo for working on this. With the patch, while running the tests in TestDataValidateWithUnsafeByteOperations, the below issue is observed.

2019-09-30 21:58:02,745 [grpc-default-executor-2] ERROR segmented.SegmentedRaftLogWorker (SegmentedRaftLogWorker.java:<init>(449)) - e4ab8454-30fe-420c-a1cf-40d223cb4898@group-D0335C23E8DA-SegmentedRaftLogWorker: writeStateMachineData failed for index 1, entry=(t:1, i:1), STATEMACHINELOGENTRY, client-6C45A0D09519, cid=8 java.lang.IndexOutOfBoundsException: End index: 135008824 >= 207 at org.apache.ratis.thirdparty.com.google.protobuf.ByteString.checkRange(ByteString.java:1233) at org.apache.ratis.thirdparty.com.google.protobuf.ByteString$LiteralByteString.substring(ByteString.java:1288) at org.apache.hadoop.hdds.ratis.ContainerCommandRequestMessage.toProto(ContainerCommandRequestMessage.java:66) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.getContainerCommandRequestProto(ContainerStateMachine.java:375) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.writeStateMachineData(ContainerStateMachine.java:494) at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker$WriteLog.<init>(SegmentedRaftLogWorker.java:447) at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker.writeLogEntry(SegmentedRaftLogWorker.java:397) at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.appendEntryImpl(SegmentedRaftLog.java:411) at org.apache.ratis.server.raftlog.RaftLog.lambda$appendEntry$10(RaftLog.java:359) at org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:77) at org.apache.ratis.server.raftlog.RaftLog.appendEntry(RaftLog.java:359) at org.apache.ratis.server.raftlog.RaftLog.appendImpl(RaftLog.java:183) at org.apache.ratis.server.raftlog.RaftLog.lambda$append$2(RaftLog.java:159) at org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:68) at org.apache.ratis.server.raftlog.RaftLog.append(RaftLog.java:159) at org.apache.ratis.server.impl.ServerState.appendLog(ServerState.java:282) at org.apache.ratis.server.impl.RaftServerImpl.appendTransaction(RaftServerImpl.java:505) at org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:576) at org.apache.ratis.server.impl.RaftServerProxy.lambda$submitClientRequestAsync$7(RaftServerProxy.java:333) at org.apache.ratis.server.impl.RaftServerProxy.lambda$null$5(RaftServerProxy.java:328) at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:109) at org.apache.ratis.server.impl.RaftServerProxy.lambda$submitRequest$6(RaftServerProxy.java:328) at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) at org.apache.ratis.server.impl.RaftServerProxy.submitRequest(RaftServerProxy.java:327) at org.apache.ratis.server.impl.RaftServerProxy.submitClientRequestAsync(RaftServerProxy.java:333) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:220) at org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:326) at org.apache.ratis.util.SlidingWindow$Server.processRequestsFromHead(SlidingWindow.java:429) at org.apache.ratis.util.SlidingWindow$Server.receivedRequest(SlidingWindow.java:421) at org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:345) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:240) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:168) at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:248) at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:263) at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:686) at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

We need to change to code in ContainerStateMachine to parse the data correctly in writeStateMachineData. Can you please check?

Thank @bshashikant for catching the bug. The ByteString from LogEntryProto should remain unchanged. Only the message from client requests need to be changed. Just have addressed this bug in

commit e6b25d101818031b64e61764aa6ce712250a1541 (HEAD -> trunk, origin/trunk, origin/HEAD)
Author: Tsz Wo Nicholas Sze szetszwo@apache.org
Date: Tue Oct 1 10:47:10 2019 +0800

Fix a bug in ContainerStateMachine.

hadoop-yetus · 2019-10-01T03:31:47Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	96	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	20	Maven dependency ordering for branch
-1	mvninstall	31	hadoop-hdds in trunk failed.
-1	mvninstall	34	hadoop-ozone in trunk failed.
-1	compile	19	hadoop-hdds in trunk failed.
-1	compile	13	hadoop-ozone in trunk failed.
+1	checkstyle	59	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	971	branch has no errors when building and testing our client artifacts.
-1	javadoc	18	hadoop-hdds in trunk failed.
-1	javadoc	16	hadoop-ozone in trunk failed.
0	spotbugs	1058	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	31	hadoop-hdds in trunk failed.
-1	findbugs	17	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
0	mvndep	14	Maven dependency ordering for patch
-1	mvninstall	33	hadoop-hdds in the patch failed.
-1	mvninstall	36	hadoop-ozone in the patch failed.
-1	compile	21	hadoop-hdds in the patch failed.
-1	compile	16	hadoop-ozone in the patch failed.
-1	javac	21	hadoop-hdds in the patch failed.
-1	javac	17	hadoop-ozone in the patch failed.
+1	checkstyle	54	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	779	patch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in the patch failed.
-1	javadoc	16	hadoop-ozone in the patch failed.
-1	findbugs	28	hadoop-hdds in the patch failed.
-1	findbugs	18	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	24	hadoop-hdds in the patch failed.
-1	unit	23	hadoop-ozone in the patch failed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		2550

Subsystem	Report/Notes
Docker	Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/Dockerfile
GITHUB PR	#1517
JIRA Issue	HDDS-2169
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux f83947a622e3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `b3275ab`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/testReport/
Max. process+thread count	340 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-hdds/client hadoop-hdds/container-service U: hadoop-hdds
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

bshashikant · 2019-10-01T12:51:08Z

/retest

hadoop-yetus · 2019-10-02T04:01:50Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	76	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	20	Maven dependency ordering for branch
-1	mvninstall	29	hadoop-hdds in trunk failed.
-1	mvninstall	34	hadoop-ozone in trunk failed.
-1	compile	19	hadoop-hdds in trunk failed.
-1	compile	13	hadoop-ozone in trunk failed.
+1	checkstyle	57	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	934	branch has no errors when building and testing our client artifacts.
-1	javadoc	18	hadoop-hdds in trunk failed.
-1	javadoc	16	hadoop-ozone in trunk failed.
0	spotbugs	1020	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	30	hadoop-hdds in trunk failed.
-1	findbugs	17	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
0	mvndep	14	Maven dependency ordering for patch
-1	mvninstall	31	hadoop-hdds in the patch failed.
-1	mvninstall	34	hadoop-ozone in the patch failed.
-1	compile	22	hadoop-hdds in the patch failed.
-1	compile	16	hadoop-ozone in the patch failed.
-1	javac	22	hadoop-hdds in the patch failed.
-1	javac	16	hadoop-ozone in the patch failed.
+1	checkstyle	52	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	780	patch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in the patch failed.
-1	javadoc	16	hadoop-ozone in the patch failed.
-1	findbugs	28	hadoop-hdds in the patch failed.
-1	findbugs	17	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	24	hadoop-hdds in the patch failed.
-1	unit	23	hadoop-ozone in the patch failed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		2485

Subsystem	Report/Notes
Docker	Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/Dockerfile
GITHUB PR	#1517
JIRA Issue	HDDS-2169
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux fddbf2c00bca 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `3df733c`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/testReport/
Max. process+thread count	308 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-hdds/client hadoop-hdds/container-service U: hadoop-hdds
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/8/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-10-02T05:55:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	91	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	11	Maven dependency ordering for branch
-1	mvninstall	30	hadoop-hdds in trunk failed.
-1	mvninstall	32	hadoop-ozone in trunk failed.
-1	compile	18	hadoop-hdds in trunk failed.
-1	compile	13	hadoop-ozone in trunk failed.
+1	checkstyle	48	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	923	branch has no errors when building and testing our client artifacts.
-1	javadoc	20	hadoop-hdds in trunk failed.
-1	javadoc	17	hadoop-ozone in trunk failed.
0	spotbugs	1012	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	32	hadoop-hdds in trunk failed.
-1	findbugs	16	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
0	mvndep	13	Maven dependency ordering for patch
-1	mvninstall	32	hadoop-hdds in the patch failed.
-1	mvninstall	36	hadoop-ozone in the patch failed.
-1	compile	20	hadoop-hdds in the patch failed.
-1	compile	15	hadoop-ozone in the patch failed.
-1	javac	20	hadoop-hdds in the patch failed.
-1	javac	15	hadoop-ozone in the patch failed.
-0	checkstyle	25	hadoop-hdds: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	811	patch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in the patch failed.
-1	javadoc	16	hadoop-ozone in the patch failed.
-1	findbugs	27	hadoop-hdds in the patch failed.
-1	findbugs	17	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	24	hadoop-hdds in the patch failed.
-1	unit	23	hadoop-ozone in the patch failed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		2508

Subsystem	Report/Notes
Docker	Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/Dockerfile
GITHUB PR	#1517
JIRA Issue	HDDS-2169
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 33dae9ca5fa9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `41440ec`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/patch-compile-hadoop-ozone.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/diff-checkstyle-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/testReport/
Max. process+thread count	306 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-hdds/client hadoop-hdds/container-service U: hadoop-hdds
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/9/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-10-02T07:18:56Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	78	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	11	Maven dependency ordering for branch
-1	mvninstall	29	hadoop-hdds in trunk failed.
-1	mvninstall	32	hadoop-ozone in trunk failed.
-1	compile	19	hadoop-hdds in trunk failed.
-1	compile	13	hadoop-ozone in trunk failed.
+1	checkstyle	47	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	937	branch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in trunk failed.
-1	javadoc	16	hadoop-ozone in trunk failed.
0	spotbugs	1021	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	28	hadoop-hdds in trunk failed.
-1	findbugs	17	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
0	mvndep	13	Maven dependency ordering for patch
-1	mvninstall	33	hadoop-hdds in the patch failed.
-1	mvninstall	36	hadoop-ozone in the patch failed.
-1	compile	21	hadoop-hdds in the patch failed.
-1	compile	15	hadoop-ozone in the patch failed.
-1	javac	21	hadoop-hdds in the patch failed.
-1	javac	15	hadoop-ozone in the patch failed.
+1	checkstyle	52	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	785	patch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in the patch failed.
-1	javadoc	17	hadoop-ozone in the patch failed.
-1	findbugs	28	hadoop-hdds in the patch failed.
-1	findbugs	17	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	24	hadoop-hdds in the patch failed.
-1	unit	22	hadoop-ozone in the patch failed.
+1	asflicense	30	The patch does not generate ASF License warnings.
		2485

Subsystem	Report/Notes
Docker	Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/Dockerfile
GITHUB PR	#1517
JIRA Issue	HDDS-2169
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux c86c58047a36 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `41440ec`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/testReport/
Max. process+thread count	352 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-hdds/client hadoop-hdds/container-service U: hadoop-hdds
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/10/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

bshashikant · 2019-10-02T14:02:10Z

Thanks @szetszwo for updating the patch. i think We need to change the parsing logic applyTransaction call in ContainerStateMachine as well because the integration test failures are related here:
Caused by: org.apache.ratis.protocol.StateMachineException: org.apache.ratis.thirdparty.com.google.protobuf.InvalidProtocolBufferException from Server bad6dc6c-9418-448d-ad47-5f803b5e863b@group-6CEFB1F28D35: Protocol message contained an invalid tag (zero). at org.apache.ratis.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:105) at org.apache.ratis.thirdparty.com.google.protobuf.CodedInputStream$ArrayDecoder.readTag(CodedInputStream.java:646) at org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto.<init>(ContainerProtos.java:2692) at org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto.<init>(ContainerProtos.java:2655) at org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto$1.parsePartialFrom(ContainerProtos.java:7272) at org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto$1.parsePartialFrom(ContainerProtos.java:7267) at org.apache.ratis.thirdparty.com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:105) at org.apache.ratis.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:125) at org.apache.ratis.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:130) at org.apache.ratis.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49) at org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto.parseFrom(ContainerProtos.java:4110) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.getContainerCommandRequestProto(ContainerStateMachine.java:383) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.applyTransaction(ContainerStateMachine.java:708) at org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1274) at org.apache.ratis.server.impl.StateMachineUpdater.applyLog(StateMachineUpdater.java:224) at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:165)

szetszwo · 2019-10-02T14:22:05Z

Thanks @szetszwo for updating the patch. i think We need to change the parsing logic applyTransaction call in ContainerStateMachine as well because the integration test failures are related here:
Caused by: org.apache.ratis.protocol.StateMachineException: org.apache.ratis.thirdparty.com.google.protobuf.InvalidProtocolBufferException from Server bad6dc6c-9418-448d-ad47-5f803b5e863b@group-6CEFB1F28D35: Protocol message contained an invalid tag (zero). at org.apache.ratis.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:105) at org.apache.ratis.thirdparty.com.google.protobuf.CodedInputStream$ArrayDecoder.readTag(CodedInputStream.java:646) at org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto.<init>(ContainerProtos.java:2692) at org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto.<init>(ContainerProtos.java:2655) at org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto$1.parsePartialFrom(ContainerProtos.java:7272) at org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto$1.parsePartialFrom(ContainerProtos.java:7267) at org.apache.ratis.thirdparty.com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:105) at org.apache.ratis.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:125) at org.apache.ratis.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:130) at org.apache.ratis.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49) at org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto.parseFrom(ContainerProtos.java:4110) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.getContainerCommandRequestProto(ContainerStateMachine.java:383) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.applyTransaction(ContainerStateMachine.java:708) at org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1274) at org.apache.ratis.server.impl.StateMachineUpdater.applyLog(StateMachineUpdater.java:224) at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:165)

Oops, need one more fix (a4ae0852f818f6a49403b784790b1babb3df71f5) as shown below since it calls request.getMessage().getContent() again in startTransaction(..). Thanks @bshashikant.

+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java
@@ -364,7 +364,7 @@ public TransactionContext startTransaction(RaftClientRequest request)
.setStateMachine(this)
.setServerRole(RaftPeerRole.LEADER)
.setStateMachineContext(startTime)
.setLogData(request.getMessage().getContent())
.build();
}

hadoop-yetus · 2019-10-02T15:02:54Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	80	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	11	Maven dependency ordering for branch
-1	mvninstall	30	hadoop-hdds in trunk failed.
-1	mvninstall	32	hadoop-ozone in trunk failed.
-1	compile	19	hadoop-hdds in trunk failed.
-1	compile	13	hadoop-ozone in trunk failed.
+1	checkstyle	47	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	930	branch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in trunk failed.
-1	javadoc	16	hadoop-ozone in trunk failed.
0	spotbugs	1015	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	29	hadoop-hdds in trunk failed.
-1	findbugs	17	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
0	mvndep	14	Maven dependency ordering for patch
-1	mvninstall	33	hadoop-hdds in the patch failed.
-1	mvninstall	34	hadoop-ozone in the patch failed.
-1	compile	22	hadoop-hdds in the patch failed.
-1	compile	15	hadoop-ozone in the patch failed.
-1	javac	22	hadoop-hdds in the patch failed.
-1	javac	15	hadoop-ozone in the patch failed.
+1	checkstyle	51	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	798	patch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in the patch failed.
-1	javadoc	16	hadoop-ozone in the patch failed.
-1	findbugs	28	hadoop-hdds in the patch failed.
-1	findbugs	17	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	24	hadoop-hdds in the patch failed.
-1	unit	23	hadoop-ozone in the patch failed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		2490

Subsystem	Report/Notes
Docker	Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/Dockerfile
GITHUB PR	#1517
JIRA Issue	HDDS-2169
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 0e56f7dcac72 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `61a8436`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/testReport/
Max. process+thread count	305 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-hdds/client hadoop-hdds/container-service U: hadoop-hdds
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/11/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

bshashikant · 2019-10-04T17:45:38Z

Thanks @szetszwo for updating the patch. I tried to run the tests in TestDataValidateWithUnsafeByteOperations and i see the following exception being thrown:
2019-10-04 23:13:25,556 [ce18dfb1-da4d-401f-9614-bec32477b5f3@group-0099BCD205B6-SegmentedRaftLogWorker] INFO segmented.SegmentedRaftLogWorker (SegmentedRaftLogWorker.java:execute(574)) - ce18dfb1-da4d-401f-9614-bec32477b5f3@group-0099BCD205B6-SegmentedRaftLogWorker: created new log segment /Users/sbanerjee/github_hadoop/hadoop-ozone/tools/target/test-dir/MiniOzoneClusterImpl-cd3ca672-68cd-49fd-bdb3-a7fc97d18c23/datanode-1/data/ratis/ace05abb-b740-47f7-95d4-0099bcd205b6/current/log_inprogress_0 2019-10-04 23:13:25,557 [ee7f2721-1de4-4264-8bf3-d340e83f8791@group-0099BCD205B6-SegmentedRaftLogWorker] INFO segmented.SegmentedRaftLogWorker (SegmentedRaftLogWorker.java:execute(574)) - ee7f2721-1de4-4264-8bf3-d340e83f8791@group-0099BCD205B6-SegmentedRaftLogWorker: created new log segment /Users/sbanerjee/github_hadoop/hadoop-ozone/tools/target/test-dir/MiniOzoneClusterImpl-cd3ca672-68cd-49fd-bdb3-a7fc97d18c23/datanode-2/data/ratis/ace05abb-b740-47f7-95d4-0099bcd205b6/current/log_inprogress_0 2019-10-04 23:13:25,874 [pool-56-thread-1] ERROR impl.ChunkManagerImpl (ChunkUtils.java:writeData(89)) - data array does not match the length specified. DataLen: 1048576 Byte Array: 1048749 2019-10-04 23:13:25,874 [pool-56-thread-1] INFO keyvalue.KeyValueHandler (ContainerUtils.java:logAndReturnError(146)) - Operation: WriteChunk : Trace ID: cab5af5eafbad5ed:6a87e816d7e0ce20:e3ff42a900c31035:0 : Message: data array does not match the length specified. DataLen: 1048576 Byte Array: 1048749 : Result: INVALID_WRITE_SIZE 2019-10-04 23:13:25,881 [EventQueue-IncrementalContainerReportForIncrementalContainerReportHandler] WARN container.IncrementalContainerReportHandler (AbstractContainerReportHandler.java:updateContainerState(143)) - Container #1 is in OPEN state, but the datanode eb79af53-823f-485d-8402-ff71443cc79f{ip: 192.168.0.64, host: 192.168.0.64, networkLocation: /default-rack, certSerialId: null} reports an UNHEALTHY replica. 23:13:25.886 [pool-56-thread-1] ERROR DNAudit - user=null | ip=null | op=WRITE_CHUNK {blockData=conID: 1 locID: 102905348118937600 bcsId: 0} | ret=FAILURE java.lang.Exception: data array does not match the length specified. DataLen: 1048576 Byte Array: 1048749 at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:330) ~[classes/:?] at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:150) ~[classes/:?] at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:411) ~[classes/:?] at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:419) ~[classes/:?] at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$handleWriteChunk$1(ContainerStateMachine.java:454) ~[classes/:?] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] 2019-10-04 23:13:25,896 [pool-56-thread-1] ERROR ratis.ContainerStateMachine (ContainerStateMachine.java:lambda$handleWriteChunk$2(474)) - group-0099BCD205B6: writeChunk writeStateMachineData failed: blockIdcontainerID: 1 localID: 102905348118937600 blockCommitSequenceId: 0 logIndex 1 chunkName 102905348118937600_chunk_1 Error message: data array does not match the length specified. DataLen: 1048576 Byte Array: 1048749 Container Result: INVALID_WRITE_SIZE

Can you please check?

szetszwo · 2019-10-05T01:14:00Z

... I tried to run the tests in TestDataValidateWithUnsafeByteOperations and i see the following exception being thrown: ...

Thanks @bshashikant. That is a bug in ChunkUtils

 // ChunkUtils.writeData(..)
    int bufferSize = data.capacity();

It should call data.remaining() instead of data.capacity().

hadoop-yetus · 2019-10-05T01:59:22Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	85	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	1	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	26	Maven dependency ordering for branch
-1	mvninstall	36	hadoop-hdds in trunk failed.
-1	mvninstall	36	hadoop-ozone in trunk failed.
-1	compile	19	hadoop-hdds in trunk failed.
-1	compile	13	hadoop-ozone in trunk failed.
+1	checkstyle	60	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	964	branch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in trunk failed.
-1	javadoc	16	hadoop-ozone in trunk failed.
0	spotbugs	1051	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	31	hadoop-hdds in trunk failed.
-1	findbugs	17	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
0	mvndep	14	Maven dependency ordering for patch
-1	mvninstall	32	hadoop-hdds in the patch failed.
-1	mvninstall	34	hadoop-ozone in the patch failed.
-1	compile	22	hadoop-hdds in the patch failed.
-1	compile	15	hadoop-ozone in the patch failed.
-1	javac	22	hadoop-hdds in the patch failed.
-1	javac	15	hadoop-ozone in the patch failed.
-0	checkstyle	25	hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	806	patch has no errors when building and testing our client artifacts.
-1	javadoc	19	hadoop-hdds in the patch failed.
-1	javadoc	16	hadoop-ozone in the patch failed.
-1	findbugs	28	hadoop-hdds in the patch failed.
-1	findbugs	17	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	24	hadoop-hdds in the patch failed.
-1	unit	23	hadoop-ozone in the patch failed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		2564

Subsystem	Report/Notes
Docker	Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/Dockerfile
GITHUB PR	#1517
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 1131b3349042 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `f209722`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/patch-compile-hadoop-ozone.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/diff-checkstyle-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/testReport/
Max. process+thread count	306 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-hdds/client hadoop-hdds/container-service U: hadoop-hdds
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/12/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

…tis.

hadoop-yetus · 2019-10-05T03:53:09Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	99	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	13	Maven dependency ordering for branch
-1	mvninstall	30	hadoop-hdds in trunk failed.
-1	mvninstall	34	hadoop-ozone in trunk failed.
-1	compile	19	hadoop-hdds in trunk failed.
-1	compile	12	hadoop-ozone in trunk failed.
+1	checkstyle	48	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	936	branch has no errors when building and testing our client artifacts.
-1	javadoc	20	hadoop-hdds in trunk failed.
-1	javadoc	16	hadoop-ozone in trunk failed.
0	spotbugs	1023	Used deprecated FindBugs config; considering switching to SpotBugs.
-1	findbugs	30	hadoop-hdds in trunk failed.
-1	findbugs	17	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
0	mvndep	14	Maven dependency ordering for patch
-1	mvninstall	34	hadoop-hdds in the patch failed.
-1	mvninstall	35	hadoop-ozone in the patch failed.
-1	compile	22	hadoop-hdds in the patch failed.
-1	compile	16	hadoop-ozone in the patch failed.
-1	javac	22	hadoop-hdds in the patch failed.
-1	javac	16	hadoop-ozone in the patch failed.
+1	checkstyle	55	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	807	patch has no errors when building and testing our client artifacts.
-1	javadoc	29	hadoop-hdds in the patch failed.
-1	javadoc	20	hadoop-ozone in the patch failed.
-1	findbugs	45	hadoop-hdds in the patch failed.
-1	findbugs	23	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	35	hadoop-hdds in the patch failed.
-1	unit	32	hadoop-ozone in the patch failed.
+1	asflicense	38	The patch does not generate ASF License warnings.
		2610

Subsystem	Report/Notes
Docker	Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/Dockerfile
GITHUB PR	#1517
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 4562c496feab 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `f209722`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/branch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/branch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/branch-findbugs-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/patch-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/patch-findbugs-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/testReport/
Max. process+thread count	309 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-hdds/client hadoop-hdds/container-service U: hadoop-hdds
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/13/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

…tis. Contributed by Tsz-wo Sze(#1517).

bshashikant · 2019-10-07T04:08:51Z

Thanks @szetszwo for working on this. I have added the relavent change below as well while submitting the close container request.
`index 179547b8444..80e91cdf55d 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java
@@ -27,6 +27,7 @@
import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.PipelineReport;
import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.ClosePipelineInfo;
import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.PipelineAction;
+import org.apache.hadoop.hdds.ratis.ContainerCommandRequestMessage;
import org.apache.hadoop.hdds.scm.HddsServerUtil;
import org.apache.hadoop.hdds.scm.pipeline.PipelineID;
import org.apache.hadoop.hdds.security.x509.SecurityConfig;
@@ -516,8 +517,8 @@ private RaftClientRequest createRaftClientRequest(
RaftClientRequest.Type type) {
return new RaftClientRequest(clientId, server.getId(),
RaftGroupId.valueOf(PipelineID.getFromProtobuf(pipelineID).getId()),

   nextCallId(), Message.valueOf(request.toByteString()), type,

```
   null);
```

   nextCallId(), ContainerCommandRequestMessage.toMessage(request, null),

```
   type, null);
```
}

`
I have verified the unit tests and committed this.

szetszwo · 2019-10-07T10:19:14Z

Thanks @bshashikant !

…tis. Contributed by Tsz-wo Sze(apache#1517).

szetszwo force-pushed the trunk branch from 57de4c6 to 6538c19 Compare September 27, 2019 08:52

szetszwo force-pushed the trunk branch from 6538c19 to 664138a Compare September 27, 2019 14:57

bshashikant added the ozone label Sep 30, 2019

szetszwo force-pushed the trunk branch from e6b25d1 to 9939a42 Compare October 2, 2019 03:19

szetszwo force-pushed the trunk branch from 9939a42 to 9fbac5b Compare October 2, 2019 05:12

szetszwo force-pushed the trunk branch from 9fbac5b to 855d584 Compare October 2, 2019 06:36

szetszwo force-pushed the trunk branch from 855d584 to a4ae085 Compare October 2, 2019 14:20

elek changed the title ~~HDDS-2169~~ HDDS-2169. Avoid buffer copies while submitting client requests in Ratis Oct 2, 2019

szetszwo force-pushed the trunk branch from a4ae085 to a863ec1 Compare October 5, 2019 01:15

HDDS-2169. Avoid buffer copies while submitting client requests in Ra…

787ac74

…tis.

szetszwo force-pushed the trunk branch from a863ec1 to 787ac74 Compare October 5, 2019 02:21

asfgit pushed a commit that referenced this pull request Oct 7, 2019

HDDS-2169. Avoid buffer copies while submitting client requests in Ra…

022fe5f

…tis. Contributed by Tsz-wo Sze(#1517).

bshashikant closed this Oct 7, 2019

amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019

HDDS-2169. Avoid buffer copies while submitting client requests in Ra…

15a7368

…tis. Contributed by Tsz-wo Sze(apache#1517).

RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020

HDDS-2169. Avoid buffer copies while submitting client requests in Ra…

2bbf419

…tis. Contributed by Tsz-wo Sze(apache#1517).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-2169. Avoid buffer copies while submitting client requests in Ratis #1517

HDDS-2169. Avoid buffer copies while submitting client requests in Ratis #1517

szetszwo commented Sep 24, 2019 •

edited

Loading

hadoop-yetus commented Sep 24, 2019

hadoop-yetus commented Sep 27, 2019

hadoop-yetus commented Sep 27, 2019

hadoop-yetus commented Sep 27, 2019

hadoop-yetus commented Sep 27, 2019

bshashikant commented Sep 30, 2019

bshashikant commented Sep 30, 2019 •

edited

Loading

hadoop-yetus commented Sep 30, 2019

szetszwo commented Oct 1, 2019

hadoop-yetus commented Oct 1, 2019

bshashikant commented Oct 1, 2019

hadoop-yetus commented Oct 2, 2019

hadoop-yetus commented Oct 2, 2019

hadoop-yetus commented Oct 2, 2019

bshashikant commented Oct 2, 2019

szetszwo commented Oct 2, 2019 •

edited

Loading

hadoop-yetus commented Oct 2, 2019

bshashikant commented Oct 4, 2019

szetszwo commented Oct 5, 2019

hadoop-yetus commented Oct 5, 2019

hadoop-yetus commented Oct 5, 2019

bshashikant commented Oct 7, 2019

szetszwo commented Oct 7, 2019

HDDS-2169. Avoid buffer copies while submitting client requests in Ratis #1517

HDDS-2169. Avoid buffer copies while submitting client requests in Ratis #1517

Conversation

szetszwo commented Sep 24, 2019 • edited Loading

hadoop-yetus commented Sep 24, 2019

hadoop-yetus commented Sep 27, 2019

hadoop-yetus commented Sep 27, 2019

hadoop-yetus commented Sep 27, 2019

hadoop-yetus commented Sep 27, 2019

bshashikant commented Sep 30, 2019

bshashikant commented Sep 30, 2019 • edited Loading

hadoop-yetus commented Sep 30, 2019

szetszwo commented Oct 1, 2019

hadoop-yetus commented Oct 1, 2019

bshashikant commented Oct 1, 2019

hadoop-yetus commented Oct 2, 2019

hadoop-yetus commented Oct 2, 2019

hadoop-yetus commented Oct 2, 2019

bshashikant commented Oct 2, 2019

szetszwo commented Oct 2, 2019 • edited Loading

hadoop-yetus commented Oct 2, 2019

bshashikant commented Oct 4, 2019

szetszwo commented Oct 5, 2019

hadoop-yetus commented Oct 5, 2019

hadoop-yetus commented Oct 5, 2019

bshashikant commented Oct 7, 2019

szetszwo commented Oct 7, 2019

szetszwo commented Sep 24, 2019 •

edited

Loading

bshashikant commented Sep 30, 2019 •

edited

Loading

szetszwo commented Oct 2, 2019 •

edited

Loading