HDDS-2048: State check during container state transition in datanode should be lock protected #1375

lokeshj1703 · 2019-08-29T08:39:20Z

Currently container state checks during state transition are not lock protected in KeyValueHandler. These can cause invalid state transitions.

…ock protected

hadoop-yetus · 2019-08-29T11:00:36Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	259	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	667	trunk passed
+1	compile	428	trunk passed
+1	checkstyle	80	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	1025	branch has no errors when building and testing our client artifacts.
+1	javadoc	186	trunk passed
0	spotbugs	466	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	702	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	629	the patch passed
+1	compile	387	the patch passed
+1	javac	387	the patch passed
+1	checkstyle	86	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	661	patch has no errors when building and testing our client artifacts.
+1	javadoc	172	the patch passed
+1	findbugs	656	the patch passed
		_ Other Tests _
+1	unit	338	hadoop-hdds in the patch passed.
-1	unit	1885	hadoop-ozone in the patch failed.
+1	asflicense	50	The patch does not generate ASF License warnings.
		8407

Reason	Tests
Failed junit tests	hadoop.ozone.TestOzoneConfigurationFields
	hadoop.hdds.scm.pipeline.TestRatisPipelineProvider

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/1/artifact/out/Dockerfile
GITHUB PR	#1375
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux d65163c71451 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `c749f62`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/1/testReport/
Max. process+thread count	4580 (vs. ulimit of 5500)
modules	C: hadoop-hdds/container-service U: hadoop-hdds/container-service
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-08-30T16:58:16Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	80	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	675	trunk passed
+1	compile	451	trunk passed
+1	checkstyle	81	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	922	branch has no errors when building and testing our client artifacts.
+1	javadoc	190	trunk passed
0	spotbugs	467	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	705	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	633	the patch passed
+1	compile	460	the patch passed
+1	javac	460	the patch passed
+1	checkstyle	92	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	707	patch has no errors when building and testing our client artifacts.
+1	javadoc	202	the patch passed
+1	findbugs	731	the patch passed
		_ Other Tests _
+1	unit	386	hadoop-hdds in the patch passed.
-1	unit	2771	hadoop-ozone in the patch failed.
+1	asflicense	54	The patch does not generate ASF License warnings.
		9308

Reason	Tests
Failed junit tests	hadoop.ozone.om.TestOzoneManagerConfiguration
	hadoop.ozone.scm.node.TestQueryNode
	hadoop.ozone.TestOzoneConfigurationFields

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/2/artifact/out/Dockerfile
GITHUB PR	#1375
JIRA Issue	HDDS-2048
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 0f27a8958463 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `c929b38`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/2/testReport/
Max. process+thread count	5288 (vs. ulimit of 5500)
modules	C: hadoop-hdds/container-service U: hadoop-hdds/container-service
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-09-02T23:11:30Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	36	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	580	trunk passed
+1	compile	378	trunk passed
+1	checkstyle	80	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	867	branch has no errors when building and testing our client artifacts.
+1	javadoc	176	trunk passed
0	spotbugs	420	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	613	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	533	the patch passed
+1	compile	404	the patch passed
+1	javac	404	the patch passed
+1	checkstyle	87	the patch passed
+1	mvnsite	1	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	698	patch has no errors when building and testing our client artifacts.
+1	javadoc	180	the patch passed
+1	findbugs	675	the patch passed
		_ Other Tests _
+1	unit	276	hadoop-hdds in the patch passed.
-1	unit	1794	hadoop-ozone in the patch failed.
+1	asflicense	49	The patch does not generate ASF License warnings.
		7595

Reason	Tests
Failed junit tests	hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
	hadoop.ozone.om.snapshot.TestOzoneManagerSnapshotProvider
	hadoop.ozone.om.TestSecureOzoneManager
	hadoop.ozone.om.TestOzoneManagerRestart
	hadoop.ozone.om.TestOMDbCheckpointServlet
	hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/3/artifact/out/Dockerfile
GITHUB PR	#1375
JIRA Issue	HDDS-2048
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux cc8fecdd31b3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `915cbc9`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/3/testReport/
Max. process+thread count	5057 (vs. ulimit of 5500)
modules	C: hadoop-hdds/container-service U: hadoop-hdds/container-service
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

nandakumar131 · 2019-09-03T11:46:24Z

Thanks @lokeshj1703 for the PR.
Can you remove the usage of lock from KeyValueContainer and use it only in KeyValueHandler?

lokeshj1703 · 2019-09-03T14:27:03Z

@nandakumar131 The api in KeyValueContainer is currently public. Therefore I did not remove those locks.

lokeshj1703 · 2019-09-05T10:53:06Z

@nandakumar131 Thanks for reviewing the PR! I have updated changes as per offline discussion.

hadoop-yetus · 2019-09-05T13:02:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	41	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	573	trunk passed
+1	compile	380	trunk passed
+1	checkstyle	81	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	914	branch has no errors when building and testing our client artifacts.
+1	javadoc	182	trunk passed
0	spotbugs	435	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	642	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	532	the patch passed
+1	compile	398	the patch passed
+1	javac	398	the patch passed
+1	checkstyle	94	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	665	patch has no errors when building and testing our client artifacts.
+1	javadoc	174	the patch passed
+1	findbugs	663	the patch passed
		_ Other Tests _
+1	unit	300	hadoop-hdds in the patch passed.
-1	unit	1955	hadoop-ozone in the patch failed.
+1	asflicense	50	The patch does not generate ASF License warnings.
		7825

Reason	Tests
Failed junit tests	hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline
	hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion
	hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
	hadoop.ozone.TestSecureOzoneCluster

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/4/artifact/out/Dockerfile
GITHUB PR	#1375
JIRA Issue	HDDS-2048
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux ab2948373bb3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `172bcd8`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/4/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/4/testReport/
Max. process+thread count	4752 (vs. ulimit of 5500)
modules	C: hadoop-hdds/container-service U: hadoop-hdds/container-service
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/4/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

nandakumar131

We should do the lock/unlock with try..finally block.

lokeshj1703 · 2019-09-09T06:14:03Z

@nandakumar131 Thanks for reviewing the changes! 3rd commit addresses your comments. There was an unlock missing for container read lock taken in BlockManagerImpl class. I have fixed it in the commit as well.

hadoop-yetus · 2019-09-09T08:06:29Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	948	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	600	trunk passed
+1	compile	380	trunk passed
+1	checkstyle	83	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	884	branch has no errors when building and testing our client artifacts.
+1	javadoc	178	trunk passed
0	spotbugs	445	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	641	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	533	the patch passed
+1	compile	389	the patch passed
+1	javac	389	the patch passed
+1	checkstyle	90	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	684	patch has no errors when building and testing our client artifacts.
+1	javadoc	174	the patch passed
+1	findbugs	628	the patch passed
		_ Other Tests _
+1	unit	295	hadoop-hdds in the patch passed.
-1	unit	181	hadoop-ozone in the patch failed.
+1	asflicense	48	The patch does not generate ASF License warnings.
		6928

Reason	Tests
Failed junit tests	hadoop.ozone.om.ratis.TestOzoneManagerRatisServer

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/5/artifact/out/Dockerfile
GITHUB PR	#1375
JIRA Issue	HDDS-2048
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux bfea94d0b409 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `3b9584d`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/5/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/5/testReport/
Max. process+thread count	1197 (vs. ulimit of 5500)
modules	C: hadoop-hdds/container-service U: hadoop-hdds/container-service
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1375/5/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

…should be lock protected (apache#1375)

State check during container state transition in datanode should be l…

450786a

…ock protected

lokeshj1703 requested a review from nandakumar131 August 29, 2019 08:39

lokeshj1703 self-assigned this Aug 29, 2019

lokeshj1703 added the ozone label Aug 29, 2019

lokeshj1703 changed the title ~~State check during container state transition in datanode should be lock protected~~ HDDS-2048: State check during container state transition in datanode should be lock protected Aug 30, 2019

Address review comments

9a03db9

nandakumar131 requested changes Sep 8, 2019

View reviewed changes

Address review comments

a388dea

nandakumar131 approved these changes Sep 10, 2019

View reviewed changes

nandakumar131 merged commit c3beeb7 into apache:trunk Sep 10, 2019

lokeshj1703 deleted the HDDS-2048 branch September 10, 2019 08:45

amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019

HDDS-2048: State check during container state transition in datanode …

71a9a9f

…should be lock protected (apache#1375)

RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020

HDDS-2048: State check during container state transition in datanode …

fecf929

…should be lock protected (apache#1375)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-2048: State check during container state transition in datanode should be lock protected #1375

HDDS-2048: State check during container state transition in datanode should be lock protected #1375

lokeshj1703 commented Aug 29, 2019

hadoop-yetus commented Aug 29, 2019

hadoop-yetus commented Aug 30, 2019

hadoop-yetus commented Sep 2, 2019

nandakumar131 commented Sep 3, 2019

lokeshj1703 commented Sep 3, 2019

lokeshj1703 commented Sep 5, 2019

hadoop-yetus commented Sep 5, 2019

nandakumar131 left a comment

lokeshj1703 commented Sep 9, 2019

hadoop-yetus commented Sep 9, 2019

HDDS-2048: State check during container state transition in datanode should be lock protected #1375

HDDS-2048: State check during container state transition in datanode should be lock protected #1375

Conversation

lokeshj1703 commented Aug 29, 2019

hadoop-yetus commented Aug 29, 2019

hadoop-yetus commented Aug 30, 2019

hadoop-yetus commented Sep 2, 2019

nandakumar131 commented Sep 3, 2019

lokeshj1703 commented Sep 3, 2019

lokeshj1703 commented Sep 5, 2019

hadoop-yetus commented Sep 5, 2019

nandakumar131 left a comment

Choose a reason for hiding this comment

lokeshj1703 commented Sep 9, 2019

hadoop-yetus commented Sep 9, 2019