HADOOP-18332. remove rs-api dependency (3.3 branch) #4552

pjfanning · 2022-07-11T23:30:55Z

Description of PR

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

pjfanning · 2022-07-12T15:04:10Z

Closing in favour of #4544

downgrade jackson to 2.12.7 because of jax-rs compatibility issues in jackson 2.13 Update pom.xml remove excludes for rs-api

pjfanning · 2022-07-12T19:49:57Z

@ayushtkn I've reopened this and removed the rs-api excludes

hadoop-yetus · 2022-07-13T11:48:12Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	14m 14s		Maven dependency ordering for branch
+1 💚	mvninstall	27m 4s		branch-3.3 passed
+1 💚	compile	18m 45s		branch-3.3 passed
+1 💚	mvnsite	21m 7s		branch-3.3 passed
+1 💚	javadoc	7m 11s		branch-3.3 passed
+1 💚	shadedclient	32m 12s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 46s		Maven dependency ordering for patch
+1 💚	mvninstall	35m 37s		the patch passed
+1 💚	compile	18m 15s		the patch passed
+1 💚	javac	18m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	20m 46s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	7m 1s		the patch passed
+1 💚	shadedclient	32m 45s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	735m 34s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	2m 10s		The patch does not generate ASF License warnings.
		962m 48s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
	hadoop.hdfs.server.federation.router.TestRouterRpc
	hadoop.hdfs.server.balancer.TestBalancer

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4552/2/artifact/out/Dockerfile
GITHUB PR	#4552
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname	Linux 3c2d3ee92ff4 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `3f84def`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4552/2/testReport/
Max. process+thread count	2150 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4552/2/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

pjfanning · 2022-07-13T18:34:53Z

@ayushtkn this failed with issues based on a missing method:

Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.protocol.ClientProtocol.setBalancerBandwidth(java.lang.Long)
	at java.lang.Class.getDeclaredMethod(Class.java:2130)
	at org.apache.hadoop.hdfs.server.federation.router.RemoteMethod.getMethod(RemoteMethod.java:140)
	... 18 more

Any ideas?

ayushtkn · 2022-07-14T01:13:03Z

@pjfanning , that isn't related to this PR, spent some time checking. Looks like a bad cherry-pick caused it. Have dropped a comment at:
https://issues.apache.org/jira/browse/HDFS-16310?focusedCommentId=17566569&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17566569

ayushtkn

The RBF test failure is due to a bad cherry-pick, fails without this patch as well.
TestBalancer passes locally.

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.hdfs.server.balancer.TestBalancer
[INFO] Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 174.095 s - in org.apache.hadoop.hdfs.server.balancer.TestBalancer
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 27, Failures: 0, Errors: 0, Skipped: 0

Changes LGTM.

@aajisaka / @steveloughran / @virajjasani can you folks give a check as well.

virajjasani · 2022-07-14T02:43:26Z

The changes look good, basically this patch is (revert of HADOOP-18033 + Jackson 2.12.7), QA looks good and since Tez tests are now passing as per @ayushtkn's testing, it seems this is good enough to unblock 3.3.4. +1 (non-binding) from my side.

Thank you @pjfanning @ayushtkn !!

hadoop-yetus · 2022-07-14T10:10:04Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 47s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	14m 8s		Maven dependency ordering for branch
+1 💚	mvninstall	27m 33s		branch-3.3 passed
+1 💚	compile	20m 29s		branch-3.3 passed
+1 💚	mvnsite	22m 15s		branch-3.3 passed
+1 💚	javadoc	8m 0s		branch-3.3 passed
+1 💚	shadedclient	33m 19s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 51s		Maven dependency ordering for patch
+1 💚	mvninstall	42m 56s		the patch passed
+1 💚	compile	17m 29s		the patch passed
+1 💚	javac	17m 29s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	19m 55s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	7m 9s		the patch passed
+1 💚	shadedclient	31m 55s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	688m 20s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	2m 51s		The patch does not generate ASF License warnings.
		925m 59s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
	hadoop.hdfs.server.federation.router.TestRouterRpc
	hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
	hadoop.hdfs.TestReconstructStripedFile
	hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
	hadoop.hdfs.server.balancer.TestBalancer
	hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics
	hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4552/5/artifact/out/Dockerfile
GITHUB PR	#4552
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname	Linux 32ea1991eb3f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `a8a679f`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4552/5/testReport/
Max. process+thread count	3524 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4552/5/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

virajjasani · 2022-07-14T18:15:20Z

Except for TestRouterRpc and TestRouterRpcMultiDestination (also already discussed above), other tests are not consistent failures. Shall we have a couple more rounds of builds @ayushtkn @pjfanning, or you think these results are good?

pjfanning · 2022-07-14T18:18:39Z

There were no code changes in this PR since #4552 (comment) - and a lot fewer tests failed there.

hadoop-yetus · 2022-07-15T10:18:33Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	13m 58s		Maven dependency ordering for branch
+1 💚	mvninstall	27m 12s		branch-3.3 passed
+1 💚	compile	18m 56s		branch-3.3 passed
+1 💚	mvnsite	20m 54s		branch-3.3 passed
+1 💚	javadoc	7m 15s		branch-3.3 passed
+1 💚	shadedclient	32m 1s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 46s		Maven dependency ordering for patch
+1 💚	mvninstall	46m 5s		the patch passed
+1 💚	compile	18m 15s		the patch passed
+1 💚	javac	18m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	20m 44s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	7m 7s		the patch passed
+1 💚	shadedclient	32m 42s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	714m 1s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	2m 11s		The patch does not generate ASF License warnings.
		951m 14s

Reason	Tests
Failed junit tests	hadoop.yarn.server.resourcemanager.TestRMRestart
	hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
	hadoop.hdfs.server.federation.router.TestRouterRpc
	hadoop.fs.http.server.TestHttpFSAccessControlled
	hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
	hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics
	hadoop.hdfs.TestDecommissionWithStriped

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4552/6/artifact/out/Dockerfile
GITHUB PR	#4552
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname	Linux afbfec8143c1 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `a8a679f`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4552/6/testReport/
Max. process+thread count	2173 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4552/6/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2022-07-16T17:16:39Z

+1, merging to 3.3 with a detailed commit message, then i will submit as a 3.3.4 change & merge in there

….7. (apache#4552) This downgrades jackson from the version switched to in HADOOP-18033 (2.13.0), to Jackson 2.12.7. This removes the dependency on javax.ws.rs-api, so avoiding runtime problems with applications using jersey-core v1 and/or jsr311-api. The 2.12.7 release still contains the fix for CVE-2020-36518. Contributed by PJ Fanning

….7. (#4552) This downgrades jackson from the version switched to in HADOOP-18033 (2.13.0), to Jackson 2.12.7. This removes the dependency on javax.ws.rs-api, so avoiding runtime problems with applications using jersey-core v1 and/or jsr311-api. The 2.12.7 release still contains the fix for CVE-2020-36518. Contributed by PJ Fanning

pjfanning marked this pull request as draft July 11, 2022 23:31

pjfanning mentioned this pull request Jul 11, 2022

HADOOP-18332: remove rs-api dependency as it conflicts with jsr311-api #4547

Merged

4 tasks

pjfanning closed this Jul 12, 2022

This comment was marked as outdated.

Sign in to view

pjfanning reopened this Jul 12, 2022

pjfanning marked this pull request as ready for review July 12, 2022 19:44

HADOOP-18332: remove rs-api dependency as it conflicts with jsr311-api

9d0544f

downgrade jackson to 2.12.7 because of jax-rs compatibility issues in jackson 2.13 Update pom.xml remove excludes for rs-api

pjfanning force-pushed the HADOOP-18832-remove-rs-api-3.3 branch from 3f84def to 9d0544f Compare July 12, 2022 19:48

This comment was marked as outdated.

Sign in to view

Merge branch 'branch-3.3' into HADOOP-18832-remove-rs-api-3.3

a8a679f

ayushtkn approved these changes Jul 14, 2022

View reviewed changes

ayushtkn changed the title ~~HADOOP-18832 remove rs-api dependency (3.3 branch)~~ HADOOP-18832. remove rs-api dependency (3.3 branch) Jul 14, 2022

steveloughran changed the title ~~HADOOP-18832. remove rs-api dependency (3.3 branch)~~ HADOOP-18332. remove rs-api dependency (3.3 branch) Jul 16, 2022

steveloughran merged commit 6733ba5 into apache:branch-3.3 Jul 16, 2022

pjfanning deleted the HADOOP-18832-remove-rs-api-3.3 branch July 16, 2022 17:57

HADOOP-18332. remove rs-api dependency (3.3 branch) #4552

HADOOP-18332. remove rs-api dependency (3.3 branch) #4552

Uh oh!

Conversation

pjfanning commented Jul 11, 2022 • edited by steveloughran Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

pjfanning commented Jul 12, 2022

Uh oh!

This comment was marked as outdated.

pjfanning commented Jul 12, 2022

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

hadoop-yetus commented Jul 13, 2022

Uh oh!

pjfanning commented Jul 13, 2022

Uh oh!

ayushtkn commented Jul 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

virajjasani commented Jul 14, 2022

Uh oh!

hadoop-yetus commented Jul 14, 2022

Uh oh!

virajjasani commented Jul 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pjfanning commented Jul 14, 2022

Uh oh!

hadoop-yetus commented Jul 15, 2022

Uh oh!

steveloughran commented Jul 16, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pjfanning commented Jul 11, 2022 •

edited by steveloughran

Loading

ayushtkn commented Jul 14, 2022 •

edited

Loading

virajjasani commented Jul 14, 2022 •

edited

Loading