MAPREDUCE-7517 TestPipeApplication hangs with JAVA 17+ #7952

stoty · 2025-09-11T12:45:38Z

Description of PR

MAPREDUCE-7517 TestPipeApplication hangs with JAVA 17+

Fixes TestPipeApplication to use the java.home system property to start a new Java process instead of taking it from the $PATH.

I have also added timeouts to make sure this test won't hang forever if any problems crop up in the future.

How was this patch tested?

Ran TestPipeApplication in the development docker image with JAVA_HOME set to JAVA_17.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

stoty · 2025-09-11T12:46:44Z

@slfan1989 @pan3793 @szetszwo PTAL

pan3793 · 2025-09-11T12:56:56Z

...educe-client-jobclient/src/test/java/org/apache/hadoop/mapred/pipes/TestPipeApplication.java

      os.write(("ls ").getBytes());
    } else {
-      os.write(("java -cp " + classpath + " " + clazz).getBytes());
+      os.write((System.getProperty("java.home") + "/bin/java -cp " + classpath + " " + clazz).getBytes());


This should work.

It might be worth documenting that in Java 8, System.getProperty("java.home") actually returns $JAVA_HOME/jre, commands under $JAVA_HOME/jre/bin are a subset of $JAVA_HOME/bin.

I didn't know that.

Added comment

hadoop-yetus · 2025-09-11T16:00:11Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	28m 12s		trunk passed
+1 💚	compile	0m 19s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 19s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	0m 22s		trunk passed
+1 💚	mvnsite	0m 22s		trunk passed
+1 💚	javadoc	0m 21s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 16s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	0m 30s		trunk passed
+1 💚	shadedclient	21m 37s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 16s		the patch passed
+1 💚	compile	0m 16s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 16s		the patch passed
+1 💚	compile	0m 14s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 13s	/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 1 new + 17 unchanged - 0 fixed = 18 total (was 17)
+1 💚	mvnsite	0m 17s		the patch passed
+1 💚	javadoc	0m 13s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 12s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	0m 35s		the patch passed
+1 💚	shadedclient	22m 5s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	94m 56s	/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt	hadoop-mapreduce-client-jobclient in the patch passed.
+1 💚	asflicense	0m 28s		The patch does not generate ASF License warnings.
		173m 40s

Reason	Tests
Failed junit tests	hadoop.mapreduce.v2.TestUberAM

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7952/2/artifact/out/Dockerfile
GITHUB PR	#7952
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux b54e11f53109 5.15.0-142-generic #152-Ubuntu SMP Mon May 19 10:54:31 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `0515e95`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7952/2/testReport/
Max. process+thread count	1238 (vs. ulimit of 5500)
modules	C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7952/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-09-11T16:56:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	43m 26s		trunk passed
+1 💚	compile	0m 38s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 34s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	0m 40s		trunk passed
+1 💚	mvnsite	0m 39s		trunk passed
+1 💚	javadoc	0m 35s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 28s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	0m 55s		trunk passed
+1 💚	shadedclient	41m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 27s		the patch passed
+1 💚	compile	0m 30s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 30s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 32s	/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 1 new + 17 unchanged - 0 fixed = 18 total (was 17)
+1 💚	mvnsite	0m 30s		the patch passed
+1 💚	javadoc	0m 22s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 20s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	0m 53s		the patch passed
+1 💚	shadedclient	39m 52s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	113m 9s	/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt	hadoop-mapreduce-client-jobclient in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		249m 45s

Reason	Tests
Failed junit tests	hadoop.mapreduce.v2.TestUberAM

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7952/1/artifact/out/Dockerfile
GITHUB PR	#7952
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux e72584367088 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `dad9801`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7952/1/testReport/
Max. process+thread count	1212 (vs. ulimit of 5500)
modules	C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7952/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2025-09-11T23:01:32Z

@stoty Thanks for the contribution! Merged Into trunk. @pan3793 Thanks for the review!

…Contributed by Istvan Toth. * MAPREDUCE-7517 TestPipeApplication hangs with JAVA 17+ * add comment about JRE path Reviewed-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Shilun Fan <slfan1989@apache.org>

MAPREDUCE-7517 TestPipeApplication hangs with JAVA 17+

dad9801

github-actions bot added MapReduce trunk labels Sep 11, 2025

slfan1989 approved these changes Sep 11, 2025

View reviewed changes

pan3793 reviewed Sep 11, 2025

View reviewed changes

add comment about JRE path

0515e95

pan3793 approved these changes Sep 11, 2025

View reviewed changes

slfan1989 merged commit 79f37d0 into apache:trunk Sep 11, 2025
1 of 4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

MAPREDUCE-7517 TestPipeApplication hangs with JAVA 17+ #7952

MAPREDUCE-7517 TestPipeApplication hangs with JAVA 17+ #7952

Uh oh!

stoty commented Sep 11, 2025 •

edited by slfan1989

Loading

Uh oh!

stoty commented Sep 11, 2025

Uh oh!

pan3793 Sep 11, 2025

Uh oh!

stoty Sep 11, 2025

Uh oh!

stoty Sep 11, 2025

Uh oh!

hadoop-yetus commented Sep 11, 2025

Uh oh!

hadoop-yetus commented Sep 11, 2025

Uh oh!

Uh oh!

slfan1989 commented Sep 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MAPREDUCE-7517 TestPipeApplication hangs with JAVA 17+ #7952

MAPREDUCE-7517 TestPipeApplication hangs with JAVA 17+ #7952

Uh oh!

Conversation

stoty commented Sep 11, 2025 • edited by slfan1989 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

stoty commented Sep 11, 2025

Uh oh!

pan3793 Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

stoty Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

stoty Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Sep 11, 2025

Uh oh!

hadoop-yetus commented Sep 11, 2025

Uh oh!

Uh oh!

slfan1989 commented Sep 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

stoty commented Sep 11, 2025 •

edited by slfan1989

Loading