TEZ-4110: Make Tez fail fast when DFS quota is exceeded. #313

ayushtkn · 2023-10-14T22:04:52Z

Tested on Hive.
Without Patch

----------------------------------------------------------------------------------------------
        VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED  
----------------------------------------------------------------------------------------------
Map 1            container       RUNNING      1          0        0        1       4       0  
Reducer 2        container        INITED      1          0        0        1       0       0  
----------------------------------------------------------------------------------------------
VERTICES: 00/02  [>>--------------------------] 0%    ELAPSED TIME: 16.68 s    
----------------------------------------------------------------------------------------------

With Patch

----------------------------------------------------------------------------------------------
        VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED  
----------------------------------------------------------------------------------------------
Map 1            container       RUNNING      1          0        0        1       1       0  
Reducer 2        container        INITED      1          0        0        1       0       0  
----------------------------------------------------------------------------------------------
VERTICES: 00/02  [>>--------------------------] 0%    ELAPSED TIME: 5.52 s     
----------------------------------------------------------------------------------------------

ayushtkn · 2023-10-14T22:07:37Z

Added the entire log output on the ticket:
With-Patch-Output
Without-Patch-Output

abstractdog · 2023-10-15T12:14:28Z

thanks for this patch!
I can see that MAPREDUCE-7148 has a unit test for the corresponding change, would you consider adopting it to tez also?

tez-yetus · 2023-10-15T13:10:49Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 14s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ master Compile Tests _
+1 💚	mvninstall	10m 29s	master passed
+1 💚	compile	0m 26s	master passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu122.04
+1 💚	compile	0m 24s	master passed with JDK Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
+1 💚	checkstyle	1m 13s	master passed
+1 💚	javadoc	0m 32s	master passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu122.04
+1 💚	javadoc	0m 20s	master passed with JDK Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
+0 🆗	spotbugs	1m 5s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 5s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 13s	the patch passed
+1 💚	compile	0m 13s	the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu122.04
+1 💚	javac	0m 13s	the patch passed
+1 💚	compile	0m 12s	the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
+1 💚	javac	0m 12s	the patch passed
-0 ⚠️	checkstyle	0m 8s	tez-runtime-internals: The patch generated 1 new + 74 unchanged - 0 fixed = 75 total (was 74)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 7s	the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu122.04
+1 💚	javadoc	0m 7s	the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
+1 💚	findbugs	0m 28s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 38s	tez-runtime-internals in the patch passed.
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		17m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-313/2/artifact/out/Dockerfile
GITHUB PR	#313
JIRA Issue	TEZ-4110
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 5fd9d043d6b7 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `5bba1ff`
Default Java	Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-313/2/artifact/out/diff-checkstyle-tez-runtime-internals.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-313/2/testReport/
Max. process+thread count	100 (vs. ulimit of 5500)
modules	C: tez-runtime-internals U: tez-runtime-internals
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-313/2/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn · 2023-10-15T13:59:48Z

Thanx @abstractdog I added some tests.

Checkstyle is kind of unavoidable, getting rid of that is overkill in the prod code for testing IMO

abstractdog · 2023-10-16T07:41:24Z

tez-runtime-internals/src/test/java/org/apache/tez/runtime/task/TestTaskExecution2.java

@@ -653,6 +657,40 @@ public void testKilledAfterComplete() throws IOException, InterruptedException,
 }
 }

+ @Test
+ public void testClusterStoragaeCapacityFatalError() throws IOException {


nit: Storagae

abstractdog · 2023-10-16T07:41:44Z

LGTM, only a typo

tez-yetus · 2023-10-16T08:09:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 14s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ master Compile Tests _
+1 💚	mvninstall	12m 55s	master passed
+1 💚	compile	0m 26s	master passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu122.04
+1 💚	compile	0m 25s	master passed with JDK Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
+1 💚	checkstyle	1m 18s	master passed
+1 💚	javadoc	0m 33s	master passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu122.04
+1 💚	javadoc	0m 21s	master passed with JDK Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
+0 🆗	spotbugs	1m 6s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 4s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 13s	the patch passed
+1 💚	compile	0m 14s	the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu122.04
+1 💚	javac	0m 14s	the patch passed
+1 💚	compile	0m 12s	the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
+1 💚	javac	0m 12s	the patch passed
-0 ⚠️	checkstyle	0m 8s	tez-runtime-internals: The patch generated 1 new + 74 unchanged - 0 fixed = 75 total (was 74)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 8s	the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu122.04
+1 💚	javadoc	0m 7s	the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
+1 💚	findbugs	0m 27s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 38s	tez-runtime-internals in the patch passed.
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		20m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-313/3/artifact/out/Dockerfile
GITHUB PR	#313
JIRA Issue	TEZ-4110
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 0f5c6f4dc5c8 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `5bba1ff`
Default Java	Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-313/3/artifact/out/diff-checkstyle-tez-runtime-internals.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-313/3/testReport/
Max. process+thread count	101 (vs. ulimit of 5500)
modules	C: tez-runtime-internals U: tez-runtime-internals
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-313/3/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

TEZ-4110: Make Tez fail fast when DFS quota is exceeded.

8772c99

This comment was marked as outdated.

Sign in to view

Add Test.

29b234c

abstractdog reviewed Oct 16, 2023

View reviewed changes

Fix Typo.

fa1b6fc

abstractdog requested review from abstractdog and removed request for abstractdog October 16, 2023 07:49

abstractdog approved these changes Oct 16, 2023

View reviewed changes

abstractdog merged commit 4bc87e2 into apache:master Oct 16, 2023
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TEZ-4110: Make Tez fail fast when DFS quota is exceeded. #313

TEZ-4110: Make Tez fail fast when DFS quota is exceeded. #313

ayushtkn commented Oct 14, 2023

ayushtkn commented Oct 14, 2023

This comment was marked as outdated.

abstractdog commented Oct 15, 2023

tez-yetus commented Oct 15, 2023

ayushtkn commented Oct 15, 2023

abstractdog Oct 16, 2023

abstractdog commented Oct 16, 2023

tez-yetus commented Oct 16, 2023

TEZ-4110: Make Tez fail fast when DFS quota is exceeded. #313

TEZ-4110: Make Tez fail fast when DFS quota is exceeded. #313

Conversation

ayushtkn commented Oct 14, 2023

ayushtkn commented Oct 14, 2023

This comment was marked as outdated.

abstractdog commented Oct 15, 2023

tez-yetus commented Oct 15, 2023

ayushtkn commented Oct 15, 2023

abstractdog Oct 16, 2023

Choose a reason for hiding this comment

abstractdog commented Oct 16, 2023

tez-yetus commented Oct 16, 2023