TEZ-4472: Use unique DAG names in tests. #293

ayushtkn · 2023-06-21T05:27:13Z

No description provided.

Change-Id: Ibbec29dc3ac57414943e9cb35121187c2251cf71

Change-Id: Ie57c9b944d08f107b7b2f8e839aed6392adacb27

ayushtkn · 2023-06-21T07:26:20Z

Fixed test, the checkstyle warnings in TestCommit weren't actually induced here, but got pop up because I touched those lines, had to change the variable name to get them going

abstractdog · 2023-06-22T18:22:21Z

tez-dag/src/test/java/org/apache/tez/dag/app/dag/impl/TestVertexImpl.java

 LOG.info("Setting up mixed edge dag plan");
- org.apache.tez.dag.api.DAG dag = org.apache.tez.dag.api.DAG.create("MixedEdges");
+ org.apache.tez.dag.api.DAG dag =
+ org.apache.tez.dag.api.DAG.create("Dag-MixedEdges-" + dagNeme);


abstractdog · 2023-06-22T18:27:12Z

@ayushtkn: this patch covers much more test classes than I expected, which is a huge work thanks!
I feel that as you're about addressing as much as possible, maybe it's time to figure out a test dag name convention: some uses "Dag-" prefix, other uses "testdag", yet another uses only the test case's name...I don't have strong opinion about that, and I'm afraid it can still diverge later, but now it would make sense to unify it

abstractdog · 2023-06-22T18:29:51Z

tez-api/src/test/java/org/apache/tez/client/TestTezClient.java

@@ -203,12 +203,12 @@ TezClientForTest configureAndCreateTezClient(Map<String, LocalResource> lrs, boo

 @Test (timeout = 5000)
 public void testTezclientApp() throws Exception {
- testTezClient(false, true);
+ testTezClient(false, true, "testTezclientApp");


I know this patch is not about that, but...: can you please fix all occurrences of "Tezclient" -> "TezClient"?
feel free to create a new ticket if you want

Change-Id: Ib1129ed8b9dc2edfea57c365551d466b3e6eade8

abstractdog · 2023-06-23T08:08:22Z

thanks @ayushtkn ! one more class to address here, can you please check TestLocalMode too? I remember that was root cause of my headache originally that I workarounded with something like (with respecting the junit parameters), can you adapt it?

private String generateDagName(String baseName) {
  return baseName + (useDfs ? "_useDfs" : "") + (useLocalModeWithoutNetwork ? "_useLocalModeWithoutNetwork" : "");
}

and:

   private DAG createSimpleDAG(String dagName, String processorName) {
+    dagName = generateDagName(dagName);

DAG dag1 = createSimpleDAG("dag1", FailingProcessor.class.getName());
DAG dag1 = createSimpleDAG("DAG-testNoSysExitOnFailinglDAG", FailingProcessor.class.getName());

Change-Id: I611eb74577643feebf7dbf0aa37d6d41890cd5d6

abstractdog

+1
pending tests

tez-yetus · 2023-06-23T09:32:12Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 19s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 15 new or modified test files.
		_ master Compile Tests _
+0 🆗	mvndep	6m 9s	Maven dependency ordering for branch
+1 💚	mvninstall	9m 50s	master passed
+1 💚	compile	2m 4s	master passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1
+1 💚	compile	1m 56s	master passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~22.04-b09
+1 💚	checkstyle	2m 6s	master passed
+1 💚	javadoc	1m 43s	master passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1
+1 💚	javadoc	1m 36s	master passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~22.04-b09
+0 🆗	spotbugs	0m 51s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 41s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 9s	the patch passed
+1 💚	compile	1m 16s	the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1
+1 💚	javac	1m 16s	the patch passed
+1 💚	compile	1m 4s	the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~22.04-b09
+1 💚	javac	1m 4s	the patch passed
-0 ⚠️	checkstyle	0m 27s	tez-dag: The patch generated 1 new + 276 unchanged - 7 fixed = 277 total (was 283)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 45s	the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1
+1 💚	javadoc	0m 44s	the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~22.04-b09
+1 💚	findbugs	2m 52s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 16s	tez-api in the patch passed.
-1 ❌	unit	5m 9s	tez-dag in the patch failed.
+1 💚	unit	41m 3s	tez-tests in the patch passed.
+1 💚	asflicense	0m 37s	The patch does not generate ASF License warnings.
		89m 23s

Reason	Tests
Failed junit tests	tez.dag.app.rm.TestTaskScheduler

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-293/3/artifact/out/Dockerfile
GITHUB PR	#293
JIRA Issue	TEZ-4472
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux edd9f98f2932 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `d20f334`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~22.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~22.04-b09
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-293/3/artifact/out/diff-checkstyle-tez-dag.txt
unit	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-293/3/artifact/out/patch-unit-tez-dag.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-293/3/testReport/
Max. process+thread count	1426 (vs. ulimit of 5500)
modules	C: tez-api tez-dag tez-tests U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-293/3/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

tez-yetus · 2023-06-23T10:05:14Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 19s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 16 new or modified test files.
		_ master Compile Tests _
+0 🆗	mvndep	6m 27s	Maven dependency ordering for branch
+1 💚	mvninstall	10m 14s	master passed
+1 💚	compile	2m 1s	master passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1
+1 💚	compile	1m 53s	master passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~22.04-b09
+1 💚	checkstyle	2m 6s	master passed
+1 💚	javadoc	1m 40s	master passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1
+1 💚	javadoc	1m 37s	master passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~22.04-b09
+0 🆗	spotbugs	0m 52s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 21s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 10s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 8s	the patch passed
+1 💚	compile	1m 13s	the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1
+1 💚	javac	1m 13s	the patch passed
+1 💚	compile	1m 5s	the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~22.04-b09
+1 💚	javac	1m 5s	the patch passed
-0 ⚠️	checkstyle	0m 27s	tez-dag: The patch generated 1 new + 276 unchanged - 7 fixed = 277 total (was 283)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 45s	the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1
+1 💚	javadoc	0m 46s	the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~22.04-b09
+1 💚	findbugs	2m 54s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 16s	tez-api in the patch passed.
+1 💚	unit	5m 10s	tez-dag in the patch passed.
-1 ❌	unit	39m 27s	tez-tests in the patch failed.
+1 💚	asflicense	0m 41s	The patch does not generate ASF License warnings.
		88m 5s

Reason	Tests
Failed junit tests	tez.test.TestRecovery

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-293/4/artifact/out/Dockerfile
GITHUB PR	#293
JIRA Issue	TEZ-4472
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 7e7bc7e69129 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `d20f334`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~22.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu122.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~22.04-b09
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-293/4/artifact/out/diff-checkstyle-tez-dag.txt
unit	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-293/4/artifact/out/patch-unit-tez-tests.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-293/4/testReport/
Max. process+thread count	1391 (vs. ulimit of 5500)
modules	C: tez-api tez-dag tez-tests U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-293/4/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn · 2023-06-23T11:24:09Z

That checkstyle isn’t on me, I just touched that line, test failure doesn’t look related either

abstractdog · 2023-06-23T12:04:21Z

agreed, TestTaskScheduler failure is crazy, nothing to do with this patch, we don't need a new precommit

TEZ-4472: Use unique DAG names in tests.

daf4ab1

Change-Id: Ibbec29dc3ac57414943e9cb35121187c2251cf71

This comment was marked as outdated.

Sign in to view

Fix Test & Checkstyle.

8f73d89

Change-Id: Ie57c9b944d08f107b7b2f8e839aed6392adacb27

This comment was marked as outdated.

Sign in to view

abstractdog reviewed Jun 22, 2023

View reviewed changes

change name.

1096609

Change-Id: Ib1129ed8b9dc2edfea57c365551d466b3e6eade8

Handle TestLocalMode.

2d2f9c0

Change-Id: I611eb74577643feebf7dbf0aa37d6d41890cd5d6

abstractdog self-requested a review June 23, 2023 08:41

abstractdog approved these changes Jun 23, 2023

View reviewed changes

abstractdog merged commit 154fd34 into apache:master Jun 23, 2023
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TEZ-4472: Use unique DAG names in tests. #293

TEZ-4472: Use unique DAG names in tests. #293

ayushtkn commented Jun 21, 2023

This comment was marked as outdated.

ayushtkn commented Jun 21, 2023

This comment was marked as outdated.

abstractdog Jun 22, 2023

abstractdog commented Jun 22, 2023

abstractdog Jun 22, 2023

abstractdog commented Jun 23, 2023 •

edited

Loading

abstractdog left a comment

tez-yetus commented Jun 23, 2023

tez-yetus commented Jun 23, 2023

ayushtkn commented Jun 23, 2023

abstractdog commented Jun 23, 2023

TEZ-4472: Use unique DAG names in tests. #293

TEZ-4472: Use unique DAG names in tests. #293

Conversation

ayushtkn commented Jun 21, 2023

This comment was marked as outdated.

ayushtkn commented Jun 21, 2023

This comment was marked as outdated.

abstractdog Jun 22, 2023

Choose a reason for hiding this comment

abstractdog commented Jun 22, 2023

abstractdog Jun 22, 2023

Choose a reason for hiding this comment

abstractdog commented Jun 23, 2023 • edited Loading

abstractdog left a comment

Choose a reason for hiding this comment

tez-yetus commented Jun 23, 2023

tez-yetus commented Jun 23, 2023

ayushtkn commented Jun 23, 2023

abstractdog commented Jun 23, 2023

abstractdog commented Jun 23, 2023 •

edited

Loading