Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TEZ-4508: Allow the FAIR_PARALLELISM mode to accept multiple source vertices #306

Closed
wants to merge 1 commit into from

Conversation

okumin
Copy link
Contributor

@okumin okumin commented Aug 7, 2023

As explained in TEZ-4508, I expect there are valid use cases where FairShuffleVertexManager with FAIR_PARALLELISM should accept multiple SCATTER_GATHER edges.

throw new TezUncheckedException(
"Having more than one destination task process same partition(s) " +
"only works with one bipartite source.");
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I expect this assertion is not must because Tez users currently use FairShuffleVertexManager in specific cases on purpose.
I guess we need to add a new DataMovementType or new FairShuffleVertexManager for JOIN, but we don't immediately need it.

@tez-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 25m 59s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 💚 mvninstall 17m 2s master passed
+1 💚 compile 0m 39s master passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu122.04
+1 💚 compile 0m 32s master passed with JDK Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
+1 💚 checkstyle 1m 21s master passed
+1 💚 javadoc 0m 43s master passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu122.04
+1 💚 javadoc 0m 29s master passed with JDK Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
+0 🆗 spotbugs 1m 32s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 1m 31s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 22s the patch passed
+1 💚 compile 0m 24s the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu122.04
+1 💚 javac 0m 24s the patch passed
+1 💚 compile 0m 21s the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
+1 💚 javac 0m 21s the patch passed
-0 ⚠️ checkstyle 0m 15s tez-runtime-library: The patch generated 1 new + 12 unchanged - 0 fixed = 13 total (was 12)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 javadoc 0m 19s the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu122.04
+1 💚 javadoc 0m 17s the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
+1 💚 findbugs 0m 58s the patch passed
_ Other Tests _
+1 💚 unit 5m 31s tez-runtime-library in the patch passed.
+1 💚 asflicense 0m 15s The patch does not generate ASF License warnings.
58m 3s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-306/1/artifact/out/Dockerfile
GITHUB PR #306
JIRA Issue TEZ-4508
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname Linux ee80f5606113 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/tez.sh
git revision master / 5beab4c
Default Java Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~22.04.1-b05
checkstyle https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-306/1/artifact/out/diff-checkstyle-tez-runtime-library.txt
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-306/1/testReport/
Max. process+thread count 1090 (vs. ulimit of 5500)
modules C: tez-runtime-library U: tez-runtime-library
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-306/1/console
versions git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@abstractdog abstractdog changed the title TEZ-4508: Allow the FAIR_PARALLELISM mode to accept multiple source v… TEZ-4508: Allow the FAIR_PARALLELISM mode to accept multiple source vertices Dec 12, 2023
@okumin
Copy link
Contributor Author

okumin commented May 28, 2024

While testing various cases, I found this might not work when amounts of unified vertices are uneven. I will close this one once.
https://github.com/apache/tez/blob/rel/release-0.10.3/tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/FairShuffleVertexManager.java#L443-L447

@okumin okumin closed this May 28, 2024
@okumin okumin deleted the TEZ-4508-fair-multiple branch August 4, 2024 12:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants