-
Notifications
You must be signed in to change notification settings - Fork 310
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/array node workflow parallelism #2268
Conversation
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #2268 +/- ##
===========================================
- Coverage 83.09% 53.57% -29.52%
===========================================
Files 324 180 -144
Lines 24860 18133 -6727
Branches 3761 3547 -214
===========================================
- Hits 20657 9715 -10942
- Misses 3582 7991 +4409
+ Partials 621 427 -194 ☔ View full report in Codecov by Sentry. |
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Can you fix the unit test failure? The error:
|
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
@eapolinario oof on my end. Just updated - will re-request your review after the builds pass |
* default array node concurrency to -1 Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * typo Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * set default concurrency to None for backwards compatibility Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * update unit test - hashed name Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> --------- Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> Signed-off-by: Jan Fiedler <jan@union.ai>
Tracking issue
flyteorg/flyte#3924
Why are the changes needed?
Want to add support for array node map tasks to utilize threads under the parent workflow's set parallelism. Decrease confusion on default/expected map task parallelism
What changes were proposed in this pull request?
Set default parallelism to None. If a user bumps their Flyte version to include flyteorg/flyte#5214 this will default the array node map task to utilize the parent wf's parallelism. If a user has an older flyte version this will map to 0, which will maintain the existing parallelism behavior.
NOTE this will cause cache misses on the first re-runs as the hashed name will change
How was this patch tested?
Setup process
Screenshots
Check all the applicable boxes
Related PRs
flyteorg/flyte#5062 + flyteorg/flyte#5214 were already merged
Docs link