Add task affinity support with compute_scope and result_scope in Dagger.jl's @spawn macro
#610
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…er.jl's @Spawn macro - Enhanced the Thunk struct to include compute_scope and result_scope for better task execution control. - Updated the Thunk constructor to accept new scope parameters. - Modified the spawn function to handle the new scope parameters appropriately. - Introduced a new test suite for task affinity, covering various scenarios with scope interactions. - Added comprehensive documentation for task affinity, detailing the usage of scope, compute_scope, and result_scope. - Implemented tests to validate behavior when using chunks as inputs in tasks, ensuring correct scope handling.
AkhilAkkapelli
changed the title
@spawn macro
AkhilAkkapelli
changed the title
@spawn macro@spawn macro
jpsamaroo
requested changes
Jun 15, 2025
test/task-affinity.jl
Outdated
|
|
||
| end | ||
|
|
||
| # @testset "Dagger Chunk with Chunk Arguments in Task: scope, compute_scope and result_scope Tests with non-Intersection of chunk arg and chunk scope" begin |
There was a problem hiding this comment.
What's the plan with these tests?
| task3 = Dagger.@spawn compute_scope=compute_scope_no_intersect scope=scope_rand result_scope=result_scope_no_intersect f(30); wait(task3) | ||
|
|
||
| @test get_compute_scope(task1) == get_compute_scope(task2) == get_compute_scope(task3) == compute_scope_no_intersect | ||
| @test get_result_scope(task1) == get_result_scope(task2) == get_result_scope(task3) == Dagger.InvalidScope |
There was a problem hiding this comment.
Suggested change
| @test get_result_scope(task1) == get_result_scope(task2) == get_result_scope(task3) == Dagger.InvalidScope | |
| @test get_result_scope(task1) == get_result_scope(task2) == get_result_scope(task3) && get_result_scope(task1) isa Dagger.InvalidScope |
I think this is what you really mean? That the resulting intersection is empty?
- Updated documentation for the `@spawn` macro to clarify the usage of `scope`, `compute_scope`, and `result_scope`, including examples with the new syntax. - Improved error messages in the scheduling logic to provide clearer feedback when scopes are incompatible. - Refactored test cases for task affinity to ensure they align with the new scope handling and provide better coverage for edge cases. - Removed deprecated comments and cleaned up the code for better readability.
jpsamaroo
reviewed
Jun 20, 2025
| @@ -1,123 +1,127 @@ | |||
| # Task Affinity | |||
|
|
|||
| Dagger.jl's `@spawn` macro allows precise control over task execution and result access using `scope`, `compute_scope`, and `result_scope`. | |||
| Dagger.jl's `@spawn` macro allows precise control over task execution and result accessibility using `scope`, `compute_scope`, and `result_scope`, which specify various chunk scopes of the task. | |||
There was a problem hiding this comment.
Not "chunk scopes", just "scopes". My point is that we can add language that mentions that chunk scopes (which are distinct from these 3 scope types) also factor in to task affinity. Maybe it's worth rewording to "Dagger's tasks are assigned to processors according to the combination of multiple scopes, namely scope, compute_scope, and result_scope (which can all be specified with @spawn), and additionally the scopes of any arguments to the task (in the form of a scope attached to a Chunk argument). Let's take a look at how these work together to direct task placement."
|
|
||
| --- | ||
|
|
||
| ## Key Terms | ||
|
|
||
| ### Scope | ||
| `scope` defines the general set of locations where a Dagger task can execute. If `compute_scope` and `result_scope` are not explicitly set, the task's `compute_scope` defaults to its `scope`, and its `result_scope` defaults to `AnyScope()`, meaning the result can be accessed by any processor. Execution occurs on any processor within the defined scope. | ||
| `scope` defines the general set of locations where a Dagger task can execute. If `scope` is not explicitly set, the task runs within the `compute_scope`. If both `scope` and `compute_scope` both are unspecified, the task falls back to `DefaultScope()`, allowing it to run wherever execution is possible. Execution occurs on any worker within the defined scope. |
There was a problem hiding this comment.
"wherever execution is possible by default (on CPUs, but not on GPUs)."
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
compute_scopeandresult_scopefor better task execution control.@spawnmacro to handle the new scope parameters appropriately.scope,compute_scope, andresult_scope.