[Data] Refactor PhysicalOperator.completed to fix side effects
#58915
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…n_finished() to has_execution_finished(). Make has_execution_finished() and complete() pure functions to remove side effects. Signed-off-by: Simeet Nayan <simeetnayan.8100@gmail.com>
bveeramani
reviewed
Nov 23, 2025
Member
bveeramani
left a comment
bveeramani
left a comment
There was a problem hiding this comment.
High-level change LGTM.
I think you'll need update references to execution_finished with has_execution_finished.
Would you mind renaming completed to has_completed to make it clear that the method doesn't modify state, and also ensure that this PR passes the microcheck status check?
…nished in limit_operator Signed-off-by: Simeet Nayan <simeetnayan.8100@gmail.com>
…nished in test_operators.py Signed-off-by: Simeet Nayan <simeetnayan.8100@gmail.com>
Signed-off-by: Simeet Nayan <simeetnayan.8100@gmail.com>
Contributor
Author
|
Sure thing @bveeramani. Will do the changes. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
python/ray/data/_internal/execution/operators/limit_operator.py
Outdated
Show resolved
Hide resolved
ray-gardener
bot
added
data
bveeramani
approved these changes
Nov 23, 2025
python/ray/data/_internal/execution/operators/limit_operator.py
Outdated
Show resolved
Hide resolved
… has_execution_finished() Signed-off-by: Simeet Nayan <simeetnayan.8100@gmail.com>
…refactor-PhysicalOperator Signed-off-by: Simeet Nayan <simeetnayan.8100@gmail.com>
Contributor
Author
Can I take this in a follow-up pr? Regarding the test failing, it's related to test_stats::test_large_args_scheduling_strategy, particularly at |
Member
Yeah, ofc. |
bveeramani
changed the title
PhysicalOperator.completed to fix side effects
SheldonTsen
pushed a commit
to SheldonTsen/ray
that referenced
this pull request
Dec 1, 2025
…-project#58915) # Description This PR refactors the `PhysicalOperator` class to eliminate hidden side effects in the `completed()` method. Previously, calling `completed()` could inadvertently modify the internal state of the operator, which could lead to unexpected behavior. This change separates the logic for checking if the operator is marked as finished from the logic that computes whether it is actually finished. Key changes include: - Renaming `_execution_finished` to `_is_execution_marked_finished` to clarify its purpose. - Renaming `execution_finished()` to `has_execution_finished()` and making it a pure computed property without side effects. - Updating the `completed()` method to call `has_execution_finished()` instead of modifying internal state. - Ensuring that `mark_execution_finished()` correctly sets the renamed field. ## Related issues Fixes ray-project#58884 ## Additional information This refactor ensures that both `has_execution_finished()` and `completed()` are pure query methods, allowing them to be called multiple times without altering the state of the operator. T --------- Signed-off-by: Simeet Nayan <simeetnayan.8100@gmail.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR refactors the
PhysicalOperatorclass to eliminate hidden side effects in thecompleted()method. Previously, callingcompleted()could inadvertently modify the internal state of the operator, which could lead to unexpected behavior. This change separates the logic for checking if the operator is marked as finished from the logic that computes whether it is actually finished.Key changes include:
_execution_finishedto_is_execution_marked_finishedto clarify its purpose.execution_finished()tohas_execution_finished()and making it a pure computed property without side effects.completed()method to callhas_execution_finished()instead of modifying internal state.mark_execution_finished()correctly sets the renamed field.Related issues
Fixes #58884
Additional information
This refactor ensures that both
has_execution_finished()andcompleted()are pure query methods, allowing them to be called multiple times without altering the state of the operator. T