[SPARK-45133][CONNECT] Make Spark Connect queries be FINISHED when last result task is finished #42889
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
cc @jdesjean @gjxdxh @gengliangwang @hvanhovell @grundprinzip |
Contributor
|
The change seems logical to me as well. |
HyukjinKwon
approved these changes
Sep 13, 2023
HyukjinKwon
changed the title
Member
|
Merged to master. |
Member
|
Hi, @juliuszsompolski and @HyukjinKwon . I hoped this recovers the master branch, but it seems that CI is still failing since the commit which |
Contributor
Author
|
In description I wrote "Will add test in #42560", but since that one was merged yesterday, I planned to rebase and add test here. I will add test in a followup. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
In the situation before, query will only be FINISHED when all results have been pushed into the output buffers (not necessarily received by client, but pushed out of the server).
For LocalTableScanExec, post FINISHED before sending result batches, because nothing is executed, only cached local results are returned. For regular execution, post FINISHED after all task results have been returned from Spark, not after they have been processed and sent out.
Why are the changes needed?
Currently, even if a query finished running in Spark, it keeps being RUNNING until all results are sent. Then there is a very small difference between FINISHED and CLOSED. This change makes it behave more similar to e.g. Thriftserver.
Does this PR introduce any user-facing change?
Yes. Queries will be posted as FINISHED when they finish executing, not when they finish sending results.
How was this patch tested?
Will add test in #42560
Was this patch authored or co-authored using generative AI tooling?
No.