-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ARROW-17178: [R] Support head() in arrow_dplyr_query with user-defined function #13706
Merged
Merged
Changes from all commits
Commits
Show all changes
34 commits
Select commit
Hold shift + click to select a range
f9f696d
consolidate exec plan reader into a dedicated reader class
paleolimbot dbd2d61
fix the head thing
paleolimbot 095b3f0
undo some kludges introduced
paleolimbot 4a32870
explicitly close readers where possible
paleolimbot 24bc1cd
just kidding don't do that
paleolimbot 1542829
just a shot in the dark
paleolimbot 9e9170f
try a new strategy for stopproducing
paleolimbot 30a8684
try a more raw approach to stoppinh production
paleolimbot 8225dc4
always set status
paleolimbot d69e365
another approach to stopping production
paleolimbot e860650
more sane but still deadlocking stop_producing
paleolimbot 7a54584
uncomment
paleolimbot 7093723
go back to the other approach
paleolimbot 27143d6
remove unnecessary Close()
paleolimbot afe61cd
maybe fix python test
paleolimbot bcd17bc
with ExecPlanReader R6
paleolimbot 2f9a827
undo python workaround
paleolimbot a5d24a2
with tests to enforce laziness
paleolimbot 9ee0602
document the rbr subclasses
paleolimbot a915050
exec plan reader head() + tests
paleolimbot 40a99b2
remove collect with udf kludge
paleolimbot 020d338
nix the custom head method
paleolimbot 93764f6
wait for idle thread pool on valgrind job
paleolimbot 0b8ddfa
remove thread pool stuff (it didn't work)
paleolimbot 05f7067
simplify arrow_dplyr_query to table
paleolimbot b6b9102
add test for lazy head()
paleolimbot 5ec7c1a
don't use R function call recordbatchreader just now
paleolimbot adc61f6
add a note about lazy stopping but don't test it quite yet
paleolimbot 921296f
Apply suggestions from code review
paleolimbot 4fece2a
make the test less funny and less confusing
paleolimbot df724b4
attempt to simplify the call to StopProducing()
paleolimbot 1e6d571
comment to remind myself why the complicated callback thing needs to …
paleolimbot 7d8e92f
update for C++17
paleolimbot 92e0416
clang-format
paleolimbot File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why don't we want/need this anymore?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because of the change to
head.RecordBatchReader()
, no batches get pulled whenhead(out)
is called (which is need to make sure that the ExecPlan can continue to defer StartProducing()). The C++ implementation ofhead.RecordBatchReader()
takes care of callingClose()
on the upstream reader (which, if it's an ExecPlanReader, will callStopProducing()
. I should add a test to make sure that's the case though!There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Test added!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm...the test passes locally and on Windows but fails on all the Linux CI. My guess is that this is related to how the request to stop producing is (or is not) relayed to the source node (or maybe how quickly that happens?).