-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(r): Add callback-based async option for calling ArrowArrayStream::get_next()
#211
Conversation
🤩 🤩 🤩 Give me some time to dig deeper into it. I might add a promises-based wrapper around that. |
Codecov Report
@@ Coverage Diff @@
## main #211 +/- ##
==========================================
+ Coverage 87.87% 87.94% +0.06%
==========================================
Files 60 60
Lines 9249 9301 +52
==========================================
+ Hits 8128 8180 +52
Misses 1121 1121
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
Just thinking out loud, but maybe a better home for this would be adbcdrivermanager (where a dependency on later/promises would be more than worth it to support async). |
Yeah, if we can expose the relevant C interface here, we can support a higher-level interface in adbcdrivermanager. Sequence diagrams FTW! |
I was wondering if the thread should be a property of the connection object. That way we don't need to create and destroy threads, only wake them for each operation (limitless producer-consumer design?). |
That's a great idea. This functionality is much better suited to adbcdrivermanager anyway and keeping all the async stuff in one place will be very helpful. |
Do you plan to further enhance this? |
Sorry for the delay...I was taking some time away from the keyboard! I think the best place for this right now is the ADBC R bindings since there will also be other async-related code and it is probably best to keep that in the same place. I'm happy to take direction on that...you have spent more time considering async database access than I have and the purpose of this change would be for the express purpose of enabling that in ADBC/R. |
Total proof-of-concept of one way to go about async
stream$get_next()
, as discussed briefly with @krlmlr, @lidavidm, and @nbenn in connection with ADBC in R. When the stream is wrapping a database result set, theget_next()
call can block for a considerable amount of time...this async version would support a world where it's easier to cancel running queries since we can loop + wait in R land (possibly calling the brand-new "cancel" function coming to ADBC 1.1).This PR currently will leak memory and doesn't communicate any error information beyond "result code".
See also r-dbi/adbc#6
Reprex:
Created on 2023-06-07 with reprex v2.0.2