Update Polars on Arrow2 and implement Polars on Arrow-rs #726
Conversation
Update arrow-rs with .polars
Update polars
connectorx/Cargo.toml
Outdated
| @@ -71,11 +72,12 @@ iai = "0.1" | |||
| pprof = {version = "0.5", features = ["flamegraph"]} | |||
|
|
|||
| [features] | |||
| all = ["src_sqlite", "src_postgres", "src_mysql", "src_mssql", "src_oracle", "src_bigquery", "src_csv", "src_dummy", "src_trino", "dst_arrow", "dst_arrow2", "federation", "fed_exec"] | |||
| all = ["src_sqlite", "src_postgres", "src_mysql", "src_mssql", "src_oracle", "src_bigquery", "src_csv", "src_dummy", "src_trino", "dst_arrow", "dst_arrow2", "dst_arrow", "federation", "fed_exec", "dst_polars"] | |||
There was a problem hiding this comment.
a duplicate dst_arrow?
|
Thanks for drafting the design and submitting the PR! @EricFecteau I will take a deeper look at this by this weekend. |
|
@EricFecteau , the code looks good to me! In terms of resolving the array type issue, I left a comment in the discussion |
Added arrays & returned to polars 0.32 for arrow2
|
@wangxiaoying This was simpler than I anticipated (great Crate design)! I implemented arrays for arrow and reverted to using polars_old = 0.32 for arrow2. I am interested in doing another PR to remove arrow2 from the crate, but I think it deserves it’s own PR. Now all the test (plus the tests I added) pass. Please let me know what you think. More info in the discussion: #720 (reply in thread) |
|
Thank you for the update @EricFecteau ! I did some minor changes but I think the code looks good in general. Will merge it after the CI passes. I left some comments in the discussion for concrete questions. |
|
This PR also closes #657. |
Added the ability to send the data to a modern version of Polars from both Arrow and Arrow2, through the FFI.
Some design decisions were made along the way (described here: #720), but more than willing to change those decisions! Had to disable two tests for Arrow2 as the data was not able to be moved between arrow2 and arrow_polars and these the functionality needed was also not implemented in Arrow.