polars backend?

[pwwang]

https://pola-rs.github.io/polars

Efforts:
https://github.com/markfairbanks/tidypolars

[AhmedAliStats]

It would be really nice given the performance benefits.

[GitHunter0]

Hey @pwwang , datar version 0.7.0 release note states the support for modin backend, however there is no documentation about that feature. Is modin really supported? It would be very useful to have it (and easier since it uses the same syntax as pandas).

[pwwang]

It was there but most of tests failed when I tried to enable unittest for it. So I removed it. The syntax is not exactly the same as pandas as they claimed, especially the fundamental functions that are necessary for us to wrap up the APIs. It needs a lot of effort to really support it and might need a totally separate backend (datar-modin?). Let's see when they release a stable major version (1.x.x) and if it is worth the effort. It's the same reason (not 1.x.x yet) why polars support is lagging behind.

[GitHunter0]

@pwwang , got it, thank you for the clarification.

[GitHunter0]

This seems promising too:

• https://github.com/Eventual-Inc/Daft
• https://www.getdaft.io/

[pwwang]

Thanks for sharing. worth keeping an eye on.

[GitHunter0]

Another benchmark for Daft, maybe I'm wrong but it seems to be soon the best backend.
https://delta.io/blog/daft-delta-lake-integration/

[pwwang]

Looks interesting