Update Lazy Polars to support streaming and in-built parquet support #838
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Pascal Bro <git@pascalbrokmeier.de>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Currently playing with Polars and noticing that we don't support streaming which is quite nice for larger than memory datasets.
Note I'm just opening this as a trigger for discussion, I realize this is not yet done and cannot promise I will continue on this since this was just me playing around on the weekend. But hoping I get to play more and bring this into our pipeline (and thus here) :)
Development notes
Updated to the latest features of polars which includes streaming=True and leaving the option to the user to do partitioned writing using rust based parquet support
Checklist
RELEASE.mdfile