Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] as_dicts for DataFrames #521

Closed
goodwanghan opened this issue Oct 20, 2023 · 0 comments · Fixed by #522
Closed

[FEATURE] as_dicts for DataFrames #521

goodwanghan opened this issue Oct 20, 2023 · 0 comments · Fixed by #522

Comments

@goodwanghan
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
Now we only has as_dict_iterable, but if we want to get an iterable, the execution behavior can be very different. Spark is the best example, when there are a lot of partitions (thousands of), as_dict_iterable can be very slow because it will go through partition by partition.

Describe the solution you'd like
as_dicts means getting dicts as a whole, with this semantic, we could collect all data in parallel, and then convert to dicts.

@goodwanghan goodwanghan added this to the 0.8.7 milestone Oct 20, 2023
@goodwanghan goodwanghan linked a pull request Oct 29, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant