Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] Use _collect_as_arrow for fugue_api.as_arrow(spark_df) #516

Closed
ion-elgreco opened this issue Sep 22, 2023 · 3 comments · Fixed by #511
Closed

[FEATURE] Use _collect_as_arrow for fugue_api.as_arrow(spark_df) #516

ion-elgreco opened this issue Sep 22, 2023 · 3 comments · Fixed by #511

Comments

@ion-elgreco
Copy link

Is your feature request related to a problem? Please describe.
Convert spark df to arrow can be done with private method inside pyspark: _collect_as_arrow https://github.com/apache/spark/blob/06ccb6d434476afacc08936cf473670102d41010/python/pyspark/sql/pandas/conversion.py#L244
@ion-elgreco ion-elgreco changed the title [FEATURE] Use _collect_as_arrow for sparkdf.as_arrow() [FEATURE] Use _collect_as_arrow for fugue_api.as_arrow(spark_df) Sep 22, 2023
@goodwanghan
Copy link
Collaborator

This is a great idea, will release a dev version

@goodwanghan goodwanghan linked a pull request Oct 5, 2023 that will close this issue
Improve comments to be LLM friendly, improve Spark compatibility #511
Merged
@goodwanghan
Copy link
Collaborator

@ion-elgreco please try 0.8.7.dev5, it uses _collect_as_arrow

@ion-elgreco
Copy link
Author

@ion-elgreco please try 0.8.7.dev5, it uses _collect_as_arrow

Nice, I'll try it out later this week!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Improve comments to be LLM friendly, improve Spark compatibility fugue-project/fugue
2 participants
@ion-elgreco @goodwanghan