Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support CREATE EXTERNAL TABLE dt STORED AS DELTATABLE syntax with datafusion #891

Closed
avantgardnerio opened this issue Oct 17, 2022 · 2 comments · Fixed by #892
Closed

Support CREATE EXTERNAL TABLE dt STORED AS DELTATABLE syntax with datafusion #891

avantgardnerio opened this issue Oct 17, 2022 · 2 comments · Fixed by #892
Labels
enhancement New feature or request

Comments

@avantgardnerio
Copy link
Contributor

Description

In datafusion, SQL users can use the create external table xyz stored as csv option to register CSV, JSON, and parquet data. It would be nice (essential in my company's case) to allow users to register deltalake tables in the same manner.
@avantgardnerio avantgardnerio added the enhancement New feature or request label Oct 17, 2022
@avantgardnerio avantgardnerio mentioned this issue Oct 17, 2022
Merged
@MrPowers
Copy link
Contributor

@avantgardnerio - this is awesome. Can't wait to have Datafusion support! I will definitely help update the docs / write blog posts when this functionality is added.

@avantgardnerio
Copy link
Contributor Author

Thanks! Related datafusion PR if you'd like to follow along: apache/datafusion#3907 . I've definitely seen it work in Datafusion, so I don't think support is far away for that. Ballista will be some more work as we'll need to do serde on the physical plan, and re-register the object store on the executors.

wjones127 pushed a commit that referenced this issue Nov 17, 2022
Add TableProviderFactory and test for SQL to register tables dynamica…
a8f2ff2 
…lly at runtime (#892)

# Description

Use (hopefully soon-to-be-merged) Datafusion functionality to allow
dynamic (i.e. SQL-based) registration of deltalake tables at runtime.

# Related Issue(s)
- closes #891

# Documentation

- TBD
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
2 participants
@MrPowers @avantgardnerio