-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement select * from 'foo.parquet'
support
#125
Comments
It would be really cool if you could inspect parquet metadata with this |
This seems like a really fun thing to work on. If no one else is working on it I'd love to take it on. I really like the metadata idea. |
Contributions very welcome :) IMO it would be best to first integrate the I also just realized that their may be some existing metadata functionality for datafusion although its not clear to me from the docs whether that is only in datafusion-cli or a built-in function to datafusion that we could also use. If it was specific to datafusion cli then it would be great if we could add that. |
Also you could potentially get some inspiration for additional metadata capabilities from duckdb |
BTW we can take inspiration from / copy outright the I would like to suggest creating those functions in their own crate (perhaps In fact maybe once dft gets good enough we could remove the |
Sorry I missed this -- it is only in datafusion-cli Implementation is here: https://github.com/apache/datafusion/blob/257e1409eca81cfff024ecc5e2567e9f67e6b5a3/datafusion-cli/src/functions.rs#L317-L459 |
I suggest we file a second ticket for implementing parquet_metadata and other duckdb metadata functions |
Sounds good -- I'll go ahead and get these both assigned to myself and then start cracking on it in the next few days :) |
take edit: looks like github actions is not set up to auto-assign like datafusion 😅 |
How about you take #148 and I'll try this one? DataFusion 42 was just released and I very much want this particular feature in |
I want queries like this to work:
This works great in datafusion-cli:
It currently doesn't in
dft
Once datafusion 42.0.0 is released, we can likely use the
DynamicFileProvider
that @goldmedal added in apache/datafusion#11035The text was updated successfully, but these errors were encountered: