Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDFS storage support via datafusion-objectstore-hdfs #1279

Merged
merged 10 commits into from
Apr 14, 2023

Conversation

iajoiner
Copy link
Contributor

@iajoiner iajoiner commented Apr 13, 2023

Description

The description of the main changes of your pull request
This is basically the continuation of #1273 (really thanks @yjshen for your contributions! The work is all his..I'm just the guy who tries to get it merged!). I addressed the comments from @wjones127.

Related Issue(s)

Documentation

@github-actions github-actions bot added binding/rust Issues for the Rust crate documentation Improvements or additions to documentation rust labels Apr 13, 2023
@iajoiner iajoiner changed the title Hdfs read HDFS storage support via datafusion-objectstore-hdfs Apr 13, 2023
@iajoiner
Copy link
Contributor Author

@yjshen Please check out the integration tests: https://github.com/delta-io/delta-rs/actions/runs/4684975786/jobs/8301645920?pr=1279

There are plenty of java.io.FileNotFoundExceptions. However all the HDFS tests nominally passed. Is this intended behavior?

@iajoiner
Copy link
Contributor Author

Looks like the errors are really just us not being able to read non-parquet files in Delta when reading Parquet files. So I guess they are ok?

Copy link
Member

@rtyler rtyler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my perspective this looks ready to merge. I would like @wjones127 to click the green button though since he had comments on the original pull request

@rtyler
Copy link
Member

rtyler commented Apr 13, 2023

@iajoiner please merge main to resolve the conflicts

@iajoiner
Copy link
Contributor Author

@rtyler Really thanks! I just merged main and there are no conflicts haha.

Copy link
Collaborator

@wjones127 wjones127 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding tests @iajoiner! This looks good to me 👍

@wjones127 wjones127 merged commit 4f9665d into delta-io:main Apr 14, 2023
@iajoiner iajoiner deleted the hdfs_read branch April 14, 2023 03:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
binding/rust Issues for the Rust crate documentation Improvements or additions to documentation rust
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants