feat(rust, python): add HDFS support via hdfs-native package

[Kimahriman]

Description

Add support for HDFS using hdfs-native, a pure* Rust client for interacting with HDFS. Creates a new hdfs sub-crate, adds it as a feature to deltalake meta crate, and includes it in Python wheels by default. There is a Rust integration test that requires Hadoop and Java to be installed, and makes use of a small Maven program I ship under the integration-test feature flag to run a MiniDFS server.

*Dynamically loads libgssapi_krb5 using libloading for Kerberos support

Related Issue(s)

Resolves #2611

Documentation

[ion-elgreco]

@Kimahriman LGTM! Can you please also add some docs on the integration in the .MD files and if possible links some docs of the possible configs that can be set

@rtyler can you also go over it?

[Kimahriman]

@Kimahriman LGTM! Can you please also add some docs on the integration in the .MD files and if possible links some docs of the possible configs that can be set

Yeah I agree there should be some, but I didn't see anything for other storage backends so I wasn't sure where to add it. Any recommendation? A new page under integrations/?

[ion-elgreco]

@Kimahriman LGTM! Can you please also add some docs on the integration in the .MD files and if possible links some docs of the possible configs that can be set

Yeah I agree there should be some, but I didn't see anything for other storage backends so I wasn't sure where to add it. Any recommendation? A new page under integrations/?

Yeah under integrations makes sense, perhaps called object storage and then a page for hdfs there

@avriiil any inputs on this? Having some small explanation per object store for S3, adls, gcs and mounted storage would make sense, do you want to help on this?

[Kimahriman]

Yeah under integrations makes sense, perhaps called object storage and then a page for hdfs there

I added a page. I couldn't get the docs to build locally to verify things though. Just kept getting

griffe.exceptions.AliasResolutionError: Could not resolve alias deltalake._internal.DeltaError pointing at _internal.DeltaError (in python/deltalake/_internal.abi3.so:None)

[ion-elgreco]

Yeah under integrations makes sense, perhaps called object storage and then a page for hdfs there

I added a page. I couldn't get the docs to build locally to verify things though. Just kept getting

griffe.exceptions.AliasResolutionError: Could not resolve alias deltalake._internal.DeltaError pointing at _internal.DeltaError (in python/deltalake/_internal.abi3.so:None)

Yeah there is something broken with the docs for some time now

[ion-elgreco]

Thanks for adding this!

[avriiil]

Late to the party, I was OOO for a couple of days. Happy to take a look at creating some more docs if that still makes sense @ion-elgreco? ~ ~ ~ *Avril Aysha* DevRel & Content +351 963681498

…

On Fri, 21 Jun 2024 at 07:22, Ion Koutsouris ***@***.***> wrote: Merged #2612 <#2612> into main. — Reply to this email directly, view it on GitHub <#2612 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQLWMSQ4OSQHETPDAUM7PSLZIPBA7AVCNFSM6AAAAABJSWGJNKVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJTGIZTQMBSG4YDSNY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ion-elgreco]

@avriiil yeah some docs for each object store would be great 😃

[avriiil]

sounds good, adding this to my list for next week @ion-elgreco