lakefs-spec - a Python fsspec implementation for lakeFS

[nicholasjng]

Link to the project: https://github.com/appliedAI-Initiative/lakefs-spec

The title says it all - with lakefs-spec, you can easily pull (and push) files from a lakeFS instance. As an added bonus, a growing number of Python data science projects support fsspec, so e.g. reading a Pandas DataFrame in a versioned manner is now a one-liner:

import pandas as pd

storage_options={
    "host": "localhost:8000",
    "username": "username",
    "password": "password",
}

df = pd.read_parquet('lakefs://quickstart/main/lakes.parquet', storage_options=storage_options)

Feedback, ideas, and contributions are most welcome!

[ozkatz]

This is incredible, thank you for building and releasing this!
Implementation looks great, and so do the tests :)

A couple of ideas for enhancements:

1. Would _love to see support for pre-signed URLs, maybe by setting a storage option?
   This would allow users that transfer a lot of data back and forth to not have to go through the lakeFS server.
2. Perhaps read ~/.lakectl.yaml if exists, or environment variables if set, to configure the client automatically from the environment and avoid having to pass storage_options on every call?

[nicholasjng]

Thanks for the feedback and the enhancements! We implemented 2) now, so that zero-config initialization is possible:

cat ~/.lakectl.yaml

# credentials:
#     access_key_id: AKIAIOSFOLQUICKSTART
#     secret_access_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
# server:
#     endpoint_url: http://127.0.0.1:8000

# with the above config, we can just do:
import pandas as pd

df = pd.read_parquet('lakefs://quickstart/main/lakes.parquet')

Pre-signed URL support also sounds really interesting. Is there some documentation on how to use pre-signed URLs with lakeFS (ideally from the Python client)?