[data] Read previous versions of Lance dataset into ray data

[iatkinson]

Description

When reading a lance dataset using ray.data.read_lance there is no way to specify the lance version.

Use case

Thanks for the great package.

I am looking for a way to read Lance versions other than the latest into Ray data.

[bveeramani]

Hey @iatkinson, thanks for opening this Issue.

What do you envision the API for this would look like?

[iatkinson]

@bveeramani

This could be pretty basic, add a version param to read_lance so becomes:

def read_lance(  # noqa: PLR0913
    uri: str,
    *,
    version: Optional[int | str] = None,  # NEW PARAM
    columns: Optional[List[str]] = None,
    filter: Optional[str] = None,  # noqa: A002
    storage_options: Optional[Dict[str, str]] = None,
    scanner_options: Optional[Dict[str, Any]] = None,
    ray_remote_args: Optional[Dict[str, Any]] = None,
    concurrency: Optional[int] = None,
    override_num_blocks: Optional[int] = None,
) -> Dataset:

Calling then can be similar to

read_lance(some_dataset, version=2, ...)

And in the impl the version parameter would be passed to the LanceDatasource (also updated to take version) and ultimately the lance.dataset (already accepts version) call made in LanceDatasource.__init__

[bveeramani]

That sounds reasonable to me!

@iatkinson would you down to open a PR for this? I can guide you if you have any questions

[simeetnayan81]

Hey @iatkinson @bveeramani mind if I take this up?
I can ping in slack if I get stuck

[bveeramani]

@simeetnayan81 go for it!

Do you have any questions/is there anything I can clear up?