feat: propagate storage_options to LanceFileWriter and LanceFileReader

[ankitvij-db]

The PR propogates the storage_options to the LanceFileWriter and LanceFileReader and updates the code to create the object reader.

How is this tested?

Installed the package with local change:

maturin develop

Verified the following code didn't work before but works now

from lance.file import LanceFileReader, LanceFileWriter
import pyarrow as pa

uri = 's3://xxxxx'
storage_options = {
    'aws_access_key_id': 'xxx',
    'aws_secret_access_key': 'xxx'
}

table = pa.table({"a": [1, 2, 3]})
path = f"{lance_uri}/foo.lance"
global_buffer_text = "hello"
global_buffer_bytes = bytes(global_buffer_text, "utf-8")
with LanceFileWriter(str(path),storage_options=lance_storage_options) as writer:
    writer.write_batch(table)
    global_buffer_pos = writer.add_global_buffer(global_buffer_bytes)
reader = LanceFileReader(str(path), storage_options=lance_storage_options)
read_table = reader.read_all().to_table

[wjones127]

Thanks for making a PR for this.

Could you add an integration test so we can validate storage options are working? Here is where we have integration tests where we pass storage options to connect to S3: https://github.com/lancedb/lance/blob/main/python/python/tests/test_s3_ddb.py

[wjones127]

Nice work!

[ankitvij-db]

@wjones127 I have added the IT. However, will the wheel used for ITs be generated using local changes or will it pull a released version?

[ankitvij-db]

@wjones127 Actually there was a race when I requested a re-review and when you approved. Can you re-stamp?

[wjones127]

@wjones127 I have added the IT. However, will the wheel used for ITs be generated using local changes or will it pull a released version?

They are always generated with local changes.