fix: fix storage options for dataset builder

[chenkovsky]

No description provided.

[wjones127]

This fix looks reasonable, but we don't have a test for it, which worries me.

Do you know why this test doesn't catch it?

lance/python/python/tests/test_s3_ddb.py

Lines 230 to 243 in d79e870

	@pytest.mark.integration
	def test_s3_ddb_distributed_commit(s3_bucket: str, ddb_table: str):
	table_name = uuid.uuid4().hex
	table_dir = f"s3+ddb://{s3_bucket}/{table_name}?ddbTableName={ddb_table}"
	schema = pa.schema([pa.field("a", pa.int64())])
	fragments = write_fragments(
	pa.table({"a": pa.array(range(1024))}),
	f"s3+ddb://{s3_bucket}/distributed_commit?ddbTableName={ddb_table}",
	storage_options=CONFIG,
	)
	operation = lance.LanceOperation.Overwrite(schema, fragments)
	ds = lance.LanceDataset.commit(table_dir, operation, storage_options=CONFIG)
	assert ds.count_rows() == 1024

Would it catch this if we added an append part to that test?

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.

Project coverage is 78.64%. Comparing base (1d3b204) to head (1123e5e).
Report is 22 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/fragment/write.rs	0.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3156      +/-   ##
==========================================
+ Coverage   77.95%   78.64%   +0.68%     
==========================================
  Files         242      243       +1     
  Lines       81904    82860     +956     
  Branches    81904    82860     +956     
==========================================
+ Hits        63848    65162    +1314     
- Misses      14890    14919      +29     
+ Partials     3166     2779     -387     

Flag	Coverage Δ
unittests	78.64% <0.00%> (+0.68%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[chenkovsky]

@wjones127 could you please review it again?

[wjones127]

How does this test append? There is no existing dataset at s3://{s3_bucket}/test_append.lance, is there?

[chenkovsky]

Yes, no existing dataset. before checking the dataset. object store will first check bucket name. without this patch, it will throw bucket not found exception.

[wjones127]

Please don't put this test in test_fragment.py. I'd like to keep all the integration tests in lance/python/python/tests/test_s3_ddb.py

[chenkovsky]

moved.