refactor!: introduce storage options accessor

[jackye1995]

Same as #5547, but that one is too hard to rebase at this moment.

For easier integration with distributed engines, we want a way to get dataset.latest_storage_options() through the initial static storage options + the dynamic storage options provider.

However, current implementation pushes this logic down to AWS. This PR lifts the logic up into a component StorageOptionsAccessor, so that we can invoke it at dataset level, and also just wrap it as a CredentialsProvider for AWS. We also make ObjectStoreParams now hold a StorageOptionsAccessor instead of just the storage_options map.

For user API, we continue to let them input storage options and storage options provider, StorageOptionsAccessor is used only internally.

We also remove a few features added that turned out to be unnecessary in the whole credentials vending code path:

1. ignore_namespace_storage_options: this just never turns out to be possible to be set to true, I was overthinking in the beginning, so remove related code.
2. s3_credentials_refresh_offset_seconds: we overloaded this feature to set credentials refresh lead time for the storage options provider-based AWS credentials provider. But this is now not fitting the framework since the feature will be applicable to gcp and azure as well. Note that I removed all the related code in python and java. Although s3_credentials_refresh_offset exists for a long time, there was not really a way to set it in python and java. I added it for storage options provider only, but now it is no longer needed.

Note that recently I added credentials vending server side support in DirectoryNamespace for all 3 clouds. So we can now provide a generic solution to vend credentials and test everything end to end for GCP and Azure. However this PR still keeps the feature in AWS, we will do another PR to add GCP and Azure support officially

[github-actions]

Code Review

P0: Compilation Error in Tests

The tests in rust/lance-io/src/object_store/providers/aws.rs will not compile. The function signatures for from_provider and from_provider_with_initial have changed to remove the Duration parameter, but the test calls still include it:

New signatures:

pub fn from_provider(provider: Arc<dyn StorageOptionsProvider>) -> Self
pub fn from_provider_with_initial(
    provider: Arc<dyn StorageOptionsProvider>,
    initial_options: HashMap<String, String>,
) -> Self

Test calls still have extra Duration argument:

// These calls have 2 or 3 args, but should have 1 or 2
DynamicStorageOptionsCredentialProvider::from_provider(
    mock.clone(),
    Duration::from_secs(300), // <- Should be removed
);

DynamicStorageOptionsCredentialProvider::from_provider_with_initial(
    mock.clone(),
    Duration::from_secs(300), // <- Should be removed
    initial_options,
);

Similarly, StorageOptionsAccessor::with_provider is called with a Duration in documentation/tests but the actual signature only takes one argument.

Please verify this builds and tests pass before merging.

---

P1: Breaking API Change Without Migration Path

The PR removes s3_credentials_refresh_offset_seconds from all public APIs (Python, Java, Rust) without a clear migration path for users who were relying on this parameter. While the new approach of using refresh_offset_millis in storage options is cleaner, existing users will have their code break silently (the parameter is removed, not deprecated with a warning).

Consider adding a migration note to the release notes or emitting a warning if users try to use the old parameter name.

---

Minor: Test Coverage

The new StorageOptionsAccessor has good unit test coverage. The concurrent access test is particularly valuable.

[codecov]

Codecov Report

❌ Patch coverage is 70.10786% with 194 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/builder.rs	9.09%	69 Missing and 1 partial ⚠️
rust/lance/src/dataset.rs	0.00%	53 Missing and 2 partials ⚠️
rust/lance-io/src/object_store/storage_options.rs	84.33%	45 Missing and 2 partials ⚠️
rust/lance-io/src/object_store/providers/aws.rs	90.22%	13 Missing ⚠️
rust/lance/src/dataset/fragment/write.rs	0.00%	4 Missing ⚠️
rust/lance-namespace-impls/src/dir.rs	72.72%	3 Missing ⚠️
rust/lance-io/src/object_store.rs	95.23%	1 Missing ⚠️
rust/lance-io/src/object_store/providers.rs	75.00%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[hamersaw]

does with_static_options give more uniformity re: naming?