Azure: Support multiple storage credential prefixes #13241

ChaladiMohanVamsi · 2025-06-04T19:17:35Z

Associated Issue : #13050

Changes

Add support for multiple storage credential prefixes in ADLSFileIO
ADLSFIleIO implements SupportsStorageCredentials to pass storage credentials from LoadTableResponse to FileIO

Similar PRs for other FileIOs

ChaladiMohanVamsi · 2025-06-04T19:18:53Z

Yet to implement the test cases.

azure/src/main/java/org/apache/iceberg/azure/adlsv2/PrefixedADLSFileClientBuilder.java

…ntation.

ChaladiMohanVamsi · 2025-06-19T16:36:05Z

@nastra @amogh-jahagirdar Can you please help with the review.

amogh-jahagirdar

Thank you for the change @ChaladiMohanVamsi , I'm Still going through tests a bit more but left some comments

amogh-jahagirdar · 2025-06-20T02:01:41Z

azure/src/main/java/org/apache/iceberg/azure/adlsv2/ADLSFileIO.java

-    return client(location).getFileClient(location.path());
+    Preconditions.checkState(
+        null != prefixedADLSClient,
+        "[BUG] ADLS client-builder for storage path not available: %s",


the "[BUG]" callout doesn't really fit in the pattern of error messages in the project, can we just say "Cannot build ADLS cllient for path: %s"

btw we're using the same logging pattern (which I added) in S3FileIO and GcsFileIO

Keeping it consistent with other FileIO implementations as of now.

amogh-jahagirdar · 2025-06-20T02:29:02Z

azure/src/integration/java/org/apache/iceberg/azure/adlsv2/ADLSFileIOTest.java

+  public void multipleStorageCredentialsConfigured() {
+    StorageCredential adlsCredential1 =
+        StorageCredential.create(
+            abfsPath("custom-container", "account1", "/table1"),


Can we move some of these abfsPath values into separate variables; it'll reduce duplication and make it easier to verify these tests are doing what they should be

amogh-jahagirdar · 2025-06-20T02:46:17Z

build.gradle

    testImplementation libs.testcontainers
    testImplementation libs.mockserver.netty
    testImplementation libs.mockserver.client.java
+    testImplementation(libs.hadoop3.common) {


Why did this need to change? what are we pulling from hadoop for the new tests?

In few test cases where we initialise ResolvingFileIO, it is failing with missing HaddopConf dependency. Hence had include this in test scope.

amogh-jahagirdar · 2025-06-20T02:54:46Z

azure/src/main/java/org/apache/iceberg/azure/adlsv2/PrefixedADLSClient.java

+    Optional.ofNullable(vendedAdlsCredentialProvider)
+        .map(p -> new VendedAzureSasCredentialPolicy(location.host(), p))
+        .ifPresent(clientBuilder::addPolicy);


Nit: I know this was just copied over from what already existed but imo this logic would just be a bit clearer with plain old null check and assignment:

if (credentialProvider != null) { clientBuilder.addPolicy(new VendedAzureSasCredentialPolicy(location.host(), credentialProvider))) }

Updated as per the suggestion.

amogh-jahagirdar · 2025-06-20T02:59:03Z

azure/src/integration/java/org/apache/iceberg/azure/adlsv2/ADLSFileIOTest.java

+    assertThat(fileIO.clientForStoragePath(abfsPath("custom-container", "account1", "/table1")))
+        .isNotSameAs(
+            fileIO.clientForStoragePath(abfsPath("custom-container", "account1", "/table2")))
+        .isNotSameAs(
+            fileIO.clientForStoragePath(abfsPath("my-container", "account1", "/path/to/file")));


I feel like this test isn't as strong as it could be; we're asserting that the clients for these paths are all different which is true but we're not asserting the actual client right?

Unlike S3FileIO and GCSFileIO we are not caching the clients in ADLSFileIO, we are always creating a new client with reused HTTP client during any FileIO operation. I kept the behaviour same even with prefix based clients. Hence we are not validating client as they would be different for each file.

I am not completely sure of overall benefit we can get out of caching the clients as we are already reusing HTTP client. Wanted your inputs if we should go in this route of caching the clients similar to other FileIO implementations.

cc// @nastra

amogh-jahagirdar · 2025-06-20T03:10:50Z

azure/src/integration/java/org/apache/iceberg/azure/adlsv2/ADLSFileIOTest.java

+    assertThat(
+            fileIO.clientForStoragePath(abfsPath("custom-container", "account1", "/path/to/file")))
+        .isNotSameAs(
+            fileIO.clientForStoragePath(abfsPath("my-container", "account1", "/path/to/file")));
+
+    assertThat(fileIO.clientForStoragePath(abfsPath("my-container", "account1", "/path/to/file")))
+        .isSameAs(
+            fileIO.clientForStoragePath(abfsPath("my-container2", "account2", "/path/to/file")));
+


Same as what I mentioned somewhere else in this test, could we make sure some of these abfsPath(...) are moved to separate values and can we make sure these tests are strong as possible beyond just comparing if it's the same as another path.

azure/src/integration/java/org/apache/iceberg/azure/adlsv2/PrefixedADLSClientTest.java

azure/src/main/java/org/apache/iceberg/azure/adlsv2/ADLSFileIO.java

azure/src/main/java/org/apache/iceberg/azure/adlsv2/ADLSLocation.java

azure/src/main/java/org/apache/iceberg/azure/adlsv2/ADLSOutputFile.java

azure/src/main/java/org/apache/iceberg/azure/adlsv2/PrefixedADLSClient.java

azure/src/integration/java/org/apache/iceberg/azure/adlsv2/PrefixedADLSClientTest.java

azure/src/main/java/org/apache/iceberg/azure/adlsv2/ADLSFileIO.java

nastra · 2025-06-25T07:15:19Z

azure/src/main/java/org/apache/iceberg/azure/adlsv2/ADLSFileIO.java

+      clientByPrefix.values().forEach(PrefixedADLSClient::close);
+      this.clientByPrefix = null;
    }
-


we should keep the newline after }

# Conflicts: # azure/src/main/java/org/apache/iceberg/azure/adlsv2/ADLSFileIO.java

nandorKollar · 2025-07-04T17:09:30Z

azure/src/main/java/org/apache/iceberg/azure/adlsv2/PrefixedADLSClient.java

+    return azureProperties;
+  }
+
+  public String storagePrefix() {


nit: maybe we can keep this package private until there's a good reason to make it public?

Agree, made it package private.

github-actions · 2025-08-04T00:21:10Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

ChaladiMohanVamsi added 3 commits June 4, 2025 20:40

Pass storage credentials from LoadTableResponse to FileIO

4021f7b

Support multiple storage credential prefixes

6e0e1ae

Add log statement.

e4e5d04

github-actions bot added the AZURE label Jun 4, 2025

nandorKollar reviewed Jun 4, 2025

View reviewed changes

azure/src/main/java/org/apache/iceberg/azure/adlsv2/PrefixedADLSFileClientBuilder.java Outdated Show resolved Hide resolved

nandorKollar reviewed Jun 4, 2025

View reviewed changes

azure/src/main/java/org/apache/iceberg/azure/adlsv2/PrefixedADLSFileClientBuilder.java Outdated Show resolved Hide resolved

refactor and implement test cases for prefix based credential impleme…

c791ec7

…ntation.

github-actions bot added the build label Jun 7, 2025

Merge branch 'master' into mohan/adls-prefix-based-credentials

65d3a09

amogh-jahagirdar reviewed Jun 20, 2025

View reviewed changes