Not able to connect to Azure Storage with client id/secret

[rgelsi]

deltalake >= v0.6.2

Since deltalake version v0.6.2 I am no longer able to load Delta Lake tables from an Azure Data Lake Storage Gen2 via Azure client id/secret.

import pandas as pd
import pyarrow.dataset as ds
from deltalake.table import DeltaTable

azure_container = "CONTAINER"
delta_table_path = "PATH/TO/TABLE.delta"

storage_options = {
    "AZURE_STORAGE_TENANT_ID": azure_storage_tenant_id,
    "AZURE_STORAGE_CLIENT_ID": azure_storage_client_id,
    "AZURE_STORAGE_CLIENT_SECRET": azure_storage_client_secret,
    "AZURE_STORAGE_ACCOUNT_NAME": azure_storage_account_name
}

delta = DeltaTable(f"az://{azure_container}/{delta_table_path}", storage_options=storage_options)

Error log:

---------------------------------------------------------------------------
PyDeltaTableError                         Traceback (most recent call last)
Input In [9], in <cell line: 1>()
----> 1 delta = DeltaTable(f"az://{azure_container}/{delta_table_path}", storage_options=storage_options)

File /opt/conda/lib/python3.9/site-packages/deltalake/table.py:103, in DeltaTable.__init__(self, table_uri, version, storage_options, without_files)
     90 """
     91 Create the Delta Table from a path with an optional version.
     92 Multiple StorageBackends are currently supported: AWS S3, Azure Data Lake Storage Gen2, Google Cloud Storage (GCS) and local URI.
   (...)
    100                       DeltaTable will be loaded with a significant memory reduction.
    101 """
    102 self._storage_options = storage_options
--> 103 self._table = RawDeltaTable(
    104     table_uri,
    105     version=version,
    106     storage_options=storage_options,
    107     without_files=without_files,
    108 )
    109 self._metadata = Metadata(self._table)

PyDeltaTableError: Generic DeltaTable error: Failed to find valid credential.

As far as I have understood, the following mappings for the client_secret azure configuration are missing in ALIAS_MAP in rust/src/builder/azure.rs. Before v0.6.2 the file rust/src/builder.rs had the config AZURE_STORAGE_CLIENT_SECRET.

        // client secret
        ("azure_storage_client_secret", AzureConfigKey::ClientSecret),
        ("azure_client_secret", AzureConfigKey::ClientSecret),
        ("client_secret", AzureConfigKey::ClientSecret),

[roeap]

@rgelsi - thanks for the report - will have a look into this.

[roeap]

reopening to await confirmation.

[stinodego]

Hi @roeap , we ran into this issue and have verified that the version in the main branch fixes the issue. We are eagerly awaiting a new release with this functionality. Do you by chance have an indication of when the next release will be?

We know we can build manually, but we try to avoid doing so on our production systems whenever possible.

[roeap]

@stinodego - a new release is not far away. Best guess we can get it released within roughly a week.

[stinodego]

Great, thanks for the update!

[NilsHahnBayer]

@rgelsi @stinodego has the issue been solved for you with the latest release v0.6.4?

[stinodego]

@rgelsi @stinodego has the issue been solved for you with the latest release v0.6.4?

@NilsHahnBayer No, this issue was introduced in v0.6.2 and the fix has been merged after v0.6.4 was released. We are eagerly awaiting the v0.6.5 release, as we're currently limited to using v0.6.1, but we would like to use some of the features introduced more recently.

[rgelsi]

@rgelsi @stinodego has the issue been solved for you with the latest release v0.6.4?

@NilsHahnBayer No, this issue was introduced in v0.6.2 and the fix has been merged after v0.6.4 was released. We are eagerly awaiting the v0.6.5 release, as we're currently limited to using v0.6.1, but we would like to use some of the features introduced more recently.

@NilsHahnBayer, I am also still waiting for the release of python-0.6.5

[stinodego]

@roeap Sorry to ping you - is there any chance for a 0.6.5 release soon? If not, what is the blocker? I might be able to help out.

[roeap]

@stinodego - no worries, a new release is overdue since there are a bunch of unreleased fixes on main. I opened #1073 to track if we are missing something.

[rgelsi]

@NilsHahnBayer the issue has been solved with the release python-v0.7.0.
Thanks a lot, @roeap