Issues reading/writing to delta tables in S3 buckets with name containing dots (deltalake>=0.7.0)

[gray-sat]

Environment

Delta-rs version: deltalake==0.8.1

Binding: Python

Environment: Local / Cloud

• Cloud provider: AWS
• OS: MacOS 12.2.1 / Ubuntu 20.04
• Other:

---

Bug

What happened:

Please let me know if I have missed something glaringly obvious. It seems the read tests related to S3 passed for the 0.8.1 build on at least Python 3.10.10, so I'm not sure why I'm getting this locally.

I first attempted to read a delta table from deltalake==0.6.4 before finding that AWS_PROFILE was not supported for that version. When passing credentials in storage options, I have no issues reading delta tables in S3 using 0.6.4.

Our development flows often use AWS_PROFILE, and I noticed that 0.8.0+ supported that env var so I upgraded to 0.8.1. Now when trying to read a Delta table using either AWS_PROFILE or passing aws credentials instead, I am getting errors.

If I pass the S3 path like s3://<bucket-name>/<prefix-to-table>, I get

deltalake.PyDeltaTableError: Failed to read delta log object: Generic S3 error: URL did not match any known pattern for scheme: s3://<bucket-name>/<prefix-to-table>

I am able to list / download the contents of those paths using aws cli / boto3 using the same credentials.

When using the AWS_PROFILE env, which uses the same credentials as tested above:

>>> DeltaTable("s3://<bucket-name>/<prefix-to-table>")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../lib/python3.10/site-packages/deltalake/table.py", line 122, in __init__
    self._table = RawDeltaTable(
deltalake.PyDeltaTableError: Failed to read delta log object: Generic S3 error: URL did not match any known pattern for scheme: s3://<bucket-name>/<prefix-to-table>

What you expected to happen:
Can read Delta table from S3 without issue when proper credentials are exposed to deltalake.

How to reproduce it:
On M1 Pro Mac:
Create python 3.10.10 or python 3.8.13 virtualenv

pip install deltalake==0.8.1
python
>>> from deltalake import DeltaTable
>>> DeltaTable("s3://<bucket>/<prefix>", storage_options={"AWS_ACCESS_KEY_ID": ACCESS_KEY, "AWS_SECRET_ACCESS_KEY":SECRET_KEY, "AWS_REGION": 'us-east-1'})
>>> exit()
export AWS_PROFILE=<some-profile>
python
>>> from deltalake import DeltaTable
>>> DeltaTable("s3://<bucket>/<prefix>", storage_options={"AWS_REGION": 'us-east-1'})

[gray-sat]

After hitting myself and running a quick test... this is related to dots being present in the bucket name.

The default request style being used is path style, so in theory bucket names with dots should be supported. The error indicates that the issue lies somewhere in the RawDeltaTable constructor. I'm digging in a bit more into the Rust code to see what's changed there -- admittedly I'm not familiar with Rust, so not sure how the root cause investigation is going to go. The change occurred between 0.6.4 -> 0.7.0.

[gray-sat]

After a little digging I think this can be traced to changes that went out as part of 8e0b8cb -- specifically the move from specific backends like S3StorageBackend to the generic DeltaObjectStore with ObjectStoreKind implementations.

Again, I've not use Rust before looking at this issue, so you'll have to look at some ugly code. I downloaded the project to run the following code against the deltalake 0.8.0 crate:

use std::collections::HashMap;
use url::Url;
use deltalake::{DeltaResult, DeltaTableError};
use deltalake::storage::DeltaObjectStore;
use deltalake::storage::config::StorageOptions;

pub(crate) fn ensure_table_uri(table_uri: impl AsRef<str>) -> DeltaResult<Url> {
    let table_uri = table_uri.as_ref();
    if let Ok(path) = std::fs::canonicalize(table_uri) {
        return Url::from_directory_path(path)
            .map_err(|_| DeltaTableError::InvalidTableLocation(table_uri.to_string()));
    }
    if let Ok(url) = Url::parse(table_uri) {
        return Ok(match url.scheme() {
            "file" => url,
            _ => {
                let mut new_url = url.clone();
                new_url.set_path(url.path().trim_end_matches('/'));
                new_url
            }
        });
    }
    // The table uri still might be a relative paths that does not exist.
    std::fs::create_dir_all(table_uri)
        .map_err(|_| DeltaTableError::InvalidTableLocation(table_uri.to_string()))?;
    let path = std::fs::canonicalize(table_uri)

        .map_err(|_| DeltaTableError::InvalidTableLocation(table_uri.to_string()))?;
    Url::from_directory_path(path)
        .map_err(|_| DeltaTableError::InvalidTableLocation(table_uri.to_string()))
}

fn main() {
    let mut mymap = HashMap::new();
    mymap.insert("AWS_REGION".to_string(), "us-east-1".to_string());
    let location = ensure_table_uri("s3://test-bucket/test-prefix").unwrap();
    let something = DeltaObjectStore::try_new(
        location,
        StorageOptions(mymap),
    );
    println!("{}", something.unwrap().root_uri());

    let mut mymap2 = HashMap::new();
    mymap2.insert("AWS_REGION".to_string(), "us-east-1".to_string());
    let location2 = ensure_table_uri("s3://test.bucket/test-prefix").unwrap();
    let something2 = DeltaObjectStore::try_new(
        location2,
        StorageOptions(mymap2),
    );
    println!("{}", something2.unwrap().root_uri());
}

Cargo.toml

[package]
name = "test"
version = "0.1.0"
rust-version = "1.64"
edition = "2021"

[dependencies]
deltalake = {version="0.8.0", features=["s3"]}
url = "2.3"

When running the above code, cargo run outputs:

rust-test % cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.14s
     Running `target/debug/test`
s3://test-bucket/test-prefix
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ObjectStore { source: Generic { store: "S3", source: UrlNotRecognised { url: "s3://test.bucket/test-prefix" } } }', src/main.rs:51:31
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

In other words, it spits out a root_uri just fine when the bucket name is only dashes, but it errors out when the bucket name includes dots. This may be an issue specifically in the object_store crate (https://github.com/apache/arrow-rs/tree/master/object_store), as I do think this should be supported.

[rtyler]

This does look like a bug upstream in object-store, there is an incorrect assumption being made that buckets can not have dots in the name.

I have just validated that this is legal with AWS S3 in `us-east-2.

[rtyler]

I will close this issue once the upstream changes have been incorporated

[gray-sat]

Wow -- super quick. Thank you.

[rtyler]

The change has been merged upstream, but has not yet been released. There are some other API incompatibilities with the latest arrow that we'll be working through in #1249

[rtyler]

This should have been closed a while ago, changes have been released