Deltalake 0.5 crate s3 feature dynamodb version mismatch

[mpetri]

Environment

Delta-rs version: 0.5

Binding: rust cargo crate 0.5

Environment:

• Cloud provider: aws
• OS:
• Other:

---

Bug

What happened: trying to compile a simple program that enables the s3 feature fails to compile due to a version mismatch in rusoto_dynamodb dependency

What you expected to happen: program should compile

How to reproduce it:

sample program:

#[tokio::main]
async fn main() {
    let table = deltalake::open_table_with_version("s3://foo", 0).await?;
    let files = table.get_files();
    println!("files = {:?}", files);
}

Cargo.toml

[package]
name = "delta_test"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
deltalake = { version = "0.5.0", features = ["s3"] }
tokio = { version = "1.22.0", features = ["full"] }

command:

cargo build --release
   Compiling deltalake v0.5.0
error[E0308]: mismatched types
   --> /home/mpetri/.cargo/registry/src/github.com-1ecc6299db9ec823/deltalake-0.5.0/src/storage/s3.rs:523:17
    |
522 |             let lock_client = dynamodb_lock::DynamoDbLockClient::new(
    |                               -------------------------------------- arguments to this function are incorrect
523 |                 dynamodb_client,
    |                 ^^^^^^^^^^^^^^^ expected struct `rusoto_dynamodb::generated::DynamoDbClient`, found struct `DynamoDbClient`
    |
    = note: perhaps two different versions of crate `rusoto_dynamodb` are being used?
note: associated function defined here

More details:

cargo tree shows (abbreviated):

delta_test v0.1.0 (/local/home/mpetri/workspaces/mpetri/delta_test)
├── anyhow v1.0.66
├── deltalake v0.5.0
│   ├── arrow v26.0.0
│   │   ├── ahash v0.8.2
│   │   │   ├── cfg-if v1.0.0
│   ├── chrono v0.4.23 (*)
│   ├── dynamodb_lock v0.4.1
│   │   ├── async-trait v0.1.58 (proc-macro) (*)
│   │   ├── rusoto_core v0.46.0                                            <-------------
│   │   │   ├── async-trait v0.1.58 (proc-macro) (*)
│   │   │   ├── lazy_static v1.4.0
│   │   │   ├── log v0.4.17 (*)
│   │   │   ├── rusoto_credential v0.46.0
│   │   │   │   ├── async-trait v0.1.58 (proc-macro) (*)
│   │   │   └── rustc_version v0.2.3 (*)
│   │   ├── rusoto_dynamodb v0.46.0                                            <-------------
│   │   │   ├── async-trait v0.1.58 (proc-macro) (*)
│   │   │   ├── serde v1.0.148 (*)
│   │   │   └── serde_json v1.0.89 (*)
│   │   ├── thiserror v1.0.37 (*)
│   ├── num-traits v0.2.15 (*)
│   ├── object_store v0.5.1
│   ├── rusoto_core v0.48.0
│   │   ├── async-trait v0.1.58 (proc-macro) (*)
│   │   ├── base64 v0.13.1
│   │   [build-dependencies]
│   │   └── rustc_version v0.4.0 (*)
│   ├── rusoto_credential v0.48.0 (*)
│   ├── rusoto_dynamodb v0.48.0                                            <-------------
│   │   ├── async-trait v0.1.58 (proc-macro) (*)
│   │   ├── bytes v1.3.0
│   │   ├── futures v0.3.25 (*)
│   │   ├── rusoto_core v0.48.0 (*)
│   │   ├── serde v1.0.148 (*)
│   │   └── serde_json v1.0.89 (*)
│   ├── rusoto_sts v0.48.0
│   │   ├── async-trait v0.1.58 (proc-macro) (*)
│   │   ├── bytes v1.3.0
│   │   ├── chrono v0.4.23 (*)
│   │   ├── futures v0.3.25 (*)
│   │   ├── rusoto_core v0.48.0 (*)
│   │   ├── serde_urlencoded v0.7.1 (*)
│   │   └── xml-rs v0.8.4
│   ├── serde v1.0.148 (*)
├── tokio v1.22.0 (*)

so it looks like the dynamodb_lock crate requires rusoto 0.46 whereas deltalake requires rusoto 0.48 which is not possible in one program.

[mpetri]

It seems like the dynamodb lock crate was not version bumped and released to crates.io. The code in the repo has the correct dependency.

[iajoiner]

I have the same problem. Looks like we need to improve CI so that the top level build is covered.

[Jonathan-Rosenberg]

Hi team!
Any news regarding this issue?
It reproduces in 0.6.0 and basically disables the use of the package...

[houqp]

we will release a new version after PR #1047 gets merged.

[houqp]

published dynamodb_lock 0.4.2 with rusoto pinned to 0.48, should be fixed now.