Emulator tests are broken on main

[alamb]

Describe the bug

• After Retry streaming get requests (#15) #383 is merged, the emulator CI tests are broken

To Reproduce
Here is a failure on main: https://github.com/apache/arrow-rs-object-store/actions/runs/15319692559/job/43100487313

--- gcp::builder::tests::gcs_test_proxy_url stdout ----

thread 'gcp::builder::tests::gcs_test_proxy_url' panicked at src/gcp/builder.rs:[679](https://github.com/apache/arrow-rs-object-store/actions/runs/15319692559/job/43100487313#step:8:680):14:
called `Result::unwrap_err()` on an `Ok` value: GoogleCloudStorage { client: GoogleCloudStorageClient { config: GoogleCloudStorageConfig { base_url: "https://storage.googleapis.com", credentials: StaticCredentialProvider { credential: GcpCredential { bearer: "" } }, signing_credentials: StaticCredentialProvider { credential: GcpSigningCredential { email: "", private_key: None } }, bucket_name: "foo", retry_config: RetryConfig { backoff: BackoffConfig { init_backoff: 100ms, max_backoff: 15s, base: 2.0 }, max_retries: 10, retry_timeout: 180s }, client_options: ClientOptions { user_agent: None, root_certificates: [], content_type_map: {}, default_content_type: None, default_headers: None, proxy_url: Some("asdf://example.com"), proxy_ca_certificate: None, proxy_excludes: None, allow_http: Parsed(true), allow_insecure: Parsed(false), timeout: Some(Parsed(30s)), connect_timeout: Some(Parsed(5s)), pool_idle_timeout: None, pool_max_idle_per_host: None, http2_keep_alive_interval: None, http2_keep_alive_timeout: None, http2_keep_alive_while_idle: Parsed(false), http2_max_frame_size: None, http1_only: Parsed(true), http2_only: Parsed(false), randomize_addresses: Parsed(true) }, skip_signature: false }, client: HttpClient(Client { accepts: Accepts { gzip: false }, proxies: [Matcher], referer: true, default_headers: {"accept": "*/*", "user-agent": "object_store/0.12.1"}, reqwest::config::RequestTimeout: 30s }), bucket_name_encoded: "foo", max_list_results: None } }
stack backtrace:
   0: __rustc::rust_begin_unwind
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:697:5
   1: core::panicking::panic_fmt
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panicking.rs:75:14
   2: core::result::unwrap_failed
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/result.rs:1704:5
   3: core::result::Result<T,E>::unwrap_err
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/result.rs:1203:22
   4: object_store::gcp::builder::tests::gcs_test_proxy_url::{{closure}}
             at ./src/gcp/builder.rs:674:19
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/future/future.rs:124:9
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/future/future.rs:124:9
   7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:733:54
   8: tokio::task::coop::with_budget
             at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/task/coop/mod.rs:167:5
   9: tokio::task::coop::budget
             at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/task/coop/mod.rs:133:5
  10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:733:25
  11: tokio::runtime::scheduler::current_thread::Context::enter
             at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:432:19
  12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:732:36
  13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:820:68
  14: tokio::runtime::context::scoped::Scoped<T>::set
             at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context/scoped.rs:40:9
  15: tokio::runtime::context::set_scheduler::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context.rs:176:26
  16: std::thread::local::LocalKey<T>::try_with
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/thread/local.rs:311:12
  17: std::thread::local::LocalKey<T>::with
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/thread/local.rs:275:15
  18: tokio::runtime::context::set_scheduler
             at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context.rs:176:9
  19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
             at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:820:27
  20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
             at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:720:19
  21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:200:28
  22: tokio::runtime::context::runtime::enter_runtime
             at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context/runtime.rs:65:16
  23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
             at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/current_thread/mod.rs:188:9
  24: tokio::runtime::runtime::Runtime::block_on_inner
             at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/runtime.rs:356:47
  25: tokio::runtime::runtime::Runtime::block_on
             at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/runtime.rs:330:13
  26: object_store::gcp::builder::tests::gcs_test_proxy_url
             at ./src/gcp/builder.rs:[682](https://github.com/apache/arrow-rs-object-store/actions/runs/15319692559/job/43100487313#step:8:683):9
  27: object_store::gcp::builder::tests::gcs_test_proxy_url::{{closure}}
             at ./src/gcp/builder.rs:663:34
  28: core::ops::function::FnOnce::call_once
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/ops/function.rs:250:5
  29: core::ops::function::FnOnce::call_once
             at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.


failures:
    aws::builder::tests::s3_test_proxy_url
    client::retry::tests::test_retry
    gcp::builder::tests::gcs_test_proxy_url

Expected behavior
The tests should pass

Additional context
Given the errors, it looks to me like maybe the retry logic has made some tests pass when then were expecting failures.

However, it might be unrelated to #383 (for example, maybe the emulator has changed)

[alamb]

Interestingly, the emulator tests appear to have passed on #383:

• https://github.com/apache/arrow-rs-object-store/actions/runs/15259856086/job/42915517300

[alamb]

🤔

I could not reproduce the test failures locally. I am rerunning on main

(venv) andrewlamb@Andrews-MacBook-Pro-2:~/Software/arrow-rs-object-store$ TEST_INTEGRATION=1 GOOGLE_BUCKET=test-bucket GOOGLE_SERVICE_ACCOUNT=/tmp/gcs.json cargo test -p object_store --features=gcp --lib gcs_test_proxy_url
warning: unused import: `GetResultPayload`
   --> src/throttle.rs:407:51
    |
407 |     use crate::{integration::*, memory::InMemory, GetResultPayload};
    |                                                   ^^^^^^^^^^^^^^^^
    |
    = note: `#[warn(unused_imports)]` on by default

warning: unused import: `crate::buffered::BufWriter`
    --> src/lib.rs:1433:9
     |
1433 |     use crate::buffered::BufWriter;
     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: unused import: `tokio::io::AsyncWriteExt`
    --> src/lib.rs:1435:9
     |
1435 |     use tokio::io::AsyncWriteExt;
     |         ^^^^^^^^^^^^^^^^^^^^^^^^

warning: `object_store` (lib test) generated 3 warnings (run `cargo fix --lib -p object_store --tests` to apply 3 suggestions)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.06s
     Running unittests src/lib.rs (target/debug/deps/object_store-285e67c291eace48)

running 1 test
test gcp::builder::tests::gcs_test_proxy_url ... ok

[alamb]

A break in the case, when I ran cargo update locally

(venv) andrewlamb@Andrews-MacBook-Pro-2:~/Software/arrow-rs-object-store$ cargo update
    Updating crates.io index
     Locking 66 packages to latest compatible versions
    Updating async-compression v0.4.22 -> v0.4.23
    Updating backtrace v0.3.74 -> v0.3.75
    Updating bitflags v2.9.0 -> v2.9.1
    Updating bumpalo v3.17.0 -> v3.18.0
    Updating cc v1.2.19 -> v1.2.25
    Updating chrono v0.4.40 -> v0.4.41
    Updating core-foundation v0.10.0 -> v0.10.1
    Updating errno v0.3.11 -> v0.3.12
    Removing getrandom v0.2.15
    Removing getrandom v0.3.2
      Adding getrandom v0.2.16 (available: v0.3.3)
      Adding getrandom v0.3.3
    Updating h2 v0.4.9 -> v0.4.10
    Updating hashbrown v0.15.2 -> v0.15.3
    Updating hyper-rustls v0.27.5 -> v0.27.6
    Updating hyper-util v0.1.11 -> v0.1.14
    Updating icu_collections v1.5.0 -> v2.0.0
      Adding icu_locale_core v2.0.0
    Removing icu_locid v1.5.0
    Removing icu_locid_transform v1.5.0
    Removing icu_locid_transform_data v1.5.1
    Updating icu_normalizer v1.5.0 -> v2.0.0
    Updating icu_normalizer_data v1.5.1 -> v2.0.0
    Updating icu_properties v1.5.1 -> v2.0.1
    Updating icu_properties_data v1.5.1 -> v2.0.1
    Updating icu_provider v1.5.0 -> v2.0.0
    Removing icu_provider_macros v1.5.0
    Updating idna_adapter v1.2.0 -> v1.2.1
      Adding iri-string v0.7.8
    Updating litemap v0.7.5 -> v0.8.0
    Updating lock_api v0.4.12 -> v0.4.13
      Adding lru-slab v0.1.2
    Updating mio v1.0.3 -> v1.0.4
    Updating openssl v0.10.72 -> v0.10.73
    Updating openssl-sys v0.9.107 -> v0.9.109
    Updating parking_lot v0.12.3 -> v0.12.4
    Updating parking_lot_core v0.9.10 -> v0.9.11
      Adding potential_utf v0.1.2
    Updating proc-macro2 v1.0.94 -> v1.0.95
    Updating quick-xml v0.37.4 -> v0.37.5
    Updating quinn v0.11.7 -> v0.11.8
    Updating quinn-proto v0.11.10 -> v0.11.12
    Updating quinn-udp v0.5.11 -> v0.5.12
    Updating rand v0.9.0 -> v0.9.1
    Updating redox_syscall v0.5.11 -> v0.5.12
    Updating reqwest v0.12.15 -> v0.12.19
    Updating rustix v1.0.5 -> v1.0.7
    Updating rustls v0.23.26 -> v0.23.27
    Updating rustls-pki-types v1.11.0 -> v1.12.0
    Updating rustls-webpki v0.103.1 -> v0.103.3
    Updating rustversion v1.0.20 -> v1.0.21
    Updating socket2 v0.5.9 -> v0.5.10
    Updating syn v2.0.100 -> v2.0.101
    Updating synstructure v0.13.1 -> v0.13.2
    Updating tempfile v3.19.1 -> v3.20.0
    Updating tinystr v0.7.6 -> v0.8.1
    Updating tokio v1.44.2 -> v1.45.1
    Updating tokio-util v0.7.14 -> v0.7.15
      Adding tower-http v0.6.6
    Removing utf16_iter v1.0.5
    Updating webpki-roots v0.26.8 -> v1.0.0
    Updating windows-core v0.61.0 -> v0.61.2
    Updating windows-registry v0.4.0 -> v0.5.2
    Updating windows-result v0.3.2 -> v0.3.4
    Removing windows-strings v0.3.1
    Removing windows-strings v0.4.0
      Adding windows-strings v0.4.2
    Removing windows-targets v0.53.0
    Removing windows_aarch64_gnullvm v0.53.0
    Removing windows_aarch64_msvc v0.53.0
    Removing windows_i686_gnu v0.53.0
    Removing windows_i686_gnullvm v0.53.0
    Removing windows_i686_msvc v0.53.0
    Removing windows_x86_64_gnu v0.53.0
    Removing windows_x86_64_gnullvm v0.53.0
    Removing windows_x86_64_msvc v0.53.0
    Removing write16 v1.0.0
    Updating writeable v0.5.5 -> v0.6.1
    Updating yoke v0.7.5 -> v0.8.0
    Updating yoke-derive v0.7.5 -> v0.8.0
    Updating zerocopy v0.8.24 -> v0.8.25
    Updating zerocopy-derive v0.8.24 -> v0.8.25
      Adding zerotrie v0.2.2
    Updating zerovec v0.10.4 -> v0.11.2
    Updating zerovec-derive v0.10.3 -> v0.11.1

Then the test fails as it does remotely

(venv) andrewlamb@Andrews-MacBook-Pro-2:~/Software/arrow-rs-object-store$ TEST_INTEGRATION=1 GOOGLE_BUCKET=test-bucket GOOGLE_SERVICE_ACCOUNT=/tmp/gcs.json cargo test -p object_store --features=gcp --lib gcs_test_proxy_url
...
thread 'gcp::builder::tests::gcs_test_proxy_url' panicked at src/gcp/builder.rs:679:14:
called `Result::unwrap_err()` on an `Ok` value: GoogleCloudStorage { client: GoogleCloudStorageClient { config: GoogleCloudStorageConfig { base_url: "https://storage.googleapis.com", credentials: StaticCredentialProvider { credential: GcpCredential { bearer: "" } }, signing_credentials: StaticCredentialProvider { credential: GcpSigningCredential { email: "", private_key: None } }, bucket_name: "foo", retry_config: RetryConfig { backoff: BackoffConfig { init_backoff: 100ms, max_backoff: 15s, base: 2.0 }, max_retries: 10, retry_timeout: 180s }, client_options: ClientOptions { user_agent: None, root_certificates: [], content_type_map: {}, default_content_type: None, default_headers: None, proxy_url: Some("asdf://example.com"), proxy_ca_certificate: None, proxy_excludes: None, allow_http: Parsed(true), allow_insecure: Parsed(false), timeout: Some(Parsed(30s)), connect_timeout: Some(Parsed(5s)), pool_idle_timeout: None, pool_max_idle_per_host: None, http2_keep_alive_interval: None, http2_keep_alive_timeout: None, http2_keep_alive_while_idle: Parsed(false), http2_max_frame_size: None, http1_only: Parsed(true), http2_only: Parsed(false), randomize_addresses: Parsed(true) }, skip_signature: false }, client: HttpClient(Client { accepts: Accepts { gzip: false }, proxies: [Matcher], referer: true, default_headers: {"accept": "*/*", "user-agent": "object_store/0.12.1"}, reqwest::config::RequestTimeout: 30s }), bucket_name_encoded: "foo", max_list_results: None } }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[alamb]

Updating reqwest v0.12.15 -> v0.12.19

Downgrading to reqwest 0.12.15 makes the test start passing again:

(venv) andrewlamb@Andrews-MacBook-Pro-2:~/Software/arrow-rs-object-store$ cargo update --precise 0.12.15 reqwest
    Updating crates.io index
    Removing iri-string v0.7.8
 Downgrading reqwest v0.12.19 -> v0.12.15
    Removing tower-http v0.6.6
      Adding webpki-roots v0.26.11
 Downgrading windows-registry v0.5.2 -> v0.4.0
      Adding windows-strings v0.3.1
      Adding windows-targets v0.53.0
      Adding windows_aarch64_gnullvm v0.53.0
      Adding windows_aarch64_msvc v0.53.0
      Adding windows_i686_gnu v0.53.0
      Adding windows_i686_gnullvm v0.53.0
      Adding windows_i686_msvc v0.53.0
      Adding windows_x86_64_gnu v0.53.0
      Adding windows_x86_64_gnullvm v0.53.0
      Adding windows_x86_64_msvc v0.53.0
note: pass `--verbose` to see 1 unchanged dependencies behind latest
(venv) andrewlamb@Andrews-MacBook-Pro-2:~/Software/arrow-rs-object-store$ TEST_INTEGRATION=1 GOOGLE_BUCKET=test-bucket GOOGLE_SERVICE_ACCOUNT=/tmp/gcs.json cargo test -p object_store --features=gcp --lib gcs_test_proxy_url
...
test gcp::builder::tests::gcs_test_proxy_url ... ok

[alamb]

I tested and it seems the issue was introduced wiht reqwest 0.12.16: https://crates.io/crates/reqwest/0.12.16

reqwest version	test passes
0.12.15	PASS
0.12.16	FAIL
0.12.17	FAIL
0.12.18	FAIL
0.12.19	FAIL

https://github.com/seanmonstar/reqwest/releases/tag/v0.12.16

Possibly related:

refactor: use hyper-util's proxy::Matcher by @seanmonstar in seanmonstar/reqwest#2681

[alamb]

Here is a PR to fix the tests:

• Chore: fix emulator tests due to changes in reqwest #401