Skip to content

Commit

Permalink
Swap AsyncHttpFileReader for HttpObjectStore (#607)
Browse files Browse the repository at this point in the history
Directly copied from the equivalent parquet-wasm PR 😛 .

I'm strongly considering just wrapping
[object_store_s3_wasm](https://github.com/JanKaul/object_store_s3_wasm)
in object-store-wasm this weekend, so if there's no particular rush, I
can work the coveted fsspec-like protocol-differentiated IO into this
PR.

Also, one question for any maintainer - `io_parquet_async` implies
object-store-wasm::http at a minimum, right? I'm thinking in terms of
the other cloud providers, which I imagine would be worth feature-gating
at the level of this consuming crate (it's a shame toggling transitive
dependency features isn't a thing).
  • Loading branch information
H-Plus-Time authored Apr 12, 2024
1 parent f775f0d commit eb86e3b
Show file tree
Hide file tree
Showing 3 changed files with 201 additions and 31 deletions.
183 changes: 165 additions & 18 deletions js/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 9 additions & 3 deletions js/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,16 @@ io_object_store = [
"dep:object_store",
"dep:reqwest",
"dep:tokio",
"dep:object-store-wasm",
"dep:url",
"parquet/object_store"
]
io_parquet = ["geoarrow/parquet", "table", "dep:bytes", "dep:parquet"]
io_parquet_async = [
"async-stream",
"geoarrow/parquet_async",
"io_http",
# We don't currently use object_store in Parque
# "io_object_store",
"io_object_store",
"io_parquet",
"range-reader",
"wasm-streams",
Expand Down Expand Up @@ -95,7 +97,7 @@ range-reader = { version = "0.2", optional = true }
reqwest = { version = "*", optional = true }
thiserror = "1"
tokio = { version = "*", default-features = false, optional = true }
wasm-streams = { version = "0.3.0", optional = true }
wasm-streams = { version = "0.4.0", optional = true }

serde = { version = "1.0", features = ["derive"] }
serde-wasm-bindgen = "0.6.5"
Expand All @@ -109,6 +111,10 @@ zstd = { version = "*", features = [
# TODO: temporary to fix parquet wasm build
# upstream issue: https://github.com/gyscos/zstd-rs/issues/269
zstd-sys = { version = "=2.0.9", optional = true, default-features = false }
object-store-wasm = { version = "0.0.1", optional = true, default-features = false, features = [
"http"
] }
url = { version = "2.5.0", optional = true }


[dependencies.web-sys]
Expand Down
Loading

0 comments on commit eb86e3b

Please sign in to comment.