The simplest, highest-throughput 1 Python interface to Amazon S3, Google Cloud Storage, Azure Storage, & other S3-compliant APIs, powered by Rust.
- One interface for all backends with no required Python dependencies.
- Sync and async API with full type hinting.
- Streaming downloads with configurable chunking.
- Streaming uploads from files or async or sync iterators.
- Streaming list, with no need to paginate.
- Automatic multipart uploads for large file objects.
- Automatic credential refresh before expiration.
- File-like object API and fsspec integration.
- Optionally return list results in Apache Arrow format, which is faster and more memory-efficient than materializing Python
dicts. - Zero-copy data exchange between Rust and Python via the buffer protocol.
For Rust developers looking to add object_store support to their own Python packages, refer to pyo3-object_store.
To install obstore using pip:
pip install obstoreObstore is on conda-forge and can be installed using conda, mamba, or pixi. To install obstore using conda:
conda install -c conda-forge obstore
Full documentation is available on the website.
Head to Getting Started to dig in.
Footnotes
-
Benchmarking is ongoing, but preliminary results indicate roughly 9x higher throughput than fsspec and 2.8x higher throughput than aioboto3 for many concurrent, small, get requests from an async context. ↩
