Sharding middleware #325
Labels
Comments
timuralp
added a commit
to timuralp/s3proxy
that referenced
this issue
Apr 8, 2021
Adds the sharded bucket middleware, which allows for splitting objects across multiple backend buckets for a given virtual bucket. The middleware should be configured as: s3proxy.sharded-blobstore.<bucket name>.shards=<number of shards> s3proxy.sharded-blobstore.<bucket name>.prefix=<prefix>. All shards are named <prefix>-<index>, where index is an integer from 0 to <number of shards> - 1. If the <prefix> is not supplied, the <bucket name> is used as the prefix. Listing the virtual bucket and multipart uploads are not supported. When listing all containers, the shards are elided from the result. Fixes gaul#325 Fixes gaul#351
timuralp
added a commit
to timuralp/s3proxy
that referenced
this issue
Apr 8, 2021
Adds the sharded bucket middleware, which allows for splitting objects across multiple backend buckets for a given virtual bucket. The middleware should be configured as: s3proxy.sharded-blobstore.<bucket name>.shards=<number of shards> s3proxy.sharded-blobstore.<bucket name>.prefix=<prefix>. All shards are named <prefix>-<index>, where index is an integer from 0 to <number of shards> - 1. If the <prefix> is not supplied, the <bucket name> is used as the prefix. Listing the virtual bucket and multipart uploads are not supported. When listing all containers, the shards are elided from the result. Fixes gaul#325 Fixes gaul#351
timuralp
added a commit
to timuralp/s3proxy
that referenced
this issue
May 3, 2021
Adds the sharded bucket middleware, which allows for splitting objects across multiple backend buckets for a given virtual bucket. The middleware should be configured as: s3proxy.sharded-blobstore.<bucket name>.shards=<number of shards> s3proxy.sharded-blobstore.<bucket name>.prefix=<prefix>. All shards are named <prefix>-<index>, where index is an integer from 0 to <number of shards> - 1. If the <prefix> is not supplied, the <bucket name> is used as the prefix. Listing the virtual bucket and multipart uploads are not supported. When listing all containers, the shards are elided from the result. Fixes gaul#325 Fixes gaul#351
timuralp
added a commit
to timuralp/s3proxy
that referenced
this issue
May 8, 2021
Adds the sharded bucket middleware, which allows for splitting objects across multiple backend buckets for a given virtual bucket. The middleware should be configured as: s3proxy.sharded-blobstore.<bucket name>.shards=<number of shards> s3proxy.sharded-blobstore.<bucket name>.prefix=<prefix>. All shards are named <prefix>-<index>, where index is an integer from 0 to <number of shards> - 1. If the <prefix> is not supplied, the <bucket name> is used as the prefix. Listing the virtual bucket and multipart uploads are not supported. When listing all containers, the shards are elided from the result. Fixes gaul#325 Fixes gaul#351
timuralp
added a commit
to timuralp/s3proxy
that referenced
this issue
May 8, 2021
Adds the sharded bucket middleware, which allows for splitting objects across multiple backend buckets for a given virtual bucket. The middleware should be configured as: s3proxy.sharded-blobstore.<bucket name>.shards=<number of shards> s3proxy.sharded-blobstore.<bucket name>.prefix=<prefix>. All shards are named <prefix>-<index>, where index is an integer from 0 to <number of shards> - 1. If the <prefix> is not supplied, the <bucket name> is used as the prefix. Listing the virtual bucket and multipart uploads are not supported. When listing all containers, the shards are elided from the result. Fixes gaul#325 Fixes gaul#351
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Some S3 implementations have hotspots if key names distribute poorly, e.g., date and timestamp prefixes:
https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-tricks-seattle-hiring-event/
S3Proxy could support a middleware which adds a prefix of the key name hash to the key name. This poses challenges for listing objects but this operation could either be disabled or return the prefixed name instead of the user-provided name.
The text was updated successfully, but these errors were encountered: