Middleware sharded backend

S3Proxy can spread (shard) objects across multiple buckets on the backend. The S3Proxy clients are presented with a single "virtual" bucket and the shards are opaque to them. This is helpful for either improving performance or working with backends whose performance degrades considerably with a large number of objects in a single bucket.

When the middleware is enabled, all unspecified buckets behave as before, only the specified sharded buckets are subject to the behavior. Here is a sample configuration:

s3proxy.sharded-blobstore.sharded-bucket.shards=10
s3proxy.sharded-blobstore.sharded-bucket.prefix=shard

In this case the virtual, sharded bucket is named sharded-bucket. In the backing store, 10 shards are created for the bucket. The .prefix option specifies that the shards will have the prefix shard, e.g. shard-1, shard-2, and so on. If the prefix options is omitted, then the virtual bucket name is used for the shard prefix.

Issuing CreateBucket for the virtual bucket will create all of the shards in the backing store. Similarly, removing the virtual bucket will remove all of the backing shards.

The objects are spread across the shards based on the hash of the object key. Currently, listing the virtual bucket is not supported, as it's a more challenging feature that would require listing all of the shards.