You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Timur Alperovich edited this page Jun 6, 2021
·
1 revision
S3Proxy can spread (shard) objects across multiple buckets on the backend. The S3Proxy clients are presented with a single "virtual" bucket and the shards are opaque to them. This is helpful for either improving performance or working with backends whose performance degrades considerably with a large number of objects in a single bucket.
When the middleware is enabled, all unspecified buckets behave as before, only the specified sharded buckets are subject to the behavior. Here is a sample configuration:
In this case the virtual, sharded bucket is named sharded-bucket. In the backing store, 10 shards are created for the bucket. The .prefix option specifies that the shards will have the prefix shard, e.g. shard-1, shard-2, and so on. If the prefix options is omitted, then the virtual bucket name is used for the shard prefix.
Issuing CreateBucket for the virtual bucket will create all of the shards in the backing store. Similarly, removing the virtual bucket will remove all of the backing shards.
The objects are spread across the shards based on the hash of the object key. Currently, listing the virtual bucket is not supported, as it's a more challenging feature that would require listing all of the shards.