Sharding

Available in Sidekiq 3.0+

Sidekiq has a scalability limit: normally it can only use a single Redis server. This usually isn't a problem: Redis is really fast and on good hardware you can pump ~25,000 jobs/sec through Redis before you'll start to hit the ceiling. For people who want to go beyond that limit you have two choices:

• Break your application into smaller applications. You can use one Redis server per application.
• Shard your application's jobs across many Redis instances.

How to Shard

The latter option is reasonably simple with the Sidekiq::Client updates in Sidekiq 3.0:

REDIS_A = ConnectionPool.new { Redis.new(...) }
REDIS_B = ConnectionPool.new { Redis.new(...) }

# To create a new connection pool for a namespaced Sidekiq worker:
ConnectionPool.new do
  client = Redis.new(:url => "Your Redis Url")
  Redis::Namespace.new("Your Namespace", :redis => client)
end

# Create a job in the default redis instance
SomeWorker.perform_async

# Push a job to REDIS_A using the low-level Client API
client = Sidekiq::Client.new(REDIS_A)
client.push(...)
client.push_bulk(...)

Sidekiq::Client.via(REDIS_B) do
  # All jobs defined within this block will go to B
  SomeWorker.perform_async
end

Sidekiq API

You can use sidekiq/api with any shard by wrapping your API usage within a Sidekiq::Client.via block which pre-selects the pool for that shard:

> Sidekiq::Queue.all
=> [#<Sidekiq::Queue:0x0000000108597af0 @name="default", @rname="queue:default">]
> Sidekiq::Client.via(POOL) { Sidekiq::Queue.all }
=> []

Limitations

Sharding comes with some limitations:

1. The OSS Web UI is limited to the default connection pool. Sidekiq Pro supports multiple shards in the same process, see Pro-Web-UI.
2. You need to spin up Sidekiq processes to execute jobs from each Redis instance. A Sidekiq process only executes jobs from a single Redis instance.
3. Sharding increases the complexity of your system. It's harder to mentally track which jobs are going where, making debugging harder.

I don't recommend sharding unless all other options are unavailable.

Previous: Testing Next: Problems and Troubleshooting