Best Practices

Follow these rules to improve your Sidekiq experience:

1. Make your job parameters small and simple

Sidekiq persists the arguments to perform_async to Redis as JSON. Sometimes people are tempted to make the following kind of mistake:

# MISTAKE
quote = Quote.find(quote_id)
SomeJob.perform_async(quote)

Complex Ruby objects do not convert to JSON, by default it will convert with to_s and look like #<Quote:0x0000000006e57288>. Even if they did serialize correctly, what happens if your queue backs up and that quote object changes in the meantime? Don't save state to Sidekiq, save simple identifiers. Look up the objects once you actually need them in your perform method.

SomeJob.perform_async(quote_id)

The arguments you pass to perform_async must be composed of simple JSON datatypes: string, integer, float, boolean, null(nil), array and hash. This means you must not use ruby symbols as arguments. The Sidekiq client API uses JSON.dump to send the data to Redis. The Sidekiq server pulls that JSON data from Redis and uses JSON.load to convert the data back into Ruby types to pass to your perform method. Don't pass symbols, named parameters, keyword arguments or complex Ruby objects (like Date or Time!) as those will not survive the dump/load round trip correctly. You can use methods like Hash#stringify_keys to convert Symbol keys when passing a Hash to perform_async: MyJob.perform_async(hash.stringify_keys).

2. Make your job idempotent and transactional

Idempotency means that your job can safely execute multiple times. For instance, with the error retry functionality, your job might be half-processed, throw an error, and then be re-executed over and over until it successfully completes. Let's say you have a job which voids a credit card transaction and emails the user to let them know the charge has been refunded:

def perform(card_charge_id)
  charge = CardCharge.find(card_charge_id)
  charge.void_transaction
  Emailer.charge_refunded(charge).deliver
end

What happens when the email fails to render due to a bug? Will the void_transaction method handle the case where a charge has already been refunded? You can use a database transaction to ensure data changes are rolled back if there is an error or you can write your code to be resilient in the face of errors.

Just remember that Sidekiq will execute your job at least once, not exactly once. Even a job which has completed can be re-run. Redis can go down between the point where your job finished but before Sidekiq has acknowledged it in Redis. Sidekiq makes no exactly-once guarantee at all.

3. Embrace Concurrency

Sidekiq is designed for parallel execution so design your jobs so you can run lots of them in parallel. It has basic features for tuning concurrency (e.g. targeting a sidekiq process at a queue with a defined number of threads) but your system architecture is much simpler if you don't have such specialization.

You can use a connection pool to limit the overall number of connections to a resource-limited server if your Sidekiq processes are overwhelming it with traffic.

Sidekiq will not provide features which hack around a lack of concurrency in your jobs.

4. Use Precise Terminology

Within the Sidekiq ecosystem, the term worker is ambiguous and thus meaningless.

• Sidekiq::Job is a module included in a job class.
• If you have 10 "workers" executing jobs, you have 10 threads.
• If you start a "worker", you have a process.
• If you have 10 "workers" enqueued, you have 10 jobs.

Use those terms: job class, thread, process or job. Dump "worker" from your vocabulary.

Next: Using Redis