Sidekiq: bulk manipulation of jobs and processes

Sidekiq series — part 3 of 4

1. Infra: starting, stopping, killing
2. Diagnosis via console
3. You are here — Bulk manipulation
4. UI hacks in the dashboard

After diagnosing (part 2), the invasive part usually follows: getting a problematic worker out of the way, reorganizing the queue, deleting jobs that no longer make sense. This post is the toolbox for that.

The ; nil at the end of blocks is just to prevent rails console from printing thousands of lines when the collection is large.

Selecting retry jobs by class name

job_class_name = 'SidekiqTest::SidekiqTestWorker'
jobs = Sidekiq::RetrySet.new.select { |job| job.klass == job_class_name }; nil
jobs.size

Selecting dead jobs by class name

job_class_name = 'SidekiqTest::SidekiqTestWorker'
jobs = Sidekiq::DeadSet.new.select { |job| job.klass == job_class_name }; nil
jobs.size

Selecting queued jobs in a queue by class name

queue_name = 'default'
job_class_name = 'SidekiqTest::SidekiqTestWorker'
jobs = Sidekiq::Queue.new(queue_name).select { |job| job.klass == job_class_name }; nil
jobs.count

Re-queuing selected jobs for retry

jobs.each(&:retry)

Applies to any collection of jobs from RetrySet or DeadSet. Moves everything back to the original queue.

Deleting selected jobs

jobs.each(&:delete)

Same pattern, with the inverse operation. Use once you’re sure you can discard them — there’s no going back.

Moving 1000 jobs of a class to another queue

queue_name = 'default'
new_queue_name = 'funnels_test_worker'
queue = Sidekiq::Queue.new(queue_name)

queue.first(1000).each do |job|
  if job.klass == "SidekiqTest::SidekiqTestWorker"
    SidekiqTest::SidekiqTestWorker.set(queue: new_queue_name).perform_async(*job.args)
    job.delete
  end
end; nil

This is the snippet that saves the day most often. When a worker is bringing down a shared queue, instead of pausing everything, I:

1. Create a dedicated queue (funnels_test_worker)
2. Move the problematic jobs to it
3. Spin up a separate Sidekiq process consuming that queue with reduced concurrency (e.g. 1 thread)

The rest of the operation doesn’t feel it — and I can debug the worker calmly without pressuring the main queue.

Quieting processes for deploy

ps = Sidekiq::ProcessSet.new
ps.each(&:quiet!)

quiet! makes each process stop picking up new jobs, but finish the ones currently running. The polite way to drain before bringing down — good for deploys that don’t use an orchestrator.

Stopping processes via API

ps = Sidekiq::ProcessSet.new
ps.each(&:stop!)

Equivalent to sidekiqctl stop from part 1, but via console instead of shell.

Clear all of Sidekiq’s Redis — DELETES EVERYTHING

Sidekiq.redis { |conn| conn.flushdb }

Nuclear bomb: deletes everything Sidekiq has in Redis (queues, retries, dead set, stats). Only use this when you know you can reprocess safely or you’re on a dev machine. In production, this is an incident — only do it with a clear recovery plan.

Next in the series

UI hacks for the Sidekiq dashboard — when the retries screen has 100,000 items and the native UI can’t cope.