How to perform a Sidekiq callback when a group of workers are complete

Question

Lets say I have a Sidekiq task that processes products to my database. Each product is grouped by store, so an overly simplified example of my code would be something like this...

stores.each do |store|
  store.products.each do |product|
    ProductWorker.perform_async(product.id)
  end
end

When all the products from one store have run. I'd like to update the stores last_updated column with the current time. But only when the last task for that store has run. How can I achieve this?

Answer 1

This is exactly what Sidekiq Pro's Batches feature is designed to solve:

https://github.com/mperham/sidekiq/wiki/Batches

http://sidekiq.org/pro/

You would write this code:

class ProductWorker
  include Sidekiq::Worker

  def on_complete(status, params)
    Store.find(params['sid']).update_attribute(:last_updated, Time.now)
  end

  def perform(product_id)
    # do something
  end
end


stores.each do |store|
  b = Sidekiq::Batch.new
  b.on(:complete, ProductWorker, 'sid' => store.id)
  b.jobs do
    store.products.find_each do |product|
      ProductWorker.perform_async(product.id)
    end
  end
end

Easy.

Answer 2

You can try sidekiq-batch gem to solve your problem. It's like paid Batches feature of Sidekiq Pro and has same api.