Sidekiq not processing queue

Question

What possible reasons can Sidekiq prevent from processing jobs in the queue? The queue is full. The log file sidekiq.log indicates no activity at all. Thus the queue is full but the log is empty, and Sidekiq does not seem to process items. There seem to no worker processing jobs. Restarting Redis or flush it with FLUSHALL or FLUSHDB as no effect. Sidekiq has been started with

bundle exec sidekiq -L log/sidekiq.log

and produces the following log file:

2013-05-30..Booting Sidekiq 2.12.0 using redis://localhost:6379/0 with options {}
2013-05-30..Running in ruby 1.9.3p374 (2013-01-15 revision 38858) [i686-linux]
2013-05-30..See LICENSE and the LGPL-3.0 for licensing details.
2013-05-30..Starting processing, hit Ctrl-C to stop

How can you find out what went wrong? Are there any hidden log files?

Answer 1

The reason was in our case: Sidekiq may look for the wrong queue. By default Sidekiq uses a queue named "default". We used two different queue names, and defined them in config/sidekiq.yml

# configuration file for Sidekiq
:queues:
  - queue_name_1
  - queue_name_2

The problem is that this config file is not automatically loaded by default in your development environment (unlike database.yml or thinking_sphinx.yml for instance) by a simple bundle exec sidekiq command. Thus we wrote our jobs in two certain queues, and Sidekiq was waiting for jobs in a third queue (the default one). You have to pass the path to the config file as a parameter through the -Cor --config option:

bundle exec sidekiq -C ./config/sidekiq.yml

or you can pass the queue names directly (no spaces allowed here after the comma):

bundle exec sidekiq -q queue_name_1,queue_name_2

To find the problem out it is helpful to pass the option -v or --verbose at the command line, too, or to use :verbose: true in the sidekiq.yml file. Everything which is defined in a config file is of course useless if the config file is not loaded.. Therefore make sure you are using the right config file first.

Answer 2

If you have a config/sidekiq.yml check that all the queues are defined there, check this sample file: https://github.com/mperham/sidekiq/blob/master/examples/config.yml

If you are passing queue names in the command line or Procfile, something similar to

bin/sidekiq -q queue1 -q queue2
bundle exec sidekiq -q queue1 -q queue2

check that all your queues are defined there.

In case you are not sure about the names of your queues, you can figure it out with the following script:

require "sidekiq/api"
stats = Sidekiq::Stats.new
stats.queues
# {"production_mailers"=>25, "production_default"=>1}

Then, you can do things with the queues:

queue = Sidekiq::Queue.new("production_mailers")
queue.count
queue.clear

Answer 3

I just had this issue. Turns out I had made a syntax error in my sidekiq.yml

Answer 4

I was banging my head against a brick wall on this for a while, my issue was that sidekiq required a newer version of redis-server. I ran "bundle exec sidekiq" and that revealed the error. Once I updated to a newer version of redis-server it was fine.

Answer 5

In my case, sidekiq was fine in development, but stuck in staging. It was human error on the capistrano's deploy configuration. I set the path for sidekiq.yml incorrectly in the Capfile (shared instead of current).

It failed silently:

# Capfile

# WRONG:
set :sidekiq_config, -> { File.join(shared_path, 'config', 'sidekiq.yml') }
                                    ^^^^^^^^^^^
# RIGHT:
set :sidekiq_config, -> { File.join(current_path, 'config', 'sidekiq.yml') }

Answer 6

My problem was I had a configure_server but not configure_client in my initialiser, you must have both:

Sidekiq.configure_server do |config|
  config.redis = { url: ENV.fetch('SIDEKIQ_REDIS_URL', 'redis://127.0.0.1:6379/1') }
end

Sidekiq.configure_client do |config|
  config.redis = { url: ENV.fetch('SIDEKIQ_REDIS_URL', 'redis://127.0.0.1:6379/1') }
end