Sidekiq deploy to multiple environments

Question

(See below for my detailed config, which is the result of Henley Chiu's answer).

I've been trying to wrap my brain around Sidekiq deploys, and I am not really getting it. I have an app with a staging environment, and a production environment, on the same server. Everything I see about sidekiq deploys basically say "just add sidekiq/capistrano to your deploy file", so I did that. And then the instructions are "here's a yml file with options" but nothing seems to be explained. Do I need namespaces? I see that in an initialize file, but that seems to be to point outside the server.

I deployed earlier, and each stage seems to boot sidekiq up with the proper environment, but they both process from the same queues. My emails from production were trying to be processed by the stage sidekiq, and failing. I stopped my stage for now, but eventually I will need to use it again. I hope I'm not being dense, I've really tried to understand this and am just having a hard time with finding a definitive "here's how it's done".

For what it's worth, here is config/sidekiq.yml (which is loaded fine during the deploy):

:concurrency: 5
:verbose: false
:pidfile: ./tmp/pids/sidekiq.pid
:logfile: ./log/sidekiq.log
:queues:
  - [carrierwave, 7]
  - [client_emails, 5]
  - [default, 3]
staging:
  :concurrency: 10
production:
  :concurrency: 25

Log files, and pids seem to be in the right spot, but the queues are just merged. Any help would be GREAT!

Also, if it matters:

Rails 3.2.11, passenger, nginx, rvm, Ubuntu 12.10, and Ruby 1.9.3

Detailed Configuration (answer):

First I set up a new redis server at port 7777 (or whatever port you please besides the default 6379). Pretty much followed the redis quickstart guide that I used the first time around.

Then I made the initilizer file; this has both the client and the server config. Both are required to make sidekiq work multistage.

Note that I am using an external YAML file for the settings. I am using SettingsLogic for this to make things easier, but you can just as easily do this yourself by including the file. By using a yaml file, we don't have to touch our environments/staging or production files.

# config/initializers/sidekiq.rb
server = Settings.redis.server
port = Settings.redis.port
db_num = Settings.redis.db_num
namespace = Settings.redis.namespace

Sidekiq.configure_server do |config|  
  config.redis = { url: "redis://#{server}:#{port}/#{db_num}", namespace: namespace  }
end

I am using passenger - the troubleshooting page of the sidekiq wiki recommends a change for the setup when using unicorn or passenger, so I added the code there for the client setup:

# config/initializers/sidekiq.rb (still)
if defined?(PhusionPassenger)
  PhusionPassenger.on_event(:starting_worker_process) do |forked|
    Sidekiq.configure_client do |config|
      config.redis = { url: "redis://#{server}:#{port}/#{db_num}", namespace: namespace }
    end if forked
  end
end

This is my Settings file (obviously values changed):

#config/settings.yml
defaults: &defaults
  redis: &redis_defaults
    server: 'localhost'
    port: 6379
    db_num: 0
    namespace: 'sidekiq_development'

development:
  <<: *defaults

test:
  <<: *defaults

staging:
  <<: *defaults
  redis:
    <<: *redis_defaults
    port: 8888
    namespace: 'sidekiq_staging'

production:
  <<: *defaults
  redis:
    <<: *redis_defaults
    port: 7777
    namespace: 'sidekiq_production'

I found that adding the namespace to the config/sidekiq.yml file didn't seem to work - sidekiq would boot on deploy using the right port, but wouldn't actually process anything. But since the wiki recommends using a namespace, I ended up just adding it to the init file.

I hope this helpful for others, because this was really hard for me to understand, having not done a lot of this kind of setup before.

Answer 1

In your initializers/sidekiq.rb file, you specify the Redis queue all environments boot up with. For mine it is:

redisServer = "localhost"
Sidekiq.configure_server do |config|
  config.redis = { :url => 'redis://' + redisServer + ':6379/0' }
end

If you want each environment to process from separate queues, you can have specific sidekiq.rb files in the environments folder for each environment. Each with different redis servers.

Answer 2

If all environments(development, staging and production) are on same server then use namespace. In your initializers/sidekiq.rb file,

Sidekiq.configure_server do |config|
    config.redis = { url: 'redis://localhost:6379/0', namespace: "sidekiq_app_name_#{Rails.env}" }
end

Sidekiq.configure_client do |config|
    config.redis = { url: 'redis://localhost:6379/0', namespace: "sidekiq_app_name_#{Rails.env}" }
end     

Answer 3

In addition to the namespace, it will be good if you also separate out DBs for each Rails environment in Redis too i.e.:

env_num = Rails.env == 'staging' ? 0 : 1
Redis.new(db: env_num) # existing DB is selected if already present

Sidekiq.configure_server do |config|
  config.redis = { url: "redis://localhost:6379/#{env_num}", namespace: "app_name_#{Rails.env}" }
end

Sidekiq.configure_client do |config|
  config.redis = { url: "redis://localhost:6379/#{env_num}", namespace: "app_name_#{Rails.env}" }
end