Work with two separate redis instances with sidekiq?

Question

Good afternoon,

I have two separate, but related apps. They should both have their own background queues (read: separate Sidekiq & Redis processes). Howev

Answer 1

I came across this and ran into some issues because I'm using ActiveJob, which complicates how messages are read out of the queue.

Building on ARO's answer, you will still need the redis_pool setup:

remote_sidekiq.rb

class RemoteSidekiq
  class_attribute :redis_pool
end

config/initializers/remote_sidekiq.rb

url = ENV.fetch("REDISCLOUD_URL")
namespace = 'primary'

redis = Redis::Namespace.new(namespace, redis: Redis.new(url: url))

RemoteSidekiq.redis_pool = ConnectionPool.new(size: ENV['MAX_THREADS'] || 6) { redis }

Now instead of the worker we'll create an ActiveJob Adapter to queue the request:

lib/active_job/queue_adapters/remote_sidekiq_adapter.rb

require 'sidekiq'

module ActiveJob
  module QueueAdapters
    class RemoteSidekiqAdapter
      def enqueue(job)
        #Sidekiq::Client does not support symbols as keys
        job.provider_job_id = client.push \
          "class"   => ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper,
          "wrapped" => job.class.to_s,
          "queue"   => job.queue_name,
          "args"    => [ job.serialize ]
      end

      def enqueue_at(job, timestamp)
        job.provider_job_id = client.push \
          "class"   => ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper,
          "wrapped" => job.class.to_s,
          "queue"   => job.queue_name,
          "args"    => [ job.serialize ],
          "at"      => timestamp
      end

      def client
        @client ||= ::Sidekiq::Client.new(RemoteSidekiq.redis_pool)
      end
    end
  end
end

I can use the adapter to queue the events now:

require 'active_job/queue_adapters/remote_sidekiq_adapter'

class RemoteJob < ActiveJob::Base
  self.queue_adapter = :remote_sidekiq

  queue_as :default

  def perform(_event_name, _data)
    fail "
      This job should not run here; intended to hook into
      ActiveJob and run in another system
    "
  end
end

I can now queue the job using the normal ActiveJob api. Whatever app reads this out of the queue will need to have a matching RemoteJob available to perform the action.