What is the best way to use Redis in a Multi-threaded Rails environment? (Puma / Sidekiq)

Question

I'm using Redis in my application, both for Sidekiq queues, and for model caching.

What is the best way to have a Redis connection available to my models, considering that the models that will be hitting Redis will be called both from my Web application (ran via Puma), and from background jobs inside Sidekiq?

I'm currently doing this in my initializers:

Redis.current = Redis.new(host: 'localhost', port: 6379)

And then simply use Redis.current.get / Redis.current.set (and similar) throughout the code...

This should be thread-safe, as far as I understand, since the Redis Client only runs one command at a time, using a Monitor.

Now, Sidekiq has its own connection pool to Redis, and recommends doing

Sidekiq.redis do |conn|
   conn.get
   conn.set
end

As I understand it, this would be better than the approach of just using Redis.current because you don't have multiple workers on multiple threads waiting on each other on a single connection when they hit Redis.

However, how can I make this connection that I get from Sidekiq.redis available to my models? (without having to pass it around as a parameter in every method call)

I can't set Redis.current inside that block, since it's global, and I'm back to everyone using the same connection (plus switching between them randomly, which might even be non-thread-safe)

Should I store the connection that I get from Sidekiq.Redis into a Thread-local variable, and use that thread-local variable everywhere?

In that case, what do I do in the "Puma" context? How do I set the thread-local variable?

Any thoughts on this are greatly appreciated.

Thank you!

Answer 1

You use a separate global connection pool for your application code. Put something like this in your redis.rb initializer:

require 'connection_pool'
REDIS = ConnectionPool.new(size: 10) { Redis.new }

Now in your application code anywhere, you can do this:

REDIS.with do |conn|
  # some redis operations
end

You'll have up to 10 connections to share amongst your puma/sidekiq workers. This will lead to better performance since, as you correctly note, you won't have all the threads fighting over a single Redis connection.

All of this is documented here: https://github.com/mperham/sidekiq/wiki/Advanced-Options#connection-pooling