Heroku + Sidekiq: ActiveRecord::StatementInvalid: PG::UnableToSend: SSL SYSCALL error: EOF detected

Question

Hi we are running on Heroku\'s Cedar stack with Unicorn and Sidekiq. We intermittently get the following errors

BurnThis ActiveRecord::StatementInvalid: PG::         


        

Answer 1

This error is caused by your postgis adapter trying to use a stale/dead connection from the ActiveRecord connection pool. There are two ways to address this issue:

1. Size your connection pool to match the number of threads/process
2. Lower connection reaping frequency (Reaper checks pool for dead connections, every N secs)

To implement #1, you need to set your pool size appropriate for Unicorn and for Sidekiq, which likely have different needs.

Unicorn is single threaded, so the default pool size of 5 connections per process is correct for you. This will allocate up to 5 connections for each of WEB_CONCURRENCY backend unicorn workers. You should reset the default pool size and use your existing unicorn.rb:

$> heroku config:set DB_POOL=5

Sidekiq however uses a very different model. By default, Sidekiq has a single process and N threads. You want a slightly larger DB pool size than the number of Sidekiq threads. You can implement this in your config/initializers/sidekiq.rb as follows:

Sidekiq.configure_server do |config|
  pool_size = Sidekiq.options[:concurrency] + 2

  config.redis = { :url => ENV['REDIS_URL'], :namespace => 'btsidekiq', :size => pool_size }

  if defined?(ActiveRecord::Base)
    config = Rails.application.config.database_configuration[Rails.env]
    config['adapter'] = 'postgis'
    config['pool']              = pool_size
    config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
    ActiveRecord::Base.establish_connection(config)
  end
end

My best guess is that using such a large 100 connection pool, you are more likely to accrue dead connections. Sizing the pool appropriately should fix this.

If this does not work, you should try decreasing your DB_REAP_FREQ to 5 seconds.