Requests got stuck in ActiveRecord::QueryCache middleware

Question

After deploying our Rails app (4.0.9, Ruby 2.1.2), we notice requests to our app get hang after a while , usually 1 day or so.

Using the gem rack_timer, we're able to find out requests got stuck at ActiveRecord::QueryCache middleware.

Rack Timer (incoming) -- ActiveRecord::QueryCache: 925626.7731189728 ms

After removing it, our app seems to be back to normal. However, I understand the purpose of this middleware is to increase the performance, so removing is just a temporary solution. We're using mysql (5.1.67) with adapter mysql2 (0.3.13)

Update: Right after I posted this question, the server started hang again, this time requests were stuck at ActionDispatch::Routing::RouteSet

I, [2014-10-13T23:17:03.661346 #32498]  INFO -- : Rack Timer (Application Action) -- ActionDispatch::Routing::RouteSet: 3667661.2360477448 ms
I, [2014-10-13T23:17:03.661946 #32498]  INFO -- : Rack Timer (Application Action) -- ActionDispatch::Routing::RouteSet: 4373914.719343185 ms

Do you know any reason could cause this?

Thank you in advance.

Answer 1

The most possible cause is your connection with the database died (firewall, server configuration...) and Rails didn't notice, so a timeout happens inside ActiveRecord::QueryCache. Once Rails notices the connection died, it reconnects and the execution flow continues.

Try setting the param read_timeout in your database.yml file to something like 10 seconds and check your connection and server settings.