Long running delayed_job jobs stay locked after a restart on Heroku

Question

When a Heroku worker is restarted (either on command or as the result of a deploy), Heroku sends SIGTERM to the worker process. In the case of delayed_job

Answer 1

Abort Job Cleanly on SIGTERM

A much better solution is now built into delayed_job. Use this setting to throw an exception on TERM signals by adding this in your initializer:

Delayed::Worker.raise_signal_exceptions = :term

With that setting, the job will properly clean up and exit prior to heroku issuing a final KILL signal intended for non-cooperating processes:

You may need to raise exceptions on SIGTERM signals, Delayed::Worker.raise_signal_exceptions = :term will cause the worker to raise a SignalException causing the running job to abort and be unlocked, which makes the job available to other workers. The default for this option is false.

Possible values for raise_signal_exceptions are:

• false - No exceptions will be raised (Default)
• :term - Will only raise an exception on TERM signals but INT will wait for the current job to finish.
• true - Will raise an exception on TERM and INT

Available since Version 3.0.5.

See this commit where it was introduced.