delayed_jobs vs resque vs beanstalkd?

Question

Here is my needs:

• Enqueue_in(10.hours, ... ) (DJ syntax is perfect.)
• Multiply workers, concurrently. (Resque or beanstalkd are good for this, but not

Answer 1

For my projects I will feel very comfortbale with collectiveidea/delayed_job in rails2 and 3. I don't know beanstalkd, but i will try it soon :-). I have followed the suggestions in the resque documentation. I will report it.

Resque vs DelayedJob

How does Resque compare to DelayedJob, and why would you choose one over the other?

• Resque supports multiple queues
• DelayedJob supports finer grained priorities
• Resque workers are resilient to memory leaks / bloat
• DelayedJob workers are extremely simple and easy to modify
• Resque requires Redis
• DelayedJob requires ActiveRecord
• Resque can only place JSONable Ruby objects on a queue as arguments
• DelayedJob can place any Ruby object on its queue as arguments
• Resque includes a Sinatra app for monitoring what's going on
• DelayedJob can be queried from within your Rails app if you want to add an interface

If you're doing Rails development, you already have a database and ActiveRecord. DelayedJob is super easy to setup and works great. GitHub used it for many months to process almost 200 million jobs.

Choose Resque if:

• You need multiple queues
• You don't care / dislike numeric priorities
• You don't need to persist every Ruby object ever
• You have potentially huge queues
• You want to see what's going on
• You expect a lot of failure / chaos
• You can setup Redis
• You're not running short on RAM

Choose DelayedJob if:

• You like numeric priorities
• You're not doing a gigantic amount of jobs each day
• Your queue stays small and nimble
• There is not a lot failure / chaos
• You want to easily throw anything on the queue
• You don't want to setup Redis

Choose Beanstalkd if:

• You like numeric priorities
• You want extremely fast queue
• You don't want to waste you RAM
• You want to serve high number of jobs
• You're fine with JSONable Ruby objects on a queue as arguments
• You need multiple queues

In no way is Resque a "better" DelayedJob, so make sure you pick the tool that's best for your app.

A nice comparison of queueing backend speed:

                 enqueue                work
-------------------------------------------------
delayed job |   200 jobs/sec     120 jobs/sec
resque      |  3800 jobs/sec     300 jobs/sec
rabbitmq    |  2500 jobs/sec    1300 jobs/sec
beanstalk   |  9000 jobs/sec    5200 jobs/sec

Have a nice day!

P.S. There is a RailsCast about resque, Delayed Job (revised version) and Beanstakld. Have a look!

P.P.S. My favourite choiche is now Sidekiq ( very Simple, Fast and efficient for simple jobs ), have a look at this page for comparison.

Answer 2

Amazon Beanstalk isn't Beanstalkd.

Beanstalkd - the queue - does have delayed jobs, that won't be reserved out of the queue until the given number of seconds have passed. If that is what Enqueue_in(10.hours, ... ) means, then it's just syntactic sugar to calculate the number of seconds, and not make a job available till then.

Answer 3

Just a small note: delayed_job 3.0+ supports named queues

object.delay(:queue => 'tracking').method    
Delayed::Job.enqueue job, :queue => 'tracking'    
handle_asynchronously :tweet_later, :queue => 'tweets'