To fork or not to fork?

Question

I am re-developing a system that will send messages via http to one of a number of suppliers. The original is perl scripts and it's likely that the re-development will also use perl.

In the old system, there were a number of perl scripts all running at the same time, five for each supplier. When a message was put into the database, a random thread number (1-5) and the supplier was chosen to ensure that no message was processed twice while avoiding having to lock the table/row. Additionally there was a "Fair Queue Position" field in the database to ensure that a large message send didn't delay small sends that happened while the large one was being sent.

At some times there would be just a couple of messages per minute, but at other times there would be a dump of potentially hundreds of thousands of messages. It seems to me like a resource waste to have all the scripts running and checking for messages all of the time so I am trying to work out if there is a better way to do it, or if the old way is acceptable.

My thoughts right now lie with the idea of having one script that runs and forks as many child processes as are needed (up to a limit) depending on how much traffic there is, but I am not sure how best to implement it such that each message is processed only once, while the fair queuing is maintained.

My best guess right now is that the parent script updates the DB to indicate which child process should handle it, however I am concerned that this will end up being less efficient than the original method. I have little experience of writing forking code (last time I did it was about 15 years ago).

Any thoughts or links to guides on how best to process message queues appreciated!

Answer 1

You could use Thread::Queue or any other from this: Is there a multiprocessing module for Perl?

If the old system was written in Perl this way you could reuse most part of it.

Non working example:

use strict;
use warnings;

use threads;
use Thread::Queue;

my $q = Thread::Queue->new();    # A new empty queue

# Worker thread
my @thrs = threads->create(sub {
                            while (my $item = $q->dequeue()) {
                                # Do work on $item
                            }
                         })->detach() for 1..10;#for 10 threads
my $dbh = ...
while (1){
  #get items from db
  my @items = get_items_from_db($dbh);
  # Send work to the thread
  $q->enqueue(@items);
  print "Pending items: "$q->pending()."\n";
  sleep 15;#check DB in every 15 secs
}

Answer 2

I would suggest using a message queue server like RabbitMQ.

One process feeds work into the queue, and you can have multiple worker processes consume the queue.

Advantages of this approach:

• workers block when waiting for work (no busy waiting)
• more worker processes can be started up manually if needed
• worker processes don't have to be a child of a special parent process
• RabbitMQ will distribute the work among all workers which are ready to accept work
• RabbitMQ will put work back into the queue if the worker doesn't return an ACK
• you don't have to assign work in the database
• every "agent" (worker, producer, etc.) is an independent process which means you can kill it or restart it without affecting other processes

To dynamically scale-up or down the number workers, you can implement something like:

1. have workers automatically die if they don't get work for a specified amount of time
2. have another process monitor the length of the queue and spawn more workers if the queue is getting too big

Answer 3

I would recommend using beanstalkd for a dedicated job server, and Beanstalk::Client in your perl scripts for adding jobs to the queue and removing them.

You should find beanstalkd easier to install and set up compared to RabbitMQ. It will also take care of distributing jobs among available workers, burying any failed jobs so they can be retried later, scheduling jobs to be done at a later date, and many more basic features. For your worker, you don't have to worry about forking or threading; just start up as many workers as you need, on as many servers as you have available.

Either RabbitMQ or Beanstalk would be better than rolling your own db-backed solution. These projects have already worked out many of the details needed for queueing, and implemented features you may not realize yet that you want. They should also handle polling for new jobs more efficiently, compared to sleeping and selecting from your database to see if there's more work to do.