How to execute more than one background process in laravel?

Question

First of all, I know about queues and now have good experience with queues. The problem with the queue is, it is a queue. I want to execute multiple functions or commands together in the background. Queues will keep second command or function in a queue and will execute once the first one is done executing!

For example, I have a table with ~3,000,000 records and I want to process them faster. What I can do is divide them into 5 equal chunks and execute 5 commands altogether so that I can utilize my CPU as well as process data 5 times faster.

So, How can I do this with Laravel? Queues are not going to work because they execute stuff one after another. If your idea is to create multiple 5 multiple queues and supervisors to accomplish, that's not a standard way to do it I think.

Any Idea on what can be done in this case?

Answer 1

Just to add something from my personal experience.

First install and configure supervisor for your OS accordingly, following are the confs for linux basaed OS e.g. Ubuntu

Supervisor confs: (/etc/supervisor/conf.d)

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work database --sleep=3 --tries=3
autostart=true
autorestart=true
user=username
numprocs=25
redirect_stderr=true
stderr_events_enabled=true
stderr_logfile=/var/www/app/storage/logs/worker.error.log
stdout_logfile=/var/www/app/storage/logs/worker.log

Then create jobs according to your needs and dispatch.

Supervisor will process jobs simultaneously, in this particular case 25 jobs will be processing at a time.

Answer 2

Finally, I found a solution. It is very very easy. So, Here is how it works.

First of all, I divide records into the number of chunks (For example, 5 chunks. It will divide 3 million items into 5 chunks having 600k items each)

Then, I can push a command to queues that I create instantaneously for the chunks and execute queue worker for that queue only once using --once option. To make it simple to understand, Here is the code that I am using.

$chunk_id = 0;
foreach($chunks as $chunk){
    // Adding chunk to queue
    Artisan::queue('process:items',[
        'items' => $chunk,
    ])->onQueue('processChunk'.$chunk_id);

    // Executing queue worker only once
    exec('php artisan queue:work --queue=processChunk'.$chunk_id.' --once > storage/logs/process.log &');

    $chunk_id++;
}

With exec command, we are executing queue worker for the specific queue created for the specific chunk. Also, we've added & at the end of the command which forces the command to execute in the background at OS level.

This is how it can be done. I tested it and it is working smoothly! Anything else to improve or are there any drawbacks of using this method?