MySQL producer consumer with multiple select threads

Question

I've to following situation:

There is a table containing different "jobs" to process and several worker threads consuming those jobs. As I don't want to delete those jobs once finished, I'll just set a "complete" flag for that record.

So in fact I've the following workflow (for each processing thread)

1. Select 1st record which is not complete
2. Process job
3. Set "complete" flag

How do I prevent other threads from consuming the same job (as setting it to "complete" will take a while). Also just updating the "complete" flag in the 2nd step will cause some jobs to be processed twice as there might be a big number of threads processing a small ammount of jobs.

The easiest way would be to lock just the record (yes, I'm using InnoDB), so that other threads aren't able to read just that singe record set. This is easily be doable via an "FOR UPDATE", but that will lock the entire table for future selects and all other "FOR UPDATE" selects have to wait until the 1st one has completed.

Can anyone tell me, how to solve such a situation without delaying all other threads? So in fact a select (can also be limited by LIMIT 1) should only "see" non locked rows ...

A example table structure will look like

JobID | completed
123   | 0
124   | 1
125   | 0

with around 10-50 threads doing a simple

SELECT JobID from jobs WHERE completed = 0;
UPDATE jobs SET completed = 1 WHERE JobID = ?;

Thx for any tips and tricks!

Answer 1

I have a practical solution for you, one that I have seen implemented in a project at my workplace. Instead of using just 0 and 1 for incomplete and completed, expand your set to include more cases.

Let's call that column status. Here are the different values of that column and the corresponding states of the job.

1. When status is 0, the job has not been picked up by any worker thread.
2. When status is 1, the job has been picked up by a worker thread and is under process.
3. When status is 2, the job has failed. (You should consider the possibility of failure in processing.)
4. When status is 3, the job has been completed.

Your threads should contain logic such that it only picks up jobs for whom the status is 0 and changes the status to 1. This will disallow other threads to pick up those jobs which are under process. When the job completes, the status is set to 3 and if the job fails, the status is set to 2. Then the thread can move on and look for another job that is still to be completed.

You could also ask the threads to consider picking up jobs of status 2, but you will have to define logic to specify a finite number of retries.

EDIT:

After a long discussion, we stumbled upon the solution together. My above answer is good in a more generalized state when the 'job' is a process that takes some time to complete. But that wasn't the case in the OP's problem.

So the solution that eventually worked was this:

BEGIN 
SELECT * FROM Jobs WHERE JobID = (SELECT * FROM Jobs WHERE completed = 0 LIMIT 1) LOCK IN SHARE MODE;
UPDATE Jobs SET completed = 1 WHERE JobID = (PREVIOUS ID); 
COMMIT;