How to manage two or more consumers via pthreads?

Question

I have a generic problem I am looking to solve, where chunks of binary data sent from a standard input or regular file stream to an application, which in turn converts that bina

Answer 1

Regarding the part 'How to manage two or more consumers via pthreads?' of your post let me cite these points about 'Designing Threaded Programs':

In general though, in order for a program to take advantage of Pthreads, it must be able to be organized into discrete, independent tasks which can execute concurrently. For example, if routine1 and routine2 can be interchanged, interleaved and/or overlapped in real time, they are candidates for threading.

and

Several common models for threaded programs exist:

• Manager/worker: a single thread, the manager assigns work to other threads, the workers. Typically, the manager handles all input and
  parcels out work to the other tasks. At least two forms of the
  manager/worker model are common: static worker pool and dynamic
  worker pool.
• Pipeline: a task is broken into a series of suboperations, each of which is handled in series, but concurrently, by a different thread.
  An automobile assembly line best describes this model.
• Peer: similar to the manager/worker model, but after the main thread creates other threads, it participates in the work.

Regarding your problem...

The problem I am facing is that I need to add a sleep(1) statement in consume_gunzip_chunk, before doing the read, in order to get things working properly.

Eric Lippert Best Practices with Multithreading in C# might not solve it but, they should help you finding the right solution to your multi-threaded program, particularly points 5, and 8:

5.At all costs avoid shared memory. Most threading bugs are caused by a failure to understand real-world shared memory semantics. If you must make threads, treat them as though they were processes: give them everything they need to do their work and let them work without modifying the memory associated with any other thread. Just like a process doesn't get to modify the memory of any other process.

8.If you use Thread.Sleep with an argument other than zero or one in any production code, you are possibly doing something wrong. Threads are expensive; you don't pay a worker to sleep, so don't pay a thread to sleep either. If you are using sleeps to solve a correctness issue by avoiding a timing problem -- as you appear to be in your code -- then you definitely have done something deeply wrong. Multithreaded code needs to be correct irrespective of accidents of timing.