producer-consumer

classical producer consumer threading

余生颓废 提交于 2019-12-04 16:00:29
In the Classical producer consumer problem . producer sleeps when itemCount == BUFFER_SIZE amd wakes up again when it goes down. But once itemCount grows up, producer thread is put to sleep. how can it know that itemCount has gone down and it needs to wakeup ? In pseudo-code the producer is something like: void producer_thread() { while(true) queue.push( produce() ); } so consider the queue push method (I've used pthreads here, but the same logic applies with other libraries) void SynchronizedQueue::push(Item const &i) { pthread_mutex_lock(&mutex); // queue is full, so wait for consumer while

Buffered Background InputStream Implementations

怎甘沉沦 提交于 2019-12-04 10:55:16
问题 I've written background InputStream (and OutputStream ) implementations that wrap other streams, and read ahead on a background thread, primarily allowing for decompression/compression to happen in different threads from the processing of the decompressed stream. It's a fairly standard producer/consumer model. This seems like an easy way to make good use of multi-core CPUs with simple processes that read, process, and write data, allowing for more efficient use of both CPU and disk resources.

Generic .Net Producer/Consumer

余生长醉 提交于 2019-12-04 10:52:18
问题 I'm toying with the idea of implementing a generic Producer/Consumer pair + processing queue in C# for fun. The idea is you can just create objects that implement appropriate IProducer and IConsumer interfaces (default implementations supplied), which will consist mainly of delegates, pass them to a QueueProcessor class instance, tell it how many consumers you want, and go. But I say to myself, "Self, surely this has been done before." So does anyone know of a good , generic implementation of

How to implement blocking read using POSIX threads

こ雲淡風輕ζ 提交于 2019-12-04 08:45:33
I would like to implement a producer/consumer scenario that obeys interfaces that are roughly: class Consumer { private: vector<char> read(size_t n) { // If the internal buffer has `n` elements, then dequeue them // Otherwise wait for more data and try again } public: void run() { read(10); read(4839); // etc } void feed(const vector<char> &more) { // Safely queue the data // Notify `read` that there is now more data } }; In this case, feed and run will run on separate threads and read should be a blocking read (like recv and fread ). Obviously, I will need some kind of mutual exclusion on my

Consuming from Kafka failed Iterator is in failed state

懵懂的女人 提交于 2019-12-04 01:57:47
问题 I am getting exception while consuming the messages from kafka. org.springframework.messaging.MessagingException: Consuming from Kafka failed; nested exception is java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Iterator is in failed state I have one consumer in the application context with one outbound adapter. Consumer configuration in application context <int-kafka:consumer-context id="consumerContext" consumer-timeout="4000" zookeeper-connect="zookeeperConnect">

C Confused on how to initialize and implement a pthread mutex and condition variable

帅比萌擦擦* 提交于 2019-12-03 21:43:34
I'm a little bit confused on how to initialize and implement a pthread mutex and condition variable. The goal of the program is to have producers place a set number of ints in a queue and consumers take the ints out of the queue. I must also be able to define the number of producer and consumer threads that are created. In the starter code, I am give these: // Locks & Condition Variables pthread_mutex_t lock; // Lock shared resources among theads pthread_cond_t full; // Condition indicating queue is full pthread_cond_t empty; // Condition indicating queue is empty as shared resources. In the /

Multiple Producer Multiple Consumer Multithreading Java

有些话、适合烂在心里 提交于 2019-12-03 21:01:19
I'm trying out Multiple Producer - Multiple Consumer use case of Producer-Consumer problem. I'm using BlockingQueue for sharing common queue between multiple producers/consumers. Below is my code. Producer import java.util.concurrent.BlockingQueue; public class Producer implements Runnable { private BlockingQueue inputQueue; private static volatile int i = 0; private volatile boolean isRunning = true; public Producer(BlockingQueue q){ this.inputQueue=q; } public synchronized void run() { //produce messages for(i=0; i<10; i++) { try { inputQueue.put(new Integer(i)); Thread.sleep(1000); } catch

Producer/consumer pattern with a fixed-size FIFO queue

余生长醉 提交于 2019-12-03 15:48:42
I need to implement the producer/consumer pattern around a fixed-size FIFO queue. I think a wrapper class around a ConcurrentQueue might work for this but I'm not completely sure (and I've never worked with a ConcurrentQueue before). The twist in this is that the queue needs to only hold a fixed number of items (strings, in my case). My application will have one producer task/thread and one consumer task/thread. When my consumer task runs, it needs to dequeue all of the items that exist in the queue at that moment in time and process them. For what it's worth, processing of the queued items by

Java: High-performance message-passing (single-producer/single-consumer)

女生的网名这么多〃 提交于 2019-12-03 14:23:36
I initially asked this question here , but I've realized that my question is not about a while-true loop. What I want to know is, what's the proper way to do high-performance asynchronous message-passing in Java? What I'm trying to do... I have ~10,000 consumers, each consuming messages from their private queues. I have one thread that's producing messages one by one and putting them in the correct consumer's queue. Each consumer loops indefinitely, checking for a message to appear in its queue and processing it. I believe the term is "single-producer/single-consumer", since there's one

How to aggregate the data from an async producer and write it to a file?

你。 提交于 2019-12-03 13:23:39
I'm learning about async/await patterns in C#. Currently I'm trying to solve a problem like this: There is a producer (a hardware device) that generates 1000 packets per second. I need to log this data to a file. The device only has a ReadAsync() method to report a single packet at a time. I need to buffer the packets and write them in the order they are generated to the file, only once a second. Write operation should fail if the write process is not finished in time when the next batch of packets is ready to be written. So far I have written something like below. It works but I am not sure