producer-consumer

As the title implies I have a deadlock and no idea why. I have multiple producers and only one consumer. The schedule_task method will get called by multiple processes after the thread has called the get method of the queue from logging import getLogger from time import sleep from threading import Event, Thread from multiprocessing import Process from Queue import Queue class TaskExecutor(object): def __init__(self): print("init taskExecutor") self.event = Event() self.taskInfos = Queue() task_thread = Thread(target=self._run_worker_thread) self._instantEnd = False self._running = True task

I'm trying to implement a producer-consumer like situation in my block level driver (on linux kernel version 2.6.39.1). My block driver's make_request_fn receives a stream of struct bio from a userlevel application. On receiving these BIOs, they are queued up. Next, I create a new struct bio which will hold all the information present in the queued BIOs. This new "merged_bio" will be submitted only if there is a struct request slot available in the lower level driver's request_queue . In the meantime, my block driver will continue to receive BIOs from the userlevel application and will queue

I am using Kafka and Zookeeper as the main components of my data pipeline, which is processing thousands of requests each second. I am using Samza as the real time data processing tool for small transformations that I need to make on the data. My problem is that one of my consumers (lets say ConsumerA ) consumes several topics from Kafka and processes them. Basically creating a summary of the topics that are digested. I further want to push this data to Kafka as a separate topic but that forms a loop on Kafka and my component. This is what bothers me, is this a desired architecture in Kafka?

I wrote a BlockingQueue in order to communicate two threads. You could say it follows the Producer-Consumer pattern, with an unbounded buffer. Therefore, I implemented it using a Critical Section and a Semaphore, like this: #pragma once #include "Semaphore.h" #include "Guard.h" #include <queue> namespace DRA{ namespace CommonCpp{ template<class Element> class BlockingQueue{ CCriticalSection m_csQueue; CSemaphore m_semElementCount; std::queue<Element> m_Queue; //Forbid copy and assignment BlockingQueue( const BlockingQueue& ); BlockingQueue& operator=( const BlockingQueue& ); public:

I'm looking for a single-producer, single-consumer FIFO implementation that would perform faster than the normal lock-write-unlock-signal / waitForSignal-lock-read-unlock stuff. I'm looking for something supported by most POSIX operating systems (x86 specific is fine) written in either C or C++. I'm not looking to pass anything larger than a pointer. I'm not necessarily attached to the lock-free idea, but I do want something fast and correct. One of the papers I read on the subject mentioned a two-queue approach that seemed interesting, but I haven't been able to find much about that since

When using BlockingQueue to consume data that is produced what is the most efficient method for waiting for the data to appear? Scenario: Step 1) The data list will be a data store where timestamps are added to. These timestamps are required to be ordered by closest to current time priority. This list may be empty. A thread will be inserting the timestamps into it. Produce Step 2) I want to consume the data in here in another thread that will take the timestamps from data and check if they are after the current time. Consumer then Produce Step 3) If they are after the current time then send

I have producer and consumer connected with BlockingQueue . Consumer wait records from queue and process it: Record r = mQueue.take(); process(r); I need pause this process for a while from other thread. How to implement it? Now I think implement it such, but it's looks like bad solution: private Object mLock = new Object(); private boolean mLocked = false; public void lock() { mLocked = true; } public void unlock() { mLocked = false; mLock.notify(); } public void run() { .... Record r = mQueue.take(); if (mLocked) { mLock.wait(); } process(r); } I think your solution is simple and elegant,