producer-consumer

This is a classic c/p problem where some threads produce data while other read the data. Both the producer and consumers are sharing a const sized buffer. If the buffer is empty then the consumers have to wait and if it is full then the producer has to wait. I am using semaphores to keep track of full or empty queues. The producer is going to decrement free spots semaphore, add value, and increment filled slots semaphore. So I am trying to implement a program that gets some numbers from the generator function, and then prints out the average of the numbers. By treating this as a producer

while (true) { BasicDeliverEventArgs e = (BasicDeliverEventArgs)Consumer.Queue.Dequeue(); IBasicProperties properties = e.BasicProperties; byte[] body = e.Body; Console.WriteLine("Recieved Message : " + Encoding.UTF8.GetString(body)); ch.BasicAck(e.DeliveryTag, false); } This is what we do when we Retrieve Message by subscription..We use While Loop because we want Consumer to listen Continously..what if i want to make this even based..that is when a new message arrives in the queue at that time only Consumer should Consume the message..or on any such similar event.. use the RabbitMQ.Client

When a new consumer/brorker is added or goes down, Kafka triggers a rebalance operation. Is Kafka Rebalancing a blocking operation. Are Kafka consumers blocked while a rebalancing operation is in progress? Depends on what you mean by "blocked". If you mean "are existing connections closed when rebalance is triggered" then the answer is yes. The current Kafka's rebalancing algorithm is unfortunately imperfect. Here is what is happening during consumer rebalance. Assume we have a topic with 10 partitions (0-9), and one consumer (lets name it consumer1 ) consuming it. When a second consumer

In Kafka, I would like to use only a single broker, single topic and a single partition having one producer and multiple consumers (each consumer getting its own copy of data from the broker). Given this, I do not want the overhead of using Zookeeper; Can I not just use the broker only? Why is a Zookeeper must? John Petrone Yes, Zookeeper is required for running Kafka. From the Kafka Getting Started documentation: Step 2: Start the server Kafka uses zookeeper so you need to first start a zookeeper server if you don't already have one. You can use the convenience script packaged with kafka to

I have developed a generic producer-consumer queue which pulses by Monitor in the following way: the enqueue : public void EnqueueTask(T task) { _workerQueue.Enqueue(task); Monitor.Pulse(_locker); } the dequeue: private T Dequeue() { T dequeueItem; if (_workerQueue.Count > 0) { _workerQueue.TryDequeue(out dequeueItem); if(dequeueItem!=null) return dequeueItem; } while (_workerQueue.Count == 0) { Monitor.Wait(_locker); } _workerQueue.TryDequeue(out dequeueItem); return dequeueItem; } the wait section produces the following SynchronizationLockException : "object synchronization method was called

I'm wrestling with the best way to implement my processing pipeline. My producers feed work to a BlockingQueue. On the consumer side, I poll the queue, wrap what I get in a Runnable task, and submit it to an ExecutorService. while (!isStopping()) { String work = workQueue.poll(1000L, TimeUnit.MILLISECONDS); if (work == null) { break; } executorService.execute(new Worker(work)); // needs to block if no threads! } This is not ideal; the ExecutorService has its own queue, of course, so what's really happening is that I'm always fully draining my work queue and filling the task queue, which slowly

I have a typical producer-consumer problem: Multiple producer applications write job requests to a job-table on a PostgreSQL database. The job requests have a state field that starts contains QUEUED on creation. There are multiple consumer applications that are notified by a rule when a producer inserts a new record: CREATE OR REPLACE RULE "jobrecord.added" AS ON INSERT TO jobrecord DO NOTIFY "jobrecordAdded"; They will try to reserve a new record by setting its state to RESERVED. Of course, only on consumer should succeed. All other consumers should not be able to reserve the same record.