message-queue

Consider a PHP web application whose purpose is to accept user requests to start generic asynchronous jobs, and then create a worker process/thread to run the job. The jobs are not particularly CPU or memory intensive, but are expected to block on I/O calls fairly often. No more than one or two jobs should be started per second, but due to the long run times there may be many jobs running at once. Therefore, it's of utmost importance that the jobs run in parallel. Also, each job must be monitored by a manager daemon responsible for killing hung workers, aborting workers on user request, etc.

I have some update triggers which push jobs onto the Sidekiq queue. So in some cases, there can be multiple jobs to process the same object. There are a couple of uniqueness plugins ( "Middleware" , Unique Jobs ), they're not documented much, but they seem to be more like throttlers to prevent repeat processing ; what I want is a throttler that prevents repeat creating of the same jobs. That way, an object will always be processed in its freshest state. Is there a plugin or technique for this? Update: I didn't have time to make a middleware, but I ended up with a related cleanup function to

I'm using Kafka's high-level consumer. Because I'm using Kafka as a 'queue of transactions' for my application, I need to make absolutely sure I don't miss or re-read any messages. I have 2 questions regarding this: How do I commit the offset to zookeeper? I will turn off auto-commit and commit offset after every message successfully consumed. I can't seem to find actual code examples of how to do this using high-level consumer. Can anyone help me with this? On the other hand, I've heard committing to zookeeper might be slow, so another way may be to locally keep track of the offsets? Is this

When a new consumer/brorker is added or goes down, Kafka triggers a rebalance operation. Is Kafka Rebalancing a blocking operation. Are Kafka consumers blocked while a rebalancing operation is in progress? Depends on what you mean by "blocked". If you mean "are existing connections closed when rebalance is triggered" then the answer is yes. The current Kafka's rebalancing algorithm is unfortunately imperfect. Here is what is happening during consumer rebalance. Assume we have a topic with 10 partitions (0-9), and one consumer (lets name it consumer1 ) consuming it. When a second consumer

I have a JMS client which is producing messages and sending over a JMS queue to its unique consumer. What I want is more than one consumer getting those messages. The first thing that comes to my mind is converting the queue to a topic, so current and new consumers can subscribe and get the same message delivered to all of them. This will obviously involve modifying the current clients code in both producer and consumer side of things. I would like to also look at other options like creating a second queue, so that I don't have to modify the existing consumer. I believe there are advantages in

I am new to message brokers like RabbitMQ which we can use to create tasks / message queues for a scheduling system like Celery . Now, here is the question: I can create a table in PostgreSQL which can be appended with new tasks and consumed by the consumer program like Celery. Why on earth would I want to setup a whole new tech for this like RabbitMQ? Now, I believe scaling cannot be the answer since our database like PostgreSQL can work in a distributed environment. I googled for what problems does the database poses for the particular problem, and I found: polling keeps the database busy

Under what conditions would one favor apps talking via a message queue instead of via web services (I just mean XML or JSON or YAML or whatever over HTTP here, not any particular type)? I have to talk between two apps on a local network. One will be a web app and have to request commands on another app (running on different hardware). The requests are things like creating users, moving files around, and creating directories. Under what conditions would I prefer XML Web Services (or straight TCP or something) to using a Message queue? The web app is Ruby on Rails, but I think the question is