message-queue

I am trying to read messages on Kafka topic, but I am unable to read it. The process gets killed after sometime, without reading any messages. Here is the rebalancing error which I get: [2014-03-21 10:10:53,215] ERROR Error processing message, stopping consumer: (kafka.consumer.ConsoleConsumer$) kafka.common.ConsumerRebalanceFailedException: topic-1395414642817-47bb4df2 can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:428) at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer

To cut a long story short, I am working on a project where we are rewriting a large web application for all the usual reasons. The main aim of the rewrite is to separate this large single application running on single server into many smaller decoupled applications, which can be run on many servers. Ok here's what I would like: I would like HTTP to be the main transport mechanism. When one application for example the CMS has been updated it will contact the broker via http and say "I've changed" , then the broker will send back a 200 OK to say "thanks I got the message" . The broker will then

I need a queue for passing messages from one thread (A) to another (B), however ive not been able to find one that really does what I want, since they generally allow adding an item to fail, a case which in my situation is pretty much fatal since the message needs to be processed, and the thread really cant stop and wait for spare room. Only thread A adds items, and only thread B reads them Thread A must never block, however thread B is not performance critical, so it can Adding items must always succeed, so the queue cant have an upper size limit (short of running out of memory on the system)

I ask this question after trying my best to research the best way to implement a message queue server. Why do operating systems put limits on the number of open file descriptors a process and the global system can have? My current server implementation uses zeromq, and opens a subscriber socket for each connected websocket client. Obviously that single process is only going to be able to handle clients to the limit of the fds. When I research the topic I find lots of info on how to raise system limits to levels as high as 64k fds but it never mentions how it affects system performance and why

I am looking to implement some sort of chat server. And I want it to scale. This seems like a big question, so I guess I expect the answers to be direction pointers, sort of exploratory. The end-user clients are web or phone client. I think some sort of websocket implementation, such as Socket.IO is nice. On the server side I wish to use Node.js. I want the architecture to be scalable so that the number of users are not limited (well, within reason, the chance of big hit is not expected, and if it is, the chance of having smarter, experienced people to work on it is reasonable instead of

I have been doing r&d for the scope of SQL Server Service Broker to replace current messaging solution MSMQ. I want to know disadvantages of SQL Server Service Broker in contrast to MSMQ for following criteria. Development Troubleshooting Performance (let's say we need to process 100,000 messages daily, having avg size around 25 KB) Scalability I've used Service Broker in my current project, having previously used MSMQ (brokered by MassTransit ) with great success. I was initially dubious about using Service Broker, but I have to admit it has performed very well. If you're using the Publish /

I would like to learn what are the scenarios/usecases/ where messaging like RabbitMQ can help consumer web applications. Are there any specific resources to learn from? What web applications currently are making use of such messaging schemes and how? In general, a message bus (such as RabbitMQ, but not limited to) allows for a reliable queue of job processing. What this means to you in terms of a web application is the ability to scale your app as demand grows and to keep your UI quick and responsive. Instead of forcing the user to wait while a job is processed they can request a job to be

An existing process changes the status field of a booking record in a table, in response to user input. I have another process to write, that will run asynchronously for records with a particular status. It will read the table record, perform some operations (including calls to third party web services), and update the record's status field to indicate that processing is completed (or In Error, with an error count). This operation sounds very similar to a queue. What are the benefits and tradeoffs of using MSMQ over a SQL Table in this situation, and why should I choose one over the other? It

I'm currently designing an application which I will ultimately want to move to Windows Azure. In the short term, however, it will be running on a server which I will host myself. The application involves a number of separate web applications - some of these are essentially WCF services which receive data, and some are sites for users to manage data. In addition, there will need to be a worker service running in the background which will process data in various ways. I'm very keen to use a decoupled architecture for this. Ideally I'm wanting the components (i.e. web apps and worker service) to