How to detect alarm-based blocking RabbitMQ producer?

Question

I have a producer sending durable messages to a RabbitMQ exchange. If the RabbitMQ memory or disk exceeds the watermark threshold, RabbitMQ will block my producer. The documentation says that it stops reading from the socket, and also pauses heartbeats.

What I would like is a way to know in my producer code that I have been blocked. Currently, even with a heartbeat enabled, everything just pauses forever. I'd like to receive some sort of exception so that I know I've been blocked and I can warn the user and/or take some other action, but I can't find any way to do this. I am using both the Java and C# clients and would need this functionality in both. Any advice? Thanks.

Answer 1

Sorry to tell you but with RabbitMQ (at least with 2.8.6) this isn't possible :-(

had a similar problem, which centred around trying to establish a channel when the connection was blocked. The result was the same as what you're experiencing.

I did some investigation into the actual core of the RabbitMQ C# .Net Library and discovered the root cause of the problem is that it goes into an infinite blocking state.

You can see more details on the RabbitMQ mailing list here:

http://rabbitmq.1065348.n5.nabble.com/Net-Client-locks-trying-to-create-a-channel-on-a-blocked-connection-td21588.html

One suggestion (which we didn't implement) was to do the work inside of a thread and have some other component manage the timeout and kill the thread if it is exceeded. We just accepted the risk :-(

Answer 2

The Rabbitmq uses a blocking rpc call that listens for a reply indefinitely.

If you look the Java client api, what it does is:

    AMQChannel.BlockingRpcContinuation k = new AMQChannel.SimpleBlockingRpcContinuation();
    k.getReply(-1);

Now -1 passed in the argument blocks until a reply is received.

The good thing is you could pass in your timeout in order to make it return. The bad thing is you will have to update the client jars.

If you are OK with doing that, you could pass in a timeout wherever a blocking call like above is made. The code would look something like:

try {
                    return k.getReply(200);
                } catch (TimeoutException e) {
                    throw new MyCustomRuntimeorTimeoutException("RabbitTimeout ex",e);
                }

And in your code you could handle this exception and perform your logic in this event.

Some related classes that might require this fix would be:

com.rabbitmq.client.impl.AMQChannel
com.rabbitmq.client.impl.ChannelN
com.rabbitmq.client.impl.AMQConnection

FYI: I have tried this and it works.