MQ Queue with multiple consumers but only one active

Question

We have one MQ Queue which receives messages from an external system out of our control. Our system processing the incoming messages is a critical one and needs to be up and running 27x7 no matter what.

The order in which the incoming messages are processed is also not negotiable which means we need to process them in exactly the order they arrived.

To make sure our system is 100% available we deployed our system to a bunch of physical machines able to process those messages.

Once the messages reached our system we put in place a mechanism to make sure the messages processing does not go out of order while also getting some performance gain as a result of the parallel processing. For us the performance gain is a good to have, however it is rather a side effect as our main goal is the high availability while assuring the right processing order.

My thoughts were to have on every single machine an MDB able to process the incoming messages but only have one active consumer at a time.

We are using Webshere MQ as a JMS provider and Webshere Application Server 8.5 to deploy our application.

The problem with multiple consumers listening to the same queue does not seem to be a workable solution as when messages arrive in bulk they would be round-robin passed to all consumers and there is no way to control how this is going to happen and the messages easily go out of sequence.

When I manually stopped all the listeners but one obviously the messages got processed in order. But manually shutting down and starting up such listeners is definitely not a HA solution.

We could put in place monitoring processes to check for health of the system and shut things down or start them up as required but this still looks too weak to me. What we want in fact is to have all listeners up and running but only one receiving the messages. If that one goes down for whatever reasons then another one sitting there will become active and start processing messages.

Initially we considered using a topic rather than a queue but this comes with other issues like below:

1. we cannot control the source of our messages
2. the high volume of messages we have would put us in in trouble with our going down subscribers that have to be durable and such when coming up back will have to deal with lots of pending messages
3. the input queues are already part of a cluster and changing all the infrastructure would require a lot of work

Anyway in my view it has to be an existing pattern to accommodate situations like this. Any help, suggestion would be greatly appreciated.

The solution does not have to be a specific MQ one, any idea is welcome.

Thanks in advance

Answer 1

Create a second queue, we'll call it the "control queue." Into this queue, put a single message, we'll call it the "token." Change application processing as follows:

1. Listen on the control queue for a message.
2. Get the token from the control queue under syncpoint.
3. Put the same token message back on the control queue, also under syncpoint.
4. Process a transaction from the normal input queue, also under syncpoint.
5. COMMIT the messages.
6. Loop.

The COMMIT completes the transaction on the input queue and makes the token available to the other MDBs. No processing of the input queue can occur except by the MDB that has the token under syncpoint. However you can have any number of MDBs waiting on the token. A failure of any one of them allows the others to take over instantly.

No need to use XA, by the way. WMQ's single-phase COMMIT works great with this.

Answer 2

When applications are trying to use the queue through their MDB listeners, we can restrict them by defining the queue with DEFSOPT(Exclusive). This will make sure that only one application can consume messages from that queue.

If we wish to restrict to only one instance of the application, define it as NOSHARE. So that, one instance of one application can get hold of messages on the queue at a time. Others will get their turn when current one releases the lock.

Answer 3

in my opinion synchronizing multiple consumers is not a big problem and is the most efficient solution. I don't know where processing result have to be recorded (maybe JMS queue again ? ), but i would try to use a lightweitght agent before that point. You can use timestamps or implement a counter over JMS to preserve order. Consumers could execute in parallel and then post in a support queue. Than the single agent can order them using a queuebrowser and then a transaction. This agent should be "wathdogged".

Alessandro