ThreadPoolExecutor : : TaskRejectedException from Executor

Question

My application is reading messages through Jms MessageListener class and at some point of time it is throwing TaskRejectedException. I know most of you will say that the number of threads is exceeded by maxPoolSize and queue is also full.

But I observed something. The number of messages sent to the queue from which the MessageListener class is fetching messages is 10353 and my spring property for threadPoolExecutor is below :

<bean id="ticketReaderThreadPool" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" scope="singleton" destroy-method="destroy">
    <property name="corePoolSize" value="10" />
    <property name="maxPoolSize" value="150" />
    <property name="queueCapacity" value="11000" />
</bean>

Now according to me, the maxPoolSize is more than enough to handle these many requests. So if anyone of you can give a reason apart from maxPoolSize breach then please do so.

We are facing this issue for the second time now, previously we already tried increasing the maxPoolSize but again after 15 days we are experiencing this exception for around 5000 to 8000 times a day.

Update:

This is the full stack trace of the exception:

General Exception occurred while reading from Queue/Processing the message org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor@408b9775] did not accept task: com.batman.rapid.rapidserver.sla.TicketHandler@1be5e598 at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:244) at com.batman.rapid.rapidserver.sla.JmsTicketReceiver.onMessage(JmsTicketReceiver.java:58) at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560) at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498) at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467) at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325) at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263) at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058) at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050) at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.RejectedExecutionException at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1774) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:768) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:656) at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:241) ... 10 more

This is the relevant code:

if (message instanceof TextMessage)
{
    textMessage = (TextMessage) message;
    ticketReaderThreadPool.execute(new TicketHandler(textMessage.getText()));
}

Below is the configuration requested :

    <!-- End of JMS Queue Support -->

    <bean id="ticketReaderThreadPool" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" scope="singleton" destroy-method="destroy">
    <property name="corePoolSize" value="10" />
    <property name="maxPoolSize" value="150" />
    <property name="queueCapacity" value="11000" />
</bean>

    <bean id="notificationThreadPool" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" scope="singleton" destroy-method="destroy">
            <property name="corePoolSize" value="10" />
            <property name="maxPoolSize" value="100" />
            <property name="queueCapacity" value="10000" />
    </bean>

    <bean id="notificationManager" class="com.batman.rapid.rapidserver.sla.scheduler.NotificationManager" scope="singleton">
            <property name="defaultPercent" value="80"></property>
    </bean>

    <bean id="dbUpdateThreads" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" scope="singleton" destroy-method="destroy">
            <property name="corePoolSize" value="1" />
            <property name="maxPoolSize" value="100" />
            <property name="queueCapacity" value="10000" />
    </bean>

Answer 1

Not sure what is the exact cause. However I strongly feel that, the max message count might be going beyond the maxqueueCapacity you had set. However the vary approach you are using deviates from general usage of JMS. Generally in DefaultMessageListener we configure the number of Max-consumers to process the in parallel for async processing. based on the server resources we configure the max consumer that server can handle

However in your case from DefaultMessageListener the message is read and you spawn new thread and execute your business logic in new thread. since message listener read is faster than business logic, task gets accumulated in the thread task queue.

I would suggest to re-approach your current implementation. as it doesn't support transactions. I mean when server crashes all pending message processing in tasks would be terminated/lost and since you would set auto-ack, the message would have been removed from queue. Other issue like to set manually limit of thread queue etc.

Finally coming back to your issue, you can do a quick check if there are any locks in the 150 threads you had set