Is this very likely to create a memory leak in Tomcat?

Question

I configured tomcat to work with a different external open source.

However, after the tomcat is running for a few minutes I get:

SEVERE: The w

Answer 1

This problem appears when we are using any third party solution, without using the handlers for the cleanup activitiy. For me this was happening for EhCache. We were using EhCache in our project for caching. And often we used to see following error in the logs

 SEVERE: The web application [/products] appears to have started a thread named [products_default_cache_configuration] but has failed to stop it. This is very likely to create a memory leak.
Aug 07, 2017 11:08:36 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/products] appears to have started a thread named [Statistics Thread-products_default_cache_configuration-1] but has failed to stop it. This is very likely to create a memory leak.

And we often noticed tomcat failing for OutOfMemory error during development where we used to do backend changes and deploy the application multiple times for reflecting our changes.

This is the fix we did

<listener>
  <listener-class>
     net.sf.ehcache.constructs.web.ShutdownListener
  </listener-class>
</listener>

So point I am trying to make is check the documentation of the third party libraries which you are using. They should be providing some mechanisms to clean up the threads during shutdown. Which you need to use in your application. No need to re-invent the wheel unless its not provided by them. The worst case is to provide your own implementation.

Reference for EHCache Shutdown http://www.ehcache.org/documentation/2.8/operations/shutdown.html

Answer 2

Sometimes this has to do with configuration changes. When we upgraded from Tomncat 6.0.14 to 6.0.26, we had seen something similar. here is the solution http://www.skill-guru.com/blog/2010/08/22/tomcat-6-0-26-shutdown-reports-a-web-application-created-a-threadlocal-threadlocal-has-been-forcibly-removed/

Answer 3

The key "Transactional Resources" looks like you are talking to the database without a proper transaction. Make sure transaction management is configured properly and no invocation path to the DAO exists that doesn't run under a @Transactional annotation. This can easily happen when you configured transaction management on the Controller level but are invoking DAOs in a timer or are using @PostConstruct annotations. I wrote it up here http://georgovassilis.blogspot.nl/2014/01/tomcat-spring-and-memory-leaks-when.html

Edit: It looks like this is (also?) a bug with spring-data-jpa which has been fixed with v1.4.3. I looked it up in the spring-data-jpa sources of LockModeRepositoryPostProcessor which sets the "Transactional Resources" key. In 1.4.3 it also clears the key again.

Answer 4

I added the following to @PreDestroy method in my CDI @ApplicationScoped bean, and when I shutdown TomEE 1.6.0 (tomcat7.0.39, as of today), it clears the thread locals.

/*
 * To change this template, choose Tools | Templates
 * and open the template in the editor.
 */
package pf;

import java.lang.ref.WeakReference;
import java.lang.reflect.Array;
import java.lang.reflect.Field;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
 *
 * @author Administrator
 * 
 * google-gson issue # 402: Memory Leak in web application; comment # 25
 * https://code.google.com/p/google-gson/issues/detail?id=402
 */
public class ThreadLocalImmolater {

    final Logger logger = LoggerFactory.getLogger(ThreadLocalImmolater.class);

    Boolean debug;

    public ThreadLocalImmolater() {
        debug = true;
    }

    public Integer immolate() {
        int count = 0;
        try {
            final Field threadLocalsField = Thread.class.getDeclaredField("threadLocals");
            threadLocalsField.setAccessible(true);
            final Field inheritableThreadLocalsField = Thread.class.getDeclaredField("inheritableThreadLocals");
            inheritableThreadLocalsField.setAccessible(true);
            for (final Thread thread : Thread.getAllStackTraces().keySet()) {
                    count += clear(threadLocalsField.get(thread));
                    count += clear(inheritableThreadLocalsField.get(thread));
            }
            logger.info("immolated " + count + " values in ThreadLocals");
        } catch (Exception e) {
            throw new Error("ThreadLocalImmolater.immolate()", e);
        }
        return count;
    }

    private int clear(final Object threadLocalMap) throws Exception {
        if (threadLocalMap == null)
                return 0;
        int count = 0;
        final Field tableField = threadLocalMap.getClass().getDeclaredField("table");
        tableField.setAccessible(true);
        final Object table = tableField.get(threadLocalMap);
        for (int i = 0, length = Array.getLength(table); i < length; ++i) {
            final Object entry = Array.get(table, i);
            if (entry != null) {
                final Object threadLocal = ((WeakReference)entry).get();
                if (threadLocal != null) {
                    log(i, threadLocal);
                    Array.set(table, i, null);
                    ++count;
                }
            }
        }
        return count;
    }

    private void log(int i, final Object threadLocal) {
        if (!debug) {
            return;
        }
        if (threadLocal.getClass() != null &&
            threadLocal.getClass().getEnclosingClass() != null &&
            threadLocal.getClass().getEnclosingClass().getName() != null) {

            logger.info("threadLocalMap(" + i + "): " +
                        threadLocal.getClass().getEnclosingClass().getName());
        }
        else if (threadLocal.getClass() != null &&
                 threadLocal.getClass().getName() != null) {
            logger.info("threadLocalMap(" + i + "): " + threadLocal.getClass().getName());
        }
        else {
            logger.info("threadLocalMap(" + i + "): cannot identify threadlocal class name");
        }
    }

}

Answer 5

The message is actually pretty clear: something creates a ThreadLocal with value of type org.apache.axis.MessageContext - this is a great hint. It most likely means that Apache Axis framework forgot/failed to cleanup after itself. The same problem occurred for instance in Logback. You shouldn't bother much, but reporting a bug to Axis team might be a good idea.

Tomcat reports this error because the ThreadLocals are created per HTTP worker threads. Your application is undeployed but HTTP threads remain - and these ThreadLocals as well. This may lead to memory leaks (org.apache.axis.MessageContext can't be unloaded) and some issues when these threads are reused in the future.

For details see: http://wiki.apache.org/tomcat/MemoryLeakProtection