How to resolve OutOfMemoryError with ImageIO plugins as the cause?

Question

At work we have some tomcat servers running several webapps, about half of which have to do some image processing.

Before doing their image processing, these webapps do a ImageIO.scanForPlugins() to get the appropriate image readers and writers into memory. While before this was just run anytime an image needed to be processed, we now run the scan only when the webapps are initialized (since we don't add any jars after running, why run the scan more than once?)

Some days later, the tomcat instance crashed due to an OutOfMemoryError. Luckily we had the HeapDumpOnOutOfMemoryError option set, so I looked at the heap dump. In the dump I found that 97% of memory was taken by an instance of a javax.imageio.spi.PartialOrderIterator. Most of that space was taken up by it's backing java.util.LinkedList, which had 18 million elements. The linked list is made up of javax.imageio.spi.DigraphNode, which contains the image readers and writers loaded by ImageIO.scanForPlugins().

"Aha", I thought, "we must be running the scan in a loop somewhere and we're just adding the same elements over and over again". But, I figured I should double check this assumption, so I wrote the following test class:

import javax.imageio.ImageIO;

public class ImageIOTesting {

public static void main(String[] args) {

    for (int i = 0; i < 100000; i++) {
        ImageIO.scanForPlugins();
        if (i % 1000 == 0) {
            System.out.println(Runtime.getRuntime().totalMemory() / 1024);
        }
    }
}
}

However, when I run this class on the server environment, the amount of memory in use never changes!

A quick dig through the source of the javax.imageio packages shows that the scan checks to see if a service provider is already registered, and if so it deregisters the old provider before registering the new one. So now the question is: Why do I have this giant linked list of service providers? Why are they stored as a directed graph? And more importantly, how do I prevent this from happening?

Answer 1

Late answer to an old question, but anyway:

Because the ImageIO plugin registry (the IIORegistry) is "VM global", it doesn't by default work well with servlet contexts. This is especially evident if you load plugins from the WEB-INF/lib or classes folder, as the OP seems to do.

Servlet contexts dynamically loads and unloads classes (using a new class loader per context). If you restart your application, old classes will by default remain in memory forever (because the next time scanForPlugins is called, it's another ClassLoader that scans/loads classes, and thus they will be new instances in the registry. In the test code loop, the same ClassLoader is used all the time, thus instances are replaced, and the real problem never manifests).

To work around this resource leak, I recommend using a ContextListener to make sure these "context local" plugins are explicitly removed. Here's an example IIOProviderContextListener I've used in a couple of projects, that implements dynamic loading and unloading of ImageIO plugins.

Answer 2

Looks like a ugly memory leak to me, i can't guess where it is without looking the whole code, but i know what you should check very carefully.

Possible causes:

_Global Variable List, adding elements, never remove them.

_Infinite/big loops, loading lots of elements to Lists, never remove them.

Also it would help if you use a Java profiler, such as VisualVM

If you provide more information in order to have a better idea of how it works I i may be able to improve my answer, there's nothing i can do without more information =)

Hope it helps!