what will the Finalizer thread do if there is a infinite loop or deadlock in the Java finalize method

Question

what will the Finalizer thread do if there is a infinite loop or deadlock in the Java finalize method.

Answer 1

I would say that since the Java Specification doesn't tell how the finalize method must be invoked (just that it must be invoked, before the object is garbage collected), the behaviour is implementation specific.

The spec doesn't rule out having multiple threads running the process, but doesn't require it:

It is important to note that many finalizer threads may be active (this is sometimes needed on large shared memory multiprocessors), and that if a large connected data structure becomes garbage, all of the finalize methods for every object in that data structure could be invoked at the same time, each finalizer invocation running in a different thread.

Looking at the sources of the JDK7, the FinalizerThread keeps the queue of objects scheduled for finalization (actually objects are added to the queue by the GC, when proven to be unreachable - check ReferenceQueue doc):

private static class FinalizerThread extends Thread {
    private volatile boolean running;
    FinalizerThread(ThreadGroup g) {
        super(g, "Finalizer");
    }
    public void run() {
        if (running)
            return;
        running = true;
        for (;;) {
            try {
                Finalizer f = (Finalizer)queue.remove();
                f.runFinalizer();
            } catch (InterruptedException x) {
                continue;
            }
        }
    }
}

Each object is removed from the queue, and runFinalizer method is run on it. Check is done if the finalization had run on the object, and if not it is being invoked, as a call to a native method invokeFinalizeMethod. The method simply is calling the finalize method on the object:

JNIEXPORT void JNICALL
Java_java_lang_ref_Finalizer_invokeFinalizeMethod(JNIEnv *env, jclass clazz,
                                                  jobject ob)
{
    jclass cls;
    jmethodID mid;

    cls = (*env)->GetObjectClass(env, ob);
    if (cls == NULL) return;
    mid = (*env)->GetMethodID(env, cls, "finalize", "()V");
    if (mid == NULL) return;
    (*env)->CallVoidMethod(env, ob, mid);
}

This should lead to a situation, where the objects get queued in the list, while the FinalizerThread is blocked on the faulty object, which in turn should lead to OutOfMemoryError.

So to answer the original question:

what will the Finalizer thread do if there is a infinite loop or deadlock in the Java finalize method.

It will simply sit there and run that infinite loop until OutOfMemoryError.

public class FinalizeLoop {
    public static void main(String[] args) {
        Thread thread = new Thread() {
            @Override
            public void run() {
                for (;;) {
                    new FinalizeLoop();
                }
            }
        };
        thread.setDaemon(true);
        thread.start();
        while (true);
    }

    @Override
    protected void finalize() throws Throwable {
        super.finalize();
        System.out.println("Finalize called");
        while (true);

    }
}

Note the "Finalize called" if printed only once on the JDK6 and JDK7.