How does clojure class reloading work?

Question

I've been reading code and documentation to try to understand how class reloading works in clojure. According to many websites, such as http://tutorials.jenkov.com/java-reflection/dynamic-class-loading-reloading.html , whenever you load a class essentially you obtain the bytecode (via any data mechanism), convert the bytecode into an instance of class Class (via defineClass), and then resolve (link) the class via resolveClass. (Does defineClass implicitly call resolveClass?). Any given classloader is only allowed to link a class once. If it attempts to link an existing class, it does nothing. This creates a problem since you cannot link a newly instantiated class, therefore you have to create a new instance of a classloader everytime you reload a class.

Going back to clojure, I tried examining the paths to load classes.

In clojure, you can define new classes in multiple ways depending on what you want:

Anonymous Class: reify proxy

Named Class: deftype defrecord (which uses deftype under the hood) gen-class

Ultimately, those codes point to clojure/src/jvm/clojure/lang/DynamicClassLoader.java

where DynamicClassLoader/defineClass creates an instance with super's defineClass and then caches the instance. When you want to retrieve the class, clojure load with a call to forName which calls the classloader and DynamicClassLoader/findClass, which first looks in the cache before delegating to the super class (which is contrary to the way most normal classloaders work, where they delegate first, than try it themselves.) The important point of confusion is the following: forName is documented to link the class before it returns but this would imply you can not reload a class from the existing DynamicClassLoader and instead need to create a new DynamicClassLoader, however I don't see this in the code. I understand that proxy and reify define anonymous classes, so their names are different thus can be treated as if its a different class. However, for the named classes, this breaks down. In real clojure code, you can have references to the old version of the classes and references to the new version of the classes simultaneously, but attempts to create new class instances will be of the new version.

Please explain how clojure is able to reload classes without creating new instances of DynamicClassLoader, if I can understand the mechanism to reload classes, I would like to extend this reloading functionality to java's .class files I may create using javac.

Notes: This question refers to class RELOADING, not simply dynamic loading. Reloading means that I have already interned a class but want to intern a new updated version of that instance.

I want to reiterate, that its not clear how clojure is able to reload deftype defined classes. Calling deftype eventually leads to a call to clojure.lang.DynamicClassLoader/defineClass. Doing this again leads another call to defineClass, but doing this manually results in a Linkage Error. What is happening underneath here that allows clojure to do this with deftypes?

Answer 1

Not all of these language features use the same technique.

proxy

The proxy macro generates a class name based exclusively on the class and list of interfaces being inherited. The implementation of each method in this class delegates to a Clojure fn stored in the object instance. This allows Clojure to use the very same proxy class every time the same list of interfaces is inherited, whether the body of the macro is the same or not. No actual class reloading takes place.

reify

For reify, the method bodies are compiled directly into the class, so the trick proxy uses won't work. Instead, a new class is generated when the form is compiled, so if you change the body of the form and reload it, you get a whole new class (with a new generated name). So again, no actual class reloading takes place.

gen-class

With gen-class you specify a name for the generated class, so neither of the techniques used for proxy or reify will work. A gen-class macro contains only a sort of spec for a class, but none of the method bodies. The generated class, somewhat like proxy, defers to Clojure functions for the method bodies. But because a name is tied to the spec, unlike proxy it would not work to change the body of a gen-class and reload it, so gen-class is only available when compiling ahead-of-time (AOT compilation) and no reloading is allowed without restarting the JVM.

deftype and defrecord

This is where real dynamic class reloading happens. I'm not deeply familiar with the internals of the JVM, but a little work with a debugger and the REPL makes one point clear: every time a class name needs to be resolved, such as when compiling code that uses the class or when the Class class's forName method is called, Clojure's DynamicClassLoader/findClass method is used. As you note this looks up the class name in in the DynamicClassLoader's cache, and this can be set to point to a new class by running deftype again.

Note the caveats in the tutorial you mentioned about the reloaded class being a different class, despite having the same name, still apply to Clojure classes:

(deftype T [a b])  ; define an original class named T
(def x (T. 1 2))   ; create an instance of the original class
(deftype T [a b])  ; load a new class by the same name
(cast T x)         ; cast the old instance to the new class -- fails
; ClassCastException   java.lang.Class.cast (Class.java:2990)

Each top-level form in a Clojure program gets a fresh DynamicClassLoader which is used for any new classes defined within that form. This will include not only classes defined via deftype and defrecord but also reify and fn. This means that the classloader for x above is different than the new T. Note the numbers after the @s are different -- each gets its own classloader:

(.getClassLoader (class x))
;=> #<DynamicClassLoader clojure.lang.DynamicClassLoader@337b4703>

(.getClassLoader (class (T. 3 4)))
;=> #<DynamicClassLoader clojure.lang.DynamicClassLoader@451c0d60>

But as long as we don't define a new T class, new instances will have the same class with the same classloader. Note the number after the @ here is the same as the second one above:

(.getClassLoader (class (T. 4 5)))
;=> #<DynamicClassLoader clojure.lang.DynamicClassLoader@451c0d60>