How do you compile macros in a Lisp compiler?

Question

In a Lisp interpreter, there can easily be a branch in eval that can expand a macro, and in the process of expanding it, call functions to build up the expanded exp

Answer 1

You found one of the main differences between Lisp and other languages.

In Lisp the execution of dynamically created code is essential, and for example necessary for macro expansion.

While writing a lisp to C compiler I discovered this now obvious thing myself and come to the conclusion that If you want to write a Lisp compiler there are only two solutions:

1. You write BOTH a compiler and an interpreter so that you can call the interpreter for macro expansion during compilation.

2. You must be able to dynamically compile code and call it (or using worse "tricks" like compiling a dynamically loadable module and then loading it).

If you are working on a compiler for C one possibility is to use Fabrice Bellard's TCC library that allows direct compilation of C code to a memory buffer.

I'm writing a Lisp to Javascript compiler and in this case there is of course no problem because "the hardware" can handle that nicely and you can ask Javascript to evaluate for example a string "function(...){...}" and then call the resulting object. Using Javascript also solves what is IMO one of the most difficult issues for a Lisp kernel that is proper implementation of lexical closures.

Indeed in my javascript compiler eval is just more or less

(defun eval (x)
    (funcall (js-eval (js-compile x))))

where js-compile is the main compiler interface and given a lisp form will give back a string containing javascript code that when evaluated (with the eval of javascript that I exported to the lisp level as js-eval) executes the code. Interestingly enough also eval is never used (with the only non essential exception of a convenience macro in which I've to execute user defined code during macro expansion).

One important thing to consider is that while Common Lisp has sort of separation between "read time", "compile time" and "run time" still this separation is more logical than physical as the running code is always Lisp. Compiling in Lisp is just calling a function. Even the "parsing" phase is just a regular lisp function executing... it's Lisp all the way down :-)

Links to my Lisp → Js toy compiler

• Source code
• Video demo 1
• Video demo 2
• Website