How does Clojure compile the code that's run in a REPL?

Question

Honest noobie question. Based on Russ Olsen's Getting Clojure, I know the following:

(1) Clojure code is compiled to JVM bytecode before runtime.

(2) Clojure code can be run, with almost instant feedback, in a REPL, which uses the functions (read) and (eval) or something equivalent.

So, it seems that the compilation of Clojure code to JVM bytecode must happen at some point during the REPL, presumably in either in the (read) stage or shortly thereafter.

But that's a fuzzy mental picture, and I want to clarify it.

For example, it'd be nice to know when in the REPL the code actually gets compiled, how the data created from compilation are stored in RAM and then accessed by (eval), and any significant steps that happen in between or thereafter.

In other words, I want to understand in more detail how the sausage is really made:

How does Clojure compile the code that's run in a REPL?

(Bonus points: how is this different from what Clojure does when it compiles code from a non-REPL source, say, a Leiningen project?)

Answer 1

The reader consumes characters and produces Clojure data structures (lists, vectors, symbols, etc). The read phase definitely does not know anything about JVM bytecode. That happens as part of the eval phase: the compiler consumes these data structures and produces JVM bytecode.

When running a REPL, that bytecode is stored in a DynamicClassLoader - all JVM classes must be defined by some ClassLoader, and DynamicClassLoader is the one that Clojure created to allow defining classes on the fly from Clojure data structures.

When compiling to classfiles, that same bytecode is simply written to disk in a .class file, and possibly packaged into a jar after that.

Answer 2

A short test reveals the answer:

(ns tst.demo.core
  (:use tupelo.core tupelo.test))

(defmacro with-trace
  [tag & forms]
  (println "*** running macro ***   tag:  " tag)
  `(do
     (println ~tag :enter)
     (let [result# (do ~@forms)]
       (println ~tag :leave)
       result#)))

(dotest
  (newline)
  (println :answer
    (with-trace :add-5
      (+ 2 3)))

  (newline)
  (println :using-eval
    (eval
      (quote (tst.demo.core/with-trace :the-ultimate-answer
               (clojure.core/inc 41))))))

which prints:

> lein clean ; lein test

*** running macro ***   tag:   :add-5

lein test _bootstrap

-------------------------------
   Clojure 1.10.1    Java 14
-------------------------------

lein test tst.demo.core

:add-5 :enter
:add-5 :leave
:answer 5

*** running macro ***   tag:   :the-ultimate-answer
:the-ultimate-answer :enter
:the-ultimate-answer :leave
:using-eval 42

So we see that eval does "evaluate" macros as well as functions.

---

Update

@amalloy is correct that I forgot to quote the code I sent into eval, so the macro with-trace was evaluated at compile time instead of run time. With this version the code sent into eval is quoted, so it is just a nested list of 2 symbols, 1 keyword, and an integer. Now, eval sees the macro, runs it to generate the actual code, then compiles & runs the macro output.