How to define (defmacro) a macro inside a let binding in sbcl common lisp?

Question

I am using SBCL Common Lisp. I am not an expert, but I like to think I understand it well enough to muddle along. However, I have recently encountered a strange problem with defmacro.

Why does the following code not compile, and how do I change it to make it compile?

(let ((a nil))
  (defmacro testmacro ())
  (testmacro))

The error is:

Unhandled UNDEFINED-FUNCTION in thread #<SB-THREAD:THREAD "main thread" RUNNING
                                          {100399C9A3}>:
  The function COMMON-LISP-USER::TESTMACRO is undefined.

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {100399C9A3}>
0: ((LAMBDA NIL :IN SB-DEBUG::FUNCALL-WITH-DEBUG-IO-SYNTAX))
1: (SB-IMPL::CALL-WITH-SANE-IO-SYNTAX #<CLOSURE (LAMBDA NIL :IN SB-DEBUG::FUNCALL-WITH-DEBUG-IO-SYNTAX) {1003A0EA8B}>)
2: (SB-IMPL::%WITH-STANDARD-IO-SYNTAX #<CLOSURE (LAMBDA NIL :IN SB-DEBUG::FUNCALL-WITH-DEBUG-IO-SYNTAX) {1003A0EA5B}>)
3: (PRINT-BACKTRACE :STREAM #<SB-SYS:FD-STREAM for "standard error" {10039A22B3}> :START 0 :FROM :INTERRUPTED-FRAME :COUNT NIL :PRINT-THREAD T :PRINT-FRAME-SOURCE NIL :METHOD-FRAME-STYLE NIL)
4: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<UNDEFINED-FUNCTION TESTMACRO {1003A0C193}> #<unavailable argument>)
5: (SB-DEBUG::RUN-HOOK *INVOKE-DEBUGGER-HOOK* #<UNDEFINED-FUNCTION TESTMACRO {1003A0C193}>)
6: (INVOKE-DEBUGGER #<UNDEFINED-FUNCTION TESTMACRO {1003A0C193}>)
7: (ERROR UNDEFINED-FUNCTION :NAME TESTMACRO)
8: ((LAMBDA (&REST SB-C::ARGS) :IN SB-C::INSTALL-GUARD-FUNCTION))
9: (SB-INT:SIMPLE-EVAL-IN-LEXENV (LET ((A NIL)) (DEFMACRO TESTMACRO NIL) (TESTMACRO)) #<NULL-LEXENV>)
10: (EVAL-TLF (LET ((A NIL)) (DEFMACRO TESTMACRO NIL) (TESTMACRO)) 0 NIL)
11: ((LABELS SB-FASL::EVAL-FORM :IN SB-INT:LOAD-AS-SOURCE) (LET ((A NIL)) (DEFMACRO TESTMACRO NIL) (TESTMACRO)) 0)
12: ((LAMBDA (SB-KERNEL:FORM &KEY :CURRENT-INDEX &ALLOW-OTHER-KEYS) :IN SB-INT:LOAD-AS-SOURCE) (LET ((A NIL)) (DEFMACRO TESTMACRO NIL) (TESTMACRO)) :CURRENT-INDEX 0)
13: (SB-C::%DO-FORMS-FROM-INFO #<CLOSURE (LAMBDA (SB-KERNEL:FORM &KEY :CURRENT-INDEX &ALLOW-OTHER-KEYS) :IN SB-INT:LOAD-AS-SOURCE) {10039B19FB}> #<SB-C::SOURCE-INFO {10039B19B3}> SB-C::INPUT-ERROR-IN-LOAD)
14: (SB-INT:LOAD-AS-SOURCE #<SB-SYS:FD-STREAM for "file /mnt/nas-data/Documents/Projects/SHARED PROJECTS - EVAN/pkcmd-lx/temp.lisp" {10039A5103}> :VERBOSE NIL :PRINT NIL :CONTEXT "loading")
15: ((FLET SB-FASL::LOAD-STREAM :IN LOAD) #<SB-SYS:FD-STREAM for "file /mnt/nas-data/Documents/Projects/SHARED PROJECTS - EVAN/pkcmd-lx/temp.lisp" {10039A5103}> NIL)
16: (LOAD #<SB-SYS:FD-STREAM for "file /mnt/nas-data/Documents/Projects/SHARED PROJECTS - EVAN/pkcmd-lx/temp.lisp" {10039A5103}> :VERBOSE NIL :PRINT NIL :IF-DOES-NOT-EXIST T :EXTERNAL-FORMAT :DEFAULT)
17: ((FLET SB-IMPL::LOAD-SCRIPT :IN SB-IMPL::PROCESS-SCRIPT) #<SB-SYS:FD-STREAM for "file /mnt/nas-data/Documents/Projects/SHARED PROJECTS - EVAN/pkcmd-lx/temp.lisp" {10039A5103}>)
18: ((FLET #:WITHOUT-INTERRUPTS-BODY-146 :IN SB-IMPL::PROCESS-SCRIPT))
19: (SB-IMPL::PROCESS-SCRIPT "temp.lisp")
20: (SB-IMPL::TOPLEVEL-INIT)
21: ((FLET #:WITHOUT-INTERRUPTS-BODY-82 :IN SAVE-LISP-AND-DIE))
22: ((LABELS SB-IMPL::RESTART-LISP :IN SAVE-LISP-AND-DIE))

The obvious answer is, of course, to put the defmacro outside of the let binding. But, for technical reasons, this would be very inconvenient for my project. Besides, I know of no valid reason why defining a macro under a let binding should fail.

Is 'let over macro' specifically prohibited in Common Lisp? Or am I missing something?

EDIT: The crux of my requirement is that the macro shares the same level as the function it calls, and subsequent code is not nested within it.

This is because I am attempting to write a macro that generates a function and a macro at the same time. The generated macro would call the function. So we end up with something like the following.

I write a line like:

(generate-stuff function-name)

And this resolves into:

(defun function-name-1 () ...)
(defmacro function-name (&rest args)
   `(function-name-1 ,@args)

Therefore the macro and the function it calls must be at the same lexical level, adjacent to each other, and cannot create a new lexical environment (macrolet) for subsequent code to nest within.

This would all work fine, except that I happen to be within a let binding at the time; because the function in question has to refer to variables within this binding.

Note that the macro is accessible from outside the binding:

(let ...)
   (defmacro my-macro ...)

(my-macro)

It seems absurd to me that a macro defined inside a let binding should only be accessible after the binding has ended. Nothing else in lisp behaves this way.

Answer 1

Local macros can be defined with MACROLET.

Besides, I know of no valid reason why defining a macro under a let binding should fail.

It doesn't fail. It's just not available during compile time, if you just compile the file or compile the expression. The compiler does not make defmacro definitions available in the compile-time environment, if the definition is not at top-level. Inside a progn it would be at top-level, but not inside a let.

Remember: SBCL compiles Lisp source code to machine code using a compiler. It executes machine code.

Let's look at your example:

(let ((a nil))
  (defmacro testmacro ())
  (testmacro))

Generally putting a global macro into a LET is bad practice and it's hard to understand what it actually should do. What should the influence of the binding of a be? Remember, a compiler compiles code before it executes.

Let's assume we have SBCL and SBCL loads a file with the above form:

(load "foo.lisp")

SBCL does now: READ the whole form, COMPILE the whole form, EXECUTE the whole form. In that order.

SBCL reads the first form. Which is the equivalent of:

CL-USER 155 > (read-from-string "(let ((a nil))
                                   (defmacro testmacro ())
                                   (testmacro))")
(LET ((A NIL)) (DEFMACRO TESTMACRO NIL) (TESTMACRO))

Thus we have data.

The next step is that SBCL compiles that code.

• it sees a LET expression. It creates code for the LET bindings. It does not execute the LET. We are just compiling it.
• it then compiles the body of the LET
• it sees a DEFMACRO form: the macro gets expanded and it gets compiled to machine code. It does not execute the DEFMACRO for. We are just expanding and compiling it.
• it sees a TESTMACRO form: It has no idea what it is. It compiles it as a function call to an undefined function and warns about an undefined function

The next step is that SBCL executes the compiled code

• it runs the compiled LET form and creates the binding for a
• it then executes the body of the LET
• it executes the compiled DEFMACRO form: a global macro named TESTMACRO gets defined
• it the executes the compiled TESTMACRO form: the function gets called. It is not defined. An error is signalled.

That's all logical and in no way absurd. SBCL first reads the LET, then compiles the LET to machine code and then runs the machine code.

These are three independent steps: read-time, compile-time, run-time.

Let's try it in the REPL:

First we are READing the form

* (read-from-string "(let ((a nil))
                       (defmacro testmacro ())
                       (testmacro))")
(LET ((A NIL))
  (DEFMACRO TESTMACRO ())
  (TESTMACRO))
129

Then we are compiling the form:

* (compile nil `(lambda () ,*))
; in: LAMBDA ()
;     (LET ((A NIL))
;       (DEFMACRO TESTMACRO ())
;       (TESTMACRO))
; 
; caught STYLE-WARNING:
;   The variable A is defined but never used.
; in: LAMBDA ()
;     (TESTMACRO)
; 
; caught STYLE-WARNING:
;   undefined function: COMMON-LISP-USER::TESTMACRO
; 
; compilation unit finished
;   Undefined function:
;     TESTMACRO
;   caught 2 STYLE-WARNING conditions
#<FUNCTION (LAMBDA ()) {226B025B}>
T
NIL

You can see that SBCL tells us about some problems. TESTMACRO is undefined.

Now we run the code:

* (funcall *)
STYLE-WARNING:
   TESTMACRO is being redefined as a macro when it was previously assumed to be a function.

debugger invoked on a UNDEFINED-FUNCTION in thread
#<THREAD "main thread" RUNNING {10004F84C3}>:
  The function COMMON-LISP-USER::TESTMACRO is undefined.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [CONTINUE      ] Retry calling TESTMACRO.
  1: [USE-VALUE     ] Call specified function.
  2: [RETURN-VALUE  ] Return specified values.
  3: [RETURN-NOTHING] Return zero values.
  4: [ABORT         ] Exit debugger, returning to top level.

("undefined function")
0] 

As expected - the compiler did warn us: the function TESTMACRO is undefined.

If you want SBCL to compile a macro form, you have to make sure that the macro form is known at compile-time.