Is there way to tell whether macroexpansion will or will not see lexical context?

Question

When performing row-wise operations in a table (implemented as, say, list of lists), it is convenient to refer to cells by their respective column names. I decided to write a shortcut, so that I don't have to write

(elt row 5)

and so on, and could write

(:col "Relevant Header")

instead. Not having much experience with Common Lisp, I wrote a macro

(defmacro with-named-columns! (sequence-of-rows-symbol body &key headers)
  (labels
      ((replace-col (form headers row-symbol)
         (if (listp form)
             (if (eql (car form) :col)
                 `(elt ,row-symbol ,(position (cadr form) headers :test #'equal))
                 (mapcar #'(lambda (x) (replace-col x headers row-symbol)) form))
             form)))
    (let ((row (gensym "ROW")))
      `(map 'list
            (lambda (,row) ,(replace-col body headers row))
            ,sequence-of-rows-symbol))))

that, albeit imperfect, seems to function well enough:

(defparameter *table* '(("h1" "h2") ("foo" "bar") ("" "baz") ("foo" "qux")))

(defparameter *sequence-of-rows* (cdr *table*))

(with-named-columns! *sequence-of-rows*
   (when (equal (:col "h1") "")
     'do-nothing-but-make-a-note)
   :headers ("h1" "h2"))

=> (NIL DO-NOTHING-BUT-MAKE-A-NOTE NIL)

It macroexpands the way I intended to:

(macroexpand-1
 '(with-named-columns! *sequence-of-rows*
   (when (equal (:col "h1") "")
     'do-nothing-but-make-a-note)
   :headers ("h1" "h2")))

=>
(MAP 'LIST
     (LAMBDA (#1=#:ROW724)
       (WHEN (EQUAL (ELT #1# 0) "") 'DO-NOTHING-BUT-MAKE-A-NOTE))
     *SEQUENCE-OF-ROWS*)

So far so good. However, headers are often attached to data, in the form of the first row. For such cases, it's natural to have a separate facility:

(defmacro process-rows (symbol-bound-to-table-with-exactly-one-header-row body)
  (let ((symbol-for-sequence-of-rows (gensym "SEQUENCE-OF-ROWS")))
    `(let ((,symbol-for-sequence-of-rows (rest ,symbol-bound-to-table-with-exactly-one-header-row)))
        (with-named-columns! ,symbol-for-sequence-of-rows
          ,body
          :headers ,(first (symbol-value symbol-bound-to-table-with-exactly-one-header-row))))))

Which, again, seems to work fine:

(process-rows *table*
    (when (equal (:col "h1") "")
      'do-nothing-but-make-a-note))

=> (NIL DO-NOTHING-BUT-MAKE-A-NOTE NIL)

Macroexpansion leads to a macro inside a let form, and there seems to be no problem with this

(macroexpand-1
 '(process-rows *table*
    (when (equal (:col "h1") "")
      'do-nothing-but-make-a-note)))

=>
(LET ((#1=#:SEQUENCE-OF-ROWS726 (REST *TABLE*)))
  (WITH-NAMED-COLUMNS! #1#
                       (WHEN (EQUAL (:COL "h1") "")
                         'DO-NOTHING-BUT-MAKE-A-NOTE)
                       :HEADERS ("h1" "h2")))

Side note: Hopefully, this macroexpansion demonstrates why I found it beneficial to provide explicit lists of headers to WITH-NAMED-COLUMNS! while hiding the rest rows in a symbol.

However,

(let ((table '(("h1" "h2") ("foo" "bar") ("" "baz") ("foo" "qux"))))
  (process-rows table
    (when (equal (:col "h1") "")
      'do-nothing-but-make-a-note)))

invokes a debugger (SBCL) with the message “The variable TABLE is unbound.” — during the macroexpansion of PROCESS-ROWS.

I don't yet understand Common Lisp evaluation process well enough. It seems PROCESS-ROWS can't see lexical variable, which issue I heard of. But my other macro

(let ((sequence-of-rows '(("foo" "bar") ("" "baz") ("foo" "qux"))))
  (with-named-columns! sequence-of-rows
   (when (equal (:col "h1") "")
     'do-nothing-but-make-a-note)
   :headers ("h1" "h2")))

=> (NIL DO-NOTHING-BUT-MAKE-A-NOTE NIL)

evaluates just fine in the body of LET. I don't see any difference between these two expansions. Is there a way to see? Also, I wrote a different macro (the one I intended to mostly use instead of writing LET forms in top level; as a matter of fact, it was the one macro where the issue manifested itself initially), which macroexpands to

(LET ((#1=#:TABLE727 (READ-CSV #2=#P"~/test.csv")))
  (WRITE-CSV
   (PROGN
    (PROCESS-ROWS #1#
                  (WHEN (EQUAL (:COL "h1") "")
                    'DO-NOTHING-BUT-MAKE-A-NOTE))
    #1#)
   :STREAM #2#))

which signals the same error on evaluation: “The variable #:TABLENNN is unbound.” I'm baffled as to how PROCESS-ROWS can't see the variable, since those #1= and #1# are direct references to the same place (much like in macroexpansion of PROCESS-ROWS). Apparently, this doesn't guarantee anything.

If there is a way to ensure lexical capture, how do I do it, and if not — at which point did I step into the land of unspecified behaviour? Or maybe I'm missing something simple?

Generally speaking, a short focused question will receive better answers than a long rambling one. You might want to consider splitting your question into several separate ones. — sds
I know. The question as posed in the title probably has one correct answer. I am interested in the explanation of discrepancy in two macroexpansions, rather than in solutions to the initial problem. If someone can write a pair of more concise forms that demonstrate the same behavior as the pair in question, all the better. (I can't yet.) — akater
I suggest that you try my approach (see the answer) yourself. — sds

sds sds · Accepted Answer · 2017-02-16T23:34:38

Your specific question

I think your problem is a relatively common one, and it is usually solved by making your "low-level macro" (with-named-columns!) into a function which returns the macro-expansion when calling it explicitly from the "high-level macro" (process-rows). Cf. ensure-generic-function vs defgeneric.

Your actual problem

I would like to suggest that you try using macrolet or flet and actually defining local functions or macros (e.g., (h1)) instead of your not-very-lispy syntax (:col "h1").

Is there way to tell whether macroexpansion will or will not see lexical context?

2 Answers

Your specific question

Your actual problem