Surprising behaviour related to records, protocols and compilation

Question

I encountered a, to me, slightly surprising behaviour seemingly related to clojure records.

The setup is as follows:

1. One namespace defines a record type:

   (ns defrecordissue.arecord)
   
   (defrecord ARecord [])
   

2. Another namespace defines a protocol, and extends it to the record type defined in 1:

   (ns defrecordissue.aprotocol
     (:require [defrecordissue.arecord])
     (:import [defrecordissue.arecord ARecord]))
   
   (defprotocol AProtocol
     (afn [this]))
   
   (extend-protocol AProtocol
     ARecord
     (afn [this] 42))
   

3. A third namespace constructs an instance of the record and invokes the protocol function on the record.

   (ns defrecordissue.aot1
     (:require [defrecordissue.aprotocol]
               [defrecordissue.arecord]))
   
   (defrecordissue.aprotocol/afn (defrecordissue.arecord/->ARecord))
   

When the defrecordissue.aot1 namespace is compiled, in my case using lein compile defrecordissue.aot1, compilation fails with the following exception:

Exception in thread "main" java.lang.IllegalArgumentException: No implementation of method: :afn of protocol: #'defrecordissue.aprotocol/AProtocol found for class: defrecordissue.arecord.ARecord, compiling:(aot1.clj:5:1)
    at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3463)
    at clojure.lang.Compiler.compile1(Compiler.java:7153)
    at clojure.lang.Compiler.compile(Compiler.java:7219)
    at clojure.lang.RT.compile(RT.java:398)
    at clojure.lang.RT.load(RT.java:438)
    at clojure.lang.RT.load(RT.java:411)
    at clojure.core$load$fn__5018.invoke(core.clj:5530)
    at clojure.core$load.doInvoke(core.clj:5529)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invoke(core.clj:5336)
    at clojure.core$compile$fn__5023.invoke(core.clj:5541)
    at clojure.core$compile.invoke(core.clj:5540)
    at user$eval7.invoke(NO_SOURCE_FILE:1)
    at clojure.lang.Compiler.eval(Compiler.java:6619)
    at clojure.lang.Compiler.eval(Compiler.java:6609)
    at clojure.lang.Compiler.eval(Compiler.java:6582)
    at clojure.core$eval.invoke(core.clj:2852)
    at clojure.main$eval_opt.invoke(main.clj:308)
    at clojure.main$initialize.invoke(main.clj:327)
    at clojure.main$null_opt.invoke(main.clj:362)
    at clojure.main$main.doInvoke(main.clj:440)
    at clojure.lang.RestFn.invoke(RestFn.java:421)
    at clojure.lang.Var.invoke(Var.java:419)
    at clojure.lang.AFn.applyToHelper(AFn.java:163)
    at clojure.lang.Var.applyTo(Var.java:532)
    at clojure.main.main(main.java:37)
Caused by: java.lang.IllegalArgumentException: No implementation of method: :afn of protocol: #'defrecordissue.aprotocol/AProtocol found for class: defrecordissue.arecord.ARecord
    at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:541)
    at defrecordissue.aprotocol$fn__40$G__35__45.invoke(aprotocol.clj:5)
    at clojure.lang.AFn.applyToHelper(AFn.java:161)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3458)
    ... 25 more 

If I change 3) to construct the record class directly, like so:

(ns defrecordissue.aot2
  (:require [defrecordissue.aprotocol]
            [defrecordissue.arecord]))

(defrecordissue.aprotocol/afn (defrecordissue.arecord.ARecord.))

Compilation succeeds.

My suspicion is that this is somehow related to http://dev.clojure.org/jira/browse/CLJ-371, but I don't understand exactly what is happening.

I should also add that without the lein clean, compilation succeeds the second time, since a class for the record is now available on the classpath. Therefore, I can get around this problem by AOT-compiling the namespace defining the record type.

I created a simple leiningen project on GitHub that illustrates the issue, see README for usage: https://github.com/ragnard/defrecordissue

Why am I seeing this behaviour, and what is the correct way to avoid it?

UPDATE

I added a new branch to the GitHub repo better illustrating the core issue: https://github.com/ragnard/defrecordissue/tree/more-realistic/

The problem occurs regardless of where (ie. in which namespace) the record instance is constructed.

Answer 1

I can reproduce the problem with your repo. Here's three solutions which work for me:

1. Tell lein compile to compile more namespaces:

   lein compile defrecordissue.aprotocol defrecordissue.arecord defrecordissue.aot1
   

2. Put

   :aot [defrecordissue.aprotocol defrecordissue.arecord defrecordissue.aot1]
   

   in project.clj.

3. Put

   :aot :all
   

   in project.clj.

The latter two make lein compile do the work of lein aot1 (in the case of 2.) and both lein aot1 and lein aot2 (in the case of 3.).

Answer 2

I run into this all the time. This is what I've come up with and it seems to work:

(defmacro with-datatype
  [datatype & body]
  (let [last-dot (.lastIndexOf ^String (str datatype) ".")
        ns (-> datatype
               str
               (subs 0 last-dot)
               symbol)]
     `(do
        (require (quote ~ns))
        (import ~datatype)
        ~@body)))

(defmacro extend-type*
  [datatype & extensions]
  `(with-datatype ~datatype
     (extend-type ~datatype
       ~@extensions)))

(defmacro extend-protocol*
  [protocol & specs]
  (let [splitter (let [l (atom nil)]
                   #(if (symbol? %) (reset! l %) @l))
        specs (partition-by splitter specs)]
    `(do
       ~@(for [[datatype & extensions] specs]
           `(extend-type* ~datatype ~protocol ~@extensions)))))

Then you only need to change your second code block to:

(ns defrecordissue.aprotocol)

(defprotocol AProtocol
  (afn [this]))

(extend-protocol* AProtocol
  defrecordissue.arecord.ARecord
  (afn [this] 42))

Maybe not the cleanest solution, but you don't have to AOT your libraries any more.