Julia macro expansion order

Question

I would like to write a macro @unpack t which takes an object t and copies all its fields into local scope. For example, given

immutable Foo
    i::Int
    x::Float64
end
foo = Foo(42,pi)

the expression @unpack foo should expand into

i = foo.i
x = foo.x

Unfortunately, such a macro cannot exist since it would have to know the type of the passed object. To circumvent this limitation, I introduce a type-specific macro @unpackFoo foo with the same effect, but since I'm lazy I want the compiler to write @unpackFoo for me. So I change the type definition to

@unpackable immutable Foo
    i::Int
    x::Float64
end

which should expand into

immutable Foo
    i::Int
    x::Float64
end
macro unpackFoo(t)
    return esc(quote
        i = $t.i
        x = $t.x
    end)    
end

Writing @unpackable is not too hard:

macro unpackable(expr)
    if expr.head != :type
        error("@unpackable must be applied on a type definition")
    end

    name = isa(expr.args[2], Expr) ? expr.args[2].args[1] : expr.args[2]
    fields = Symbol[]
    for bodyexpr in expr.args[3].args
        if isa(bodyexpr,Expr) && bodyexpr.head == :(::)
            push!(fields,bodyexpr.args[1])
        elseif isa(bodyexpr,Symbol)
            push!(fields,bodyexpr)
        end
    end

    return esc(quote
        $expr
        macro $(symbol("unpack"*string(name)))(t)
            return esc(Expr(:block, [:($f = $t.$f) for f in $fields]...))
        end
    end)
end

In the REPL, this definition works just fine:

julia> @unpackable immutable Foo
           i::Int
           x::Float64
       end

julia> macroexpand(:(@unpackFoo foo))
quote 
    i = foo.i
    x = foo.x
end

Problems arise if I put the @unpackFoo in the same compilation unit as the @unpackable:

julia> @eval begin
       @unpackable immutable Foo
           i::Int
           x::Float64
       end
       foo = Foo(42,pi)
       @unpackFoo foo
       end
ERROR: UndefVarError: @unpackFoo not defined

I assume the problem is that the compiler tries to proceed as follows

Expand @unpackable but do not parse it.
Try to expand @unpackFoo which fails because the expansion of @unpackable has not been parsed yet.
If we wouldn't fail already at step 2, the compiler would now parse the expansion of @unpackable.

This circumstance prevents @unpackable from being used in a source file. Is there any way of telling the compiler to swap steps 2. and 3. in the above list?

The background to this question is that I'm working on an iterator-based implementation of iterative solvers in the spirit of https://gist.github.com/jiahao/9240888. Algorithms like MinRes require quite a number of variables in the corresponding state object (8 currently), and I neither want to write state.variable every time I use a variable in e.g. the next() function, nor do I want to copy all of them manually as this bloats up the code and is hard to maintain. In the end, this is mainly an exercise in meta-programming though.

I think you can do this much more simply using generated functions. But what is the use case? — David P. Sanders
I don't think generated functions will work since functions introduce a new variable scope and can only operate within that scope. I'll add some remarks about the background to the question. — gTcV
Actually, now that I think of it it would probably be more reasonable in my use case to put away with the state type and use tuples instead, for which I can get the @unpack behaviour by tuple unpacking (i.e. i,x = (42,pi)). — gTcV
@gTcV Unpacking in Julia is done by the iteration protocol, so you can attain similar behaviour by implementing start, next, and done. See docs.julialang.org/en/release-0.4/manual/interfaces — Fengyang Wang
Try having a look at Parameters package. It has an @unpack macro and the accompanying machinery similar to what you suggest you need. — Dan Getz

one-more-minute one-more-minute · Accepted Answer · 2016-07-29T20:55:44

Firstly, I would suggest writing this as:

immutable Foo
  ...
end

unpackable(Foo)

where unpackable is a function which takes the type, constructs the appropriate expression and evals it. There are a couple of advantages to this, e.g. that you can apply it to any type without it being fixed at definition time, and the fact that you don't have to do a bunch of parsing of the type declaration (you can just call fieldnames(Foo) == [:f, :i] and work with that).

Secondly, while I don't know your use case in detail (and dislike blanket rules) I will warn that this kind of thing is frowned upon. It makes code harder to read because it introduces a non-local dependency; suddenly, in order to know whether x is a local or global variable, you have to look up the definition of a type in a whole different file. A better, and more general, approach is to explicitly unpack variables, and this is available in MacroTools.jl via the @destruct macro:

@destruct _.(x, i) = myfoo
# now we can use x and i

(You can destruct nested data structures and indexable objects too, which is nice.)

To answer your question: you're essentially right about how Julia runs code (s/parse/evaluate). The whole block is parsed, expanded and evaluated together, which means in your example you're trying to expand @unpackFoo before it's been defined.

However, when loading a .jl file, Julia evaluates blocks in the file one at a time, rather than all at once.

This means that you can happily write a file like this:

macro foo()
  :(println("hi"))
end

@foo()

and run julia foo.jl or include("foo.jl") and it will run fine. You just can't have a macro definition and its use in the same block, as in your begin block above.

Julia macro expansion order

2 Answers