Creating a DSL using nested Elixir macros that can change block context

Question

Here's what I'm trying to do:

defmodule ArbitraryContext do
  use Cake

  def make_cake do
    cake do
      name "Chocolate"

      topping do
        name "Butter cream"
        sweetener "Agave"
      end
    end
  end
end

I'd like ArbitraryContext.make_cake/0 to produce nested structs along the lines of:

%Cake{
  name: "Chocolate",
  topping: %Topping{
    name: "Butter cream",
    sweetener: "Agave"
  }
}

I've read Metaprogramming Elixir and quite a few other resources but I can't seem to be able to reproduce some of the DSL flexibility I'm used to in Ruby in Elixir. That seems wrong, as Elixir seems fundamentally more flexible.

I've been following along with the "HTML DSL" example in Metaprogramming Elixir. The HTML example is fundamentally simpler because there is only one module at play - a Tag - so its context can stay the same throughout the nesting. In my case there could be dozens of contexts. I inject Cake macros into the context, which does successfully produce the %Cake{...} with a name, but when the block for Topping is unquoted to produce %Topping{...}, the context is still Cake. No matter what I do, I can't seem to find a clean way to run that block in a new context.

defmodule Cake do
  defstruct name: nil

  defmacro __using__(_) do
    quote do
      import Cake
    end
  end

  defmacro cake(do: block) do
    quote do
      # agent-y stuff to maintain state while building the cake. not
      # super important at this time
      {:ok, var!(pid, Cake)} = %Cake{} |> start_state

      # here's where the cake is no longer a lie and the name is set
      unquote(block)

      out = get_state(var!(pid, Cake))
      :ok = stop_state(var!(pid, Cake))
      out
    end
  end

  defmacro topping(block) do
    quote do
      # oh no! block gets evaluated here. even if I double quote
      # block, it still ultimately gets the Cake scope even though I'm
      # passing it into Topping, which is very similar to Cake... meant
      # to build up a Topping struct.
      # 
      # I want to:
      # 1) get block into Topping.topping without unquoting it
      # 2) have the block unquoted in Topping's context, once in there
      Topping.topping(unquote(block))
    end
  end
end

In Ruby I'd handle this with something like Topping.class_eval... you'd end up with name and sweetener from Topping and on the other side you'd end up with a new Topping class instance.

I can solve this, arguably much cleaner, by just building the structs pre-nested without the DSL and all the macros, but I want to understand how to get the intended result using Elixir macros.

I hope I've communicated this question well enough!

Answer 1

I believe you are trying to conquer the sea with a railway locomotive. Although it’s still possible to achieve what you want, it’s completely wrong from the perspective of being elixirish, whatever it means.

In the first place, there is no notion of “context.” At all. Everything you have are just plain old good functions. There are two contexts, if you insist on using “context” word: compilation and runtime.

Elixir macros are more like C/C++ macros, but written in the same language as the main code, which probably confused you. They are being executed during the compilation phase.

Macros return plain AST, that is to be embedded inplace as is.

That said, when you declare a macro:

defmacro cake(do: block), do: block

You end up having a beam (compiled code,) that has all the macros inlined. No matter, where they were declared. That’s it. You still might use macros to produce structs, of course, macros are still just plain AST:

iex> quote do: %{name: "cake", topping: %{name: "blah"}}
{:%{}, [], [name: "cake", topping: {:%{}, [], [name: "blah"]}]}

As soon, as your macro returns quoted representation of your struct, e.g. exactly what quote do will show for it, it will work. E.g.

iex> defmodule A do
...>   defmacro cake(toppling),
...>     do: {:%{}, [], [name: "cake", topping: {:%{}, [], [name: toppling]}]}
...>   def check, do: IO.inspect A.cake("CREAM")
...> end

{:module, A,
 <<70, 79, 82, ...>>, {:check, 0}}

iex> A.check
%{name: "cake", topping: %{name: "CREAM"}}

You might use this technique to achieve what you wanted, but it makes not much sense since the whole struct produced cannot be modified in the future. Terms are immutable, remember it.

Hope it clarifies things. Feel free to ask more questions, if you are still curious.

Answer 2

I got this working thanks to tips from @dogbert and @mudasobwa. As expected, it's gross and messy but it works:

Basic DSL:

defmodule ArbitraryContext do
  def make_cake do
    use Cake

    cake do
      name "Chocolate"

      topping do
        name "Butter cream"
        sweetener "Agave"
      end
    end
  end
end

Cake:

defmodule Cake do
  require Topping
  defstruct name: nil, topping: nil

  defmacro __using__(_) do
    quote do
      import Cake
    end
  end

  defmacro cake(do: block) do
    quote do
      {:ok, var!(pid, Cake)} = %Cake{} |> start_state

      unquote(block)

      out = get_state(var!(pid, Cake))
      :ok = stop_state(var!(pid, Cake))
      out
    end
  end

  defmacro topping(do: block) do
    topping = Macro.escape(
      Topping.topping(do: block)
    )

    quote do
      put_state(var!(pid, Cake), :topping, unquote(topping))
    end
  end

  defmacro name(val) do
    quote do
      put_state(var!(pid, Cake), :name, unquote(val))
    end
  end

  def start_state(state), do: Agent.start_link(fn -> state end)
  def stop_state(pid), do: Agent.stop(pid)
  def put_state(pid, key, content), do: Agent.update(pid, fn state -> Map.put(state, key, content) end)
  def get_state(pid), do: Agent.get(pid, &(&1))
end

Topping:

defmodule Topping do
  defstruct name: nil, sweetener: nil

  def topping(do: block) do
    {:ok, pid} = %Topping{} |> start_state

    Topping.run(pid, block)

    out = get_state(pid)
    :ok = stop_state(pid)
    out
  end

  def run(pid, {_block, _context, ast}) do
    Macro.postwalk(ast, fn segment ->
      run_call(pid, segment)
    end)
  end

  def run(pid, ast), do: ast

  def run_call(pid, {method, _context, args}) do
    apply(Topping, method, [pid] ++ args)
  end

  def run_call(pid, ast), do: ast

  def name(pid, val) do
    put_state(pid, :name, val)
  end

  def sweetener(pid, val) do
    put_state(pid, :sweetener, val)
  end

  def start_state(state), do: Agent.start_link(fn -> state end)
  def stop_state(pid), do: Agent.stop(pid)
  def put_state(pid, key, content), do: Agent.update(pid, fn state -> Map.put(state, key, content) end)
  def get_state(pid), do: Agent.get(pid, &(&1))
end

And finally:

iex(1)> ArbitraryContext.make_cake
%Cake{name: "Chocolate",
 topping: %Topping{name: "Butter cream", sweetener: "Agave"}}

---

As much as I enjoy the DSL, I don't think I'll ultimately use this approach.

A slightly more sensible approach that I also tried is to abandon the Agent business and go straight to parsing the AST statelessly. In the end the complexity was not worth it.