Julia network expression Flux

Question

Learning Julia metaprogramming to automatically construct ANN by expression. While everything works well for inference, a backward stage raises an error:

LoadError: Can't differentiate foreigncall expression

The next code shows where the problem appears. While eval(net(x)) works well, for some reason it throws an error at the gradient computation stage.

# define ANN by expression
net(x) = :($w2 * relu.($w1 * $x .+ $b1) .+ $b2)

# define loss and network evaluation
loss(x, y) = Flux.logitcrossentropy(eval(net(x)), y)

θ = Flux.Params([w1, b1, w2, b2])

# eval network and calculate gradients
gs = gradient(() -> loss(features, labels), θ)  # where the problem appears

phipsgabler phipsgabler · Accepted Answer · 2020-06-05T15:06:59

eval is a primitive which Flux can't differentiate. You'd rather use something like

net = :($w2 * relu.($w1 * x .+ $b1) .+ $b2)
@eval loss(x, y) = Flux.logitcrossentropy($net, y)

That builds the expression of loss and evaluates it once, instead of evaluating (and hence, compiling) the same piece of code every time the loss function is called.

But think twice before metaprogramming. Before resorting to eval, try to write a macro that generates your network and the loss function.

Of course this does not work if the goal is to plug together expression only available at runtime. Genetic programming, where you create a lot of random expressions at runtime, sounds like a justifiable exception. But even then, there maybe are alternatives, like using your own restricted representations (which, for example, you could first symbolically differnetiate, and then "compile" to an anonymous function).

Julia network expression Flux

1 Answers