Getting the whole AST of the file / complex code

Question

Julia manual states:

Every Julia program starts life as a string:
julia> prog = "1 + 1"
"1 + 1"

I can easily get the AST of the simple expression, or even a function with the help of quote / code_*, or using Meta.parse / Meta.show_sexpr if I have the expression in a string.

The question: Is there any way to get the whole AST of the codepiece, possibly including several atomic expressions? Like, read the source file and convert it to AST?

What exactly do you mean by "whole AST"? :(1 + 1) is as whole as it gets. Do you mean to recursively expand + to a sub-AST? (If yes, that doesn't really work). — phipsgabler
@phipsgabler Yes, by “whole” I mean grab the random file from the internets and dump it’s AST. Could you point me to some ...uhmm resource giving a hint on why it does not really work? — Aleksei Matiushkin
If you're talking about files and the internet, I think we are speaking about different things. What I meant is, you can't go from 1 + 1 to Core.add_int(1, 1) via ASTs, because ASTs exist before type inference, and only after type inference you know which method of + will actually be called. — phipsgabler
@phipsgabler That I understand. I am not going to do anything with the resulting AST. I just need to dump it, that’s it. I could not find any way to dump a complex Julia source, not the simple expression like "1 + 1". — Aleksei Matiushkin
Ah, do you want to call something parse on a whole source file, containing more than one single expression? — phipsgabler

Mason Mason · Accepted Answer · 2019-11-17T16:20:55

If you want to do this from Julia instead of FemtoLisp, you can do

function parse_file(path::AbstractString)
    code = read(path, String)
    Meta.parse("begin $code end")
end

This takes in a file path, reads it and parses it to a big expression that can be evaluated.

This comes from @NHDaly's answer, here: https://stackoverflow.com/a/54317201/751061

If you already have your file as a string and don’t want to have to read it again, you can instead do

parse_all(code::AbstractString) = Meta.parse("begin $code end")

It was pointed out on Slack by Nathan Daly and Taine Zhao that this code won't work for modules:

julia> eval(parse_all("module M x = 1 end"))
ERROR: syntax: "module" expression not at top level
Stacktrace:
 [1] top-level scope at REPL[50]:1
 [2] eval at ./boot.jl:331 [inlined]
 [3] eval(::Expr) at ./client.jl:449
 [4] |>(::Expr, ::typeof(eval)) at ./operators.jl:823
 [5] top-level scope at REPL[50]:1

This can be fixed as follows:

julia> eval_all(ex::Expr) = ex.head == :block ? for e in ex eval_all(e) end : eval(e);

julia> eval_all(ex::Expr) = ex.head == :block ? eval.(ex.args) : eval(e);

julia> eval_all(parse_all("module M x = 1 end"));

julia> M.x
1

Since the question asker is not convinced that the above code produces a tree, here is a graph representation of the output of parse_all, clearly showing a tree structure.

In case you're curious, those leaves labelled #= none:1 =# are line number nodes, indicating the line on which each following expression takes place.

As suggested in the comments, one can also apply Meta.show_sexpr to an Expr object to get a more "lispy" representation of the AST without all the pretty printing julia does by default:

julia> (Meta.show_sexpr ∘ Meta.parse)("begin x = 1\n y = 2\n z = √(x^2 + y^2)\n end")
(:block,
  :(#= none:1 =#),
  (:(=), :x, 1),
  :(#= none:2 =#),
  (:(=), :y, 2),
  :(#= none:3 =#),
  (:(=), :z, (:call, :√, (:call, :+, (:call, :^, :x, 2), (:call, :^, :y, 2))))
)

Getting the whole AST of the file / complex code

2 Answers