Julia-lang: Extending a method from an installed module, "using" order

Question

I'm trying to extend a method from the JSON.jl module, specifically I'm doing this (extending 'JSON.lower', as documented, to allow serialisation), generally - I'm asking what are the current Julia "best practices" for extending a method from another module - especially - if this is a method in-change of some of some policy of the module (as in this case in JSON.jl).

Is there actually a way to guarantee the updated method will be used by the imported module, except for patching it?

Let's demonstrate both desired and sinister behaviour:

    module TestProgram
    using JSON
    import JSON.lower            # not sure whether this is needed
    JSON.print(Set([1,2,1]))
    ## output:
    ## {"dict":{"2":null,"1":null}}
    ## not the output I want or expect
    ## in this case might be a good idea to patch JSON.jl and pull request
    ## and "JSON.print" - is now compiled with this default behaviour

    JSON.lower{T}(v::Set{T}) = collect(v) # overloading 'JSON.lower'

    println(json(Set([1,2,1])))
    ## output: 
    ## [2,1]
    JSON.print(Set([1,2,1]))
    ## output still:
    ## {"dict":{"2":null,"1":null}}
    end # module

If anywhere during runtime before - some piece of code already ran this method, json(v::Set{T}) - then this method is already compiled, and will not use my extention of 'JSON.lower'
This actually happen to me and was hard to debug, as the json() call was behaving as expected, while JSON.print() was behaving differently (just by change in some run-flow)
Since any module (JSON.jl here) can be used by another package I'm using - I have no way of knowing my code actually runs first and actually extends a method correctly

So - what are the best practices to ensure that?

[Currently tested this behaviour with Julia 0.4.6 and 0.5.0]

Note that it is not advised to extend JSON.lower for types you don't own, since JSON is an important package that many others depend on. It's best to create your own types to wrap around built-ins like Set if you need custom JSON serialization. — Fengyang Wang
Cool, this is what I'll do, actually - this is the solution, since no other code will pre-compile a version with the wrapped Set type, which I can define in the same place as defining the type. — oyd11
However - in this specific case - I think the standard JSON package should add 'Set' (a basic type) support, the low-level output of serialising Set, showing the representation ( dict, keys=>null ), can also break if Set implementation changes. I understand this might break some code somewhere. — oyd11
You are correct of course; the internal representation serialization is no good. A PR to add a sensible default lower for Set would be welcome. — Fengyang Wang

mbauman mbauman · Accepted Answer · 2017-01-30T17:16:17

Yes, your analysis is correct on Julia 0.5 and prior. Many developers know this issue number by heart (#265). A large amount of work went into fixing this for the upcoming version 0.6; Julia now tracks the callers of every function and re-compiles them as needed.

In general, though, the best advice here is to patch the library directly and push your change. This is true in both 0.5 and 0.6, even with the run-time correctness fix. It's advised that a you shouldn't extend imported functions with types that you didn't define yourself since it changes that behavior for all other packages that depend upon it. This has become to be colloquially referred to as "type piracy" since you're commandeering the method for your own purposes — purposes that may not suit other callers.

As a temporary workaround, you could try adding the definition to your ~/.juliarc.jl file, which will get executed at startup and has a better chance at being defined before any other methods get compiled. Even this isn't bullet-proof, though, since the package could use precompilation to speed up its usage.

Julia-lang: Extending a method from an installed module, "using" order

1 Answers