Tracking non-.jl source files

[oscardssmith]

This PR removes the requirement that file be a ::AbstractString which makes it possible to teach Revise about non-julia source files by making your own file type (eg PyFile).

[KristofferC]

It would be good to document what are the exact methods one need to overload in order to hook up a custom file format to Revise. And then to finish it off, add a test as well.

[KristofferC]

(fatfingered the close button)

[oscardssmith]

Should I add a section to the developer reference with details on how to do this?

[KristofferC]

I think that is a good place, yes.

[oscardssmith]

Just so I don't forget, here is a relatively minimal example of using this feature

struct MyFile
    file::String
end
Base.abspath(file::MyFile) = MyFile(Base.abspath(file.file))
Base.isfile(file::MyFile) = Base.isfile(file.file)
Base.isabspath(file::MyFile) = Base.isabspath(file.file)
Base.findfirst(str::String, file::MyFile) = Base.findfirst(str, file.file)
Base.String(file::MyFile) = file.file

function make_module(file::MyFile)
    exprs = []
    for line in eachline(file.file)
       val, name = split(line, '=')
       push!(exprs, :($(Symbol(name)) = $val))
    end
    Expr(:toplevel, :(module test
       $(exprs...)
    end), :(using .test))
end

function Base.include(mod::Module, file::MyFile)
    Core.eval(mod, make_module(file))
end
Base.include(file::MyFile) = Base.include(Core.Main, file)
using Revise
function Revise.parse_source!(mod_exprs_sigs::Revise.ModuleExprsSigs, file::MyFile, mod::Module; kwargs...)
    ex = make_module(file)
    @show mod_exprs_sigs, file, mod, kwargs
    Revise.process_source!(mod_exprs_sigs, ex, file, mod; kwargs...)
end
function Revise.is_same_file(a::MyFile, b::String)
    a.file == b
end
function Revise.is_same_file(b::String, a::MyFile)
    a.file == b
end

m=MyFile("test.program")
includet(m)

[oscardssmith]

@KristofferC can you re-run CI?

[oscardssmith]

for reference, this depends on timholy/CodeTracking.jl#95

[oscardssmith]

@timholy bump on this? What are your thoughts?

[timholy]

This seems fine in principle, with the major concern being timholy/CodeTracking.jl#95 (comment).

Briefly, is the idea that this tracks changes in, e.g., Python code as you develop? Wouldn't you be testing the Python code in Python first? I'm a bit confused about exactly how this will be handy (not doubtful, just confused).

[timholy]

If you're doing it this way, any reason not to use string instead of String? Conversely, to insulate this from invalidation, String(filep)::String might be necessary.

[oscardssmith]

The use case for this is in X->julia transpilers. Revising the foreign code is useful because it allows you to use the Julia repl (including debugger.jl) as a repl for non julia source code which can be very useful especially when X is a language that doesn't have good tooling (hence why you are writing a transpiler in the first place). This wouldn't be that useful for Python or other major languages (they tend to be big and have good tooling anyway), but for small languages that traditionally don't have good tooling (i.e. C/BrainFuck/random proprietary languages people wrote in the 80s)

[timholy]

This mostly looks good. I'm proposing a couple of changes to decrease the risk of invalidation of Revise's own code when people load packages.

We do need the test working before we can merge this. I think you also need to run this test from runtests.jl?

[oscardssmith]

One complication here is that these tests won't pass until timholy/CodeTracking.jl#95 is merged and tagged.

[oscardssmith]

Invalidations fixed. Tests pass (with the codetracking PR+Julia1.7. On 1.9, the tests fail with or without this PR, but including my test file passes).

[timholy]

There still seems to be some unnecessary invalidation risk here. Realized invalidations depend on which package you load; you might try InlineStrings.jl as a package that defines a lot of new string types and methods, e.g.,

using SnoopCompileCore
using Revise    # Revise precompiles a lot, so you're probably OK, otherwise force a revision to make sure everything is compiled before the next step
invs = @snoopr using InlineStrings
using SnoopCompile
trees = invalidation_trees(invs)

[oscardssmith]

apologies. the things I resolved were changes that I made that I then lost while rebasing (and didn't check)

[timholy]

That fixed the first path issue but there are more. Presumably it fails locally in the same way with Pkg.test()?

[oscardssmith]

No, this is passing locally (even if I don't run the tests from the Revise folder).

[timholy]

OK. Then try the same trick with all the rest of your paths?

[timholy]

You have another one on line 42?

[pfitzseb]

Non-nightly tests seem to pass now.

[oscardssmith]

Thanks so much!