move to a simpler versioning policy for the Compiler.jl stdlib

[aviatesk]

Update (May 06, 2025)

As a result of discussions, the versioning and management policy for the Compiler.jl stdlib ended up being slightly different from what was originally proposed. For details, please refer to this comment and the newly added READMEs (JuliaLang/julia/Compiler/README.md and JuliaLang/BaseCompiler.jl/README.md).

---

Original OP

For external packages like Cthulhu that use the Compiler.jl stdlib, it's quite a hassle to manage all three types of compatibility: between the external package, the Julia runtime, and Compiler.jl.

I'm thinking instead of handling these compatibility issues by some complex version engineering for Compiler.jl, we should keep its use supplementary. External AbstractInterpreter packages should primarily use Base.Compiler, while managing compatibility with the Julia runtime and Base.Compiler. This would bring us back to the previous situation, but it seems better in terms of compatibility management.

On the other hand, using the Compiler.jl stdlib allows for easy switching of the Compiler implementation, which should be an optional benefit. This is especially useful when developing the Compiler itself, where it's common to do pkg> dev /path/to/Compiler. In such cases, the implementation of the Compiler.jl stdlib should be used instead of Base.Compiler.
As a mechanism to switch the Compiler implementation used by external AbstractInterpreter packages, using package extensions might be the simplest approach. As shown in
this example PR, we can switch the Compiler implementation when using Compiler is called¹, and still precompile the usual code. However, to list Compiler.jl in [extensions], it needs a version. To this end, I propose releasing Compiler.jl versions in line with the Julia runtime versions, following other Julia standard libraries. But we would only do minor version releases, not patch releases, which are harder to define. Specifically, we could release a corresponding version of Compiler.jl when a release branch is cut. After that, we wouldn't manage patch versions, and as long as pkg> dev /path/to/Compiler works, this simple versioning should suffice.
Alternatively, we could register a single version 1.0.0 of Compiler.jl with a dummy implementation, letting the external package determine if the implementation is real (developed) or dummy, but this seems more hacky than using package extensions.

Footnotes

1. Also, I'm aware of another issue: when using the Compiler.jl
   stdlib, you need to run InteractiveUtils.@activate Compiler first,
   or else some reflection utilities won't work (for example,
   Base._which(tt; method_table=Compiler.method_table(myinterp))).
   I think this problem can be solved by adding
   InteractiveUtils.@activate in the __init__ function of the package
   extension code, but I'm not sure if this is a safe solution. It might
   be better to set up a callback that only gets called when
   InteractiveUtils.@activate is executed? ↩

[topolarity]

@Keno Any input here?

This sounds like a sensible solution to me, up to how we might choose to do patch versioning.

At least until the Compiler interfaces start to stabilize enough that we are semi-frequently making one-sided changes, I don't think we'll get a ton of value out of different versioning.

[Keno]

I've thought about this some more, and I generally with the direction. I think it makes sense for downstream users of the compiler to register themselves with both versions of the compiler. Additionally, they should probably follow the base mechanism for deciding which copy of the compiler to use.

I guess the one thing that I'm still not entirely clear about is how to do the versioning itself. In particular, if one of the versions of the compiler is the base copy, then we already need to deal with the fact that they are implicitly versioned by base commit. My proposal would thus be that - for the time being, we don't actually release any formal new Compiler.jl versions and just rely on base version numbers. In the future, I could see us moving Compiler.jl out of Julia, in which case, we can tag version numbers whenever we bump the Compiler.jl version in Julia base (like we do JuliaSyntax right now), but I think that would depend on some additional decoupling.

Lastly, I think we probably need to provide some utility interfaces in the Compiler code that makes it easy for downstream packages to do this split version thing.

[aviatesk]

So, basically, we're going ahead with the approach I outlined before, but, for how to actually use the Compiler stdlib in a package, we'll use a mechanism similar to InteractiveUtils.@activate. And the package won't declare the Compiler.jl stdlib as a dependency in its Project.toml. Is that right? Maybe it would be easier to discuss with the Cthulhu PR as a concrete example.

[Keno]

for how to actually use the Compiler stdlib in a package, we'll use a mechanism similar to InteractiveUtils.@activate. And the package won't declare the Compiler.jl stdlib as a dependency in its Project.toml. Is that right? Maybe it would be easier to discuss with the Cthulhu PR as a concrete example.

It still needs the extension so that the code is available if someone asks for it even if not enabled by default. @activate only switches the default.

[serenity4]

I am not sure this design works well in practice when downstream users rely on interfaces defined in a parent package such as Cthulhu.

Current design

When a Compiler stdlib is loaded, there are two concurrent Compiler modules, each defining their own types and functions.

The solution proposed by @aviatesk works around this by duplicating the contents of a package in an extension that uses symbols from the Compiler stdlib. High-level package entry points (such as descend for Cthulhu) are not duplicated, and instead switch to calling functions from the new "base" module.

Issue

This design works well for these specific entry points, because the user will not observe any changes, but there are a few issues when trying to apply this downstream.

I will take the example of Cthulhu and Diffractor (downstream user of Cthulhu), though the following applies to any such pattern.

Cthulhu now has two modules: one that works with Base.Compiler (Cthulhu), and one that works with the Compiler stdlib (CthulhuCompilerExt). The issue is, the interface defined by Cthulhu (including e.g. AbstractCursor) used by downstream consumers will have two versions: Cthulhu.AbstractCursor, and CthulhuCompilerExt.AbstractCursor. The former, though naturally used by downstream dependents, is then incompatible with the Compiler stdlib. Furthermore, there is no possibility to use CthulhuCompilerExt.AbstractCursor in a hypothetical DiffractorCompilerExt extension, because one would require a direct dependency on CthulhuCompilerExt, which is not permitted in the current package extension mechanism.

Here are workarounds that I have attempted, and that failed (a few reasons were already mentioned above):

• Define a DiffractorCompilerExt that provides Compiler-compatible internals. This is not a solution because it needs symbols from CthulhuCompilerExt, which is not available during precompilation.
• Instead of duplicating symbols, simply add methods to support types from the Compiler stdlib (i.e. define f(::Base.Compiler.IRCode) and f(::Compiler.IRCode) for the same function f). This falls short for types that rely on subtyping, such as CthulhuInterpreter <: Base.Compiler.AbstractInterpreter; in this case, we can't work our way around having to define a second CthulhuInterpreter <: Compiler.AbstractInterpreter type.

I believe there are ways to re-architect a package so that it provides support for both Compiler packages with the current extension mechanism (such as replacing subtyping with trait-based dispatch in its interface), but not without significantly intrusive changes which I would perceive as undesirable.

[serenity4]

One possible solution could be to do the same thing as the CthulhuCompilerExt package extension but at runtime, with something like @activate which would not only switch the default but also dynamically perform the package duplication before hand. Packages could register themselves with a hook mechanism in their __init__ so that @activate triggers these hooks in order. Perhaps something like this:

Cthulhu.jl:

CTHULHU_MODULE = IdDict{Module, Module}(Base.Compiler => Cthulhu)

function duplicate_package(Compiler)
    mod = @eval module CthulhuCompilerRuntimeExt
        const CC = $Compiler
        using .CC
        include("CthulhuBase.jl")
    end
    CTHULHU_MODULE[Compiler] = mod
end

function __init__()
    register_compiler_hook(duplicate_package, @__MODULE__)
end

Diffractor.jl:

DIFFRACTOR_MODULE = IdDict{Module, Module}(Base.Compiler => Diffractor)

function duplicate_package(Compiler)
    mod = @eval module DiffractorCompilerRuntimeExt
        const CC = $Compiler
        using .CC
        const Cthulhu = Cthulhu.CTHULHU_MODULE[CC]
        using .Cthulhu
        include("DiffractorBase.jl")
    end
    DIFFRACTOR_MODULE[Compiler] = mod
end

function __init__()
    register_compiler_hook(duplicate_package, @__MODULE__)
end

This wouldn't benefit from precompilation, nor support any kind of versioning/compat, and yet again relies on __init__ to do something cursed, but at least it would provide a more composable way to support both Compiler modules. I'd be glad to see (better) alternative designs that address these concerns though.

[serenity4]

Another path worth mentioning would be to go back to managing a single Compiler dependency, but have it default to the package located under the currently installed Julia version. Essentially, that would imitate a require_stdlib-like mechanism that might be overridden to use a deved version for development purposes.

This could be expressed as a special [sources] entry:

[sources]
Compiler = {path = "@julia/Compiler"}

I personally find that simpler and clearer than any of the other solutions that try to support two different Compiler modules at the same time. Unfortunately, it seems that currently [sources] does not apply transitively, so more effort would be required to make this work.

[aviatesk]

It's been a little while, but I'd like to get this PR moving forward again.
To be honest, I'm not a big fan of the current solution using package extensions because of its complexity and invasiveness, and I'd prefer to move forward with a simpler approach reaching a consensus on how to use the Compiler.jl stdlib from packages, ideally before the v1.12 release.

The problem with the current package extension approach is that packages need to provide two essentially duplicate code paths depending on the Compiler implementation.
Even with the Cthulhu example, if we consider adding dependencies on other compiler utilities like LoweredCodeUtils in the future, not just Diffractor, it feels like we'd eventually end up forcing this package extension hack onto the whole ecosystem, which doesn't seem like the right path.

Instead, I thought about a way to switch compiler implementations within the Compiler stdlib itself: as an alternative, what about the idea of having a package that's essentially identical to Base.Compiler serve as the default fallback version for the Compiler.jl stdlib:

baremodule Compiler
for name in Base.names(Base.Compiler; all=true, imported=true)
    Core.eval(Compiler, :(using Base.Compiler: $name))
end
end

Specifically, we could register v0.0.0 of the Compiler stdlib with the fallback implementation. This version would just import everything from Base.Compiler. This way, when a package does using Compiler, all the names it gets access to are identical to those in Base.Compiler.
The advantage of this approach is that, until we start proper versioning for the Compiler stdlib, it lets packages use Base.Compiler within the existing Pkg.jl system, while also optionally allowing them to opt in to use any specific Compiler version they want.
A single Manifest.toml would only reference one Compiler.jl version, so we wouldn't need package extensions, and we'd still get precompilation support. Plus, doing dev Compiler would allow using a local development version of Compiler.jl.

To implement this, we could yank all currently released versions of Compiler.jl and, for now, release only this v0.0.0 fallback version as described.

Then packages can declare

Project.toml

Compiler = "0"

By default, this would just let them use names from Base.Compiler. But package users could still switch to a specific compiler version by:

pkg> dev path/to/Compiler.jl

or

pkg> add https://github.com/JuliaLang/julia/tree/avi/EA-overhaul/Compiler

Of course, when we eventually start proper versioning for the Compiler.jl stdlib, packages using this setup could easily migrate just by changing their Compiler version requirement (It might even be useful to keep this v0.0.0 (Base.Compiler) fallback available permanently though).

[aviatesk]

I've implemented the fallback version for the Compiler.jl stdlib, as proposed in the previous comment.

There are a couple of things we should probably discuss or note:

• This fallback version can't be bootstrapped, so maybe it doesn't really make sense to keep it in this repository. Maybe we should create a separate repository just for maintaining this fallback version and manage it there instead?
• This way, we don't need to empty the stdlib for v1.10 and v1.11. So, like we do for v1.12 onwards, it's implemented just as the fallback version that re-exports Core.Compiler. But of course switching to a custom compiler implementation isn't supported on those versions.

[aviatesk]

If we agree on this approach, here's how we'll finish switching to the new Compiler.jl stdlib infrastructure:

• Create a new repository called BaseCompiler.jl under the JuliaLang organization. This will hold the default fallback version of the Compiler.jl standard library (version 0.0.0). EDIT: See https://github.com/JuliaLang/BaseCompiler.jl
• Remove (yank) the previously released Compiler versions 0.0.1 to 0.0.3 from the General registry. EDIT: See delete Compiler.jl versions v0.0.1~v0.0.3 and register v0.0.0 only JuliaRegistries/General#130304
• Release Compiler.jl version 0.0.0. EDIT: See delete Compiler.jl versions v0.0.1~v0.0.3 and register v0.0.0 only JuliaRegistries/General#130304
• Add documentation to JuliaLang/BaseCompiler.jl/README.md and JuliaLang/julia/Compiler/README.md explaining how to use and how the Compiler stdlib works. EDIT: Done at this PR.

[StefanKarpinski]

Remove (yank) the previously released Compiler versions 0.0.1 to 0.0.3 from the General registry.

Yanking is kind of problematic. Is there a chance that people are using these?

Release Compiler.jl version 0.0.0

That's not a good version number to use for any release...

[serenity4]

Yanking is kind of problematic. Is there a chance that people are using these?

A quick package search reveals that no registered package uses Compiler in their project, so yanking should be safe.

EDIT: I wonder why Diffractor.jl does not show up in that search. Is it scoped to releases only? (Diffractor hasn't released after it added its dependency to Compiler in its project)
In any case, using a separate Compiler stdlib is a 1.12 feature, and 1.12 hasn't been released yet.

[aviatesk]

That's not a good version number to use for any release...

I agree, but this Compiler stdlib version is special. It's a temporary fallback version we'll use until we start proper versioning for the Compiler stdlib. So, I think using this unusual version number 0.0.0 might be a good way to show that it's a special version.

[aviatesk]

I’ve completed the process proposed at #57520 (comment). As a result, this PR only makes minor adjustments to Compiler/Project.toml and adds a Compiler/README.

If you have any feedbacks about the README added in this PR or about the README in the BaseCompiler.jl repository, please let me know.