A better mechanism for coordinating internal breaking changes.

[Keno]

This was origiginally supposed to be an issue, but I just started writing out the whole code in the issue text to explain what I want all the behavior to be, so instead, here's the actual implementation of it, with the motativation in the commit message, and the details of the actual behavior in the code change ;)

Sometimes packages rely on Julia internals. This is in general discouraged, but of course for some packages, there isn't really any other option. If your packages needs to hook the julia internals in a deep way or is specifically about introspecting the way that julia itself works, then some amount of reliance on internals is inevitable. In general, we're happy to let people touch the internals, as long as they (and their users) are aware that things will break and it's on them to fix things.

That said, I think we've been a little bit too caveat emptor on this entire business. There's a number of really key packages that rely on internals (I'm thinking in particular of Revise, Cthulhu and its dependency stacks) that if they're broken, it's really hard to even develop julia itself. In particular, these packages have been broken on Julia master for a more than a week now (following #52415) and there has been much frustration.

I think one of the biggest issues is that we're generally relying on VERSION checks for these kinds of things. This isn't really a problem when updating a package between released major versions, but for closely coupled packages like the above you run into two problems:

1. Since the VERSION number of a package is not known ahead of time, some breaking changes cannot be made atomically, i.e. we need to merge the base PR (which bumps people's nightly) in order to get the version number, which we then need to plug into the various PRs in all the various packages. If something goes wrong in this process (as it did this time), there can be extended breakage.

2. The VERSION based comparison can be wrong if you're on an older PR (that's a head of the base branch, but with different commits). As a result, when working on base PRs, you not infrequently run into a situation, where you get a VERSION false-positive from updating a package, introducing errors you didn't see before. This one isn't usually that terrible, because a rebase will fix it (and you'll probably need to rebase anyway), but it can be very confusing to get new and unexpected errors from random packages.

I would propose that going forward, we strongly discourage closely coupled packages from using VERSION comparisons and intead:

1. Where possible, probe for the feature or method signature that they're actually looking for, if it's something small (e.g. a rename of base type).
2. That we as julia base establish a mechanism for probing whether your current pre-release julia has a certain change. My sketch proposal is in this PR.

[KristofferC]

Ref #48764.

I think one of the biggest issues is that we're generally relying on VERSION checks for these kinds of things

I disagree with this. The real problem is the work required to adapt existing code to work with the new internals. There is no features/version checking or whatever that could be done in JuliaInterpreter to magically make it work when the whole line number system changed. It is just work required to adapt to that. And if we consider e.g. Revise crucial then the only real solution to that is to add it to CI and not merge PRs breaking them until there also has been a corresponding fix to the package. Just like if they where stdlibs.

[Keno]

Well, the change here is as much social as it is technical, but having a specific mechanism and clearly established procedures for how to do this kind of thing has big value I think. Obviously it doesn't by itself fix the maintenance burden, but I think for at least the handful of core packages we just have to take that on and enforce it with PkgEval or CI.

[oxinabox]

One advantage of VERSION is that it's easy to find and strip out

[Keno]

Having discussed this with a few more people, I think the general consensus is skepticism that it'll actually do much, but no strong objections, so I'm gonna rebase/merge this and we can try it out for a release and if we don't like it, we can just rip it out again and replace any use by regular VERSION checks (which we'd be doing anyway in this design after the 1.12 release).