Publish individual indices on Maven Central

[alexarchambault]

This adds a Mill project under publish/, that picks indices in indices, and publishes them individually on Maven Central

Tested (publication on Central included) on a separate branch in the repo here.

This results in one module per OS / architecture being published (see here for the list of modules). For each of them, we get a JAR, containing entries like

  Length      Date    Time    Name
---------  ---------- -----   ----
      360  10-21-2024 20:51   META-INF/MANIFEST.MF
        0  10-21-2024 20:51   META-INF/
        0  10-21-2024 20:51   coursier/
        0  10-21-2024 20:51   coursier/jvm/
        0  10-21-2024 20:51   coursier/jvm/indices/
        0  10-21-2024 20:51   coursier/jvm/indices/v1/
   146037  10-21-2024 20:51   coursier/jvm/indices/v1/darwin-arm64.json
---------                     -------
   146397                     7 files

Going to make changes in coursier-jvm so that it's able to pick indices from these JARs.

Beware that the format of the index in these JARs might evolve a bit (I'd like to remove the deprecated JVMs from it, some adoptium and legacy graal stuff in particular).

[lihaoyi]

@alexarchambault just an idea, but what if we publish one artifact per OS/architecture/JVM/version? e.g. something like

io.get-coursier:jvm-index-termurin-arm-linux:17.0.1

That way we can publish artifacts for the JVMs that are available now for the Mill client to use, and as new JVMs are added the Mill client can just bump the version to pick up the new JVM index artifacts while the old ones remain immutable and stable, without having to separately bump an "index version" to allow the new JVM versions to be resolved

[alexarchambault]

@lihaoyi You mean, mirroring JVM versions on Maven Central? It seems that couldn't be a full substitute to the current index, but that could be helpful in some scenarios (Mill projected use included)

It wouldn't work for all JVMs we have in the index. The Oracle ones in particular, that get removed from time to time, as Oracle only allows for 2 or 3 major versions to be downloaded without logging in. Although we could think of hacks to work around that, such as "tombstones" versions say.

Also, I'd feel kind of bad pushing lots of empty releases to Maven Central (there are currently more than 14k entries in the index, which would translate to as many releases).

But that could work for a few JVMs, Temurin and GraalVM say. And that would work around an issue with indices on Maven Central: past indices accumulating in the user's coursier cache. Instead, the version listings would be maven-metadata.xml files listing versions on Maven Central, such as this one. These are considered to be "changing" artifacts by coursier (like snapshots or the current index on GitHub that lives at a fixed URL), so these just get overwritten in the cache.

[alexarchambault]

That would require some more work on the coursier-jvm library side though

[lihaoyi]

Perhaps it's worth enumerating exactly what JVMs we support and what kind of special casing they require. I think we can consider only supporting a subset of the current listed JVMs in Mill if it allows us to constrain the problem and make it easier to manage, e.g. allowing us to generate JVM download URLs on the fly for new versions without needing to do a batch update of the entire index each time a new JDK point version is released. Gradle only supports one distribution by default after all and it seems to work fine for them(temurin)

[alexarchambault]

Maybe we could use the versions from Maven Central if users specify just a version as a JVM (8 or 17.0.2 say), and use the current index on GitHub if users ask for a specific JVM distribution, like graalvm:23 or zulu:17.0.3

[alexarchambault]

And we'd only push Temurin versions on Maven Central

[lihaoyi]

Apart from Oracle, are the download links for the other JDKs like Temurin or Azul stable and predictable?

I'm wondeirng how far we can get with the "write code to construct download URL at runtime from version number" approach. The dynamic github index seems fine for cases where it doesn't work, but if constructing URLs at runtime works for the majority of use cases that would already be a big win even if not 100%

[lihaoyi]

But I think as an immediate step to unblock Mill, publishing the whole index as one artifact to maven central for mill to use is probably fine. The finer optimizations of the long term index management strategy can come later