Julia support

[waghanza]

Hi,

It will be awesome if renovatebot, could parse Project.toml in Julia Project.

Dependencies in julia could be specified in various ways :

• ^1.2.3 => [1.2.3, 2.0.0)
• ^1.2 => [1.2.0, 2.0.0)
• ^1 => [1.0.0, 2.0.0)
• ^0.2.3 => [0.2.3, 0.3.0)
• ^0.0.3 => [0.0.3, 0.0.4)
• ^0.0 => [0.0.0, 0.1.0)
• ^0 => [0.0.0, 1.0.0)
• ~1.2.3 => [1.2.3, 1.3.0)
• ~1.2 => [1.2.0, 1.3.0)
• ~1 => [1.0.0, 2.0.0)
• ~0.2.3 => [0.2.3, 0.3.0)
• ~0.0.3 => [0.0.3, 0.0.4)
• ~0.0 => [0.0.0, 0.1.0)
• ~0 => [0.0.0, 1.0.0)
• >= 1.2.3 => [1.2.3, ∞)
• ≥ 1.2.3 => [1.2.3, ∞)
• = 1.2.3 => [1.2.3, 1.2.3]
• < 1.2.3 => [0.0.0, 1.2.2]

@codeneomatrix Do I forget anything according julia and deps management

@rarkins You can find a example in https://github.com/the-benchmarker/web-frameworks/blob/e2c82e935f9038fe7a54a10d04654b31eacf8ee8/julia/merly/Project.toml#L4

Regards,

[rarkins]

First thing is: do we need a new datasource? Seems like we do, is it https://github.com/JuliaRegistries/General? Example versions list for a random package: https://raw.githubusercontent.com/JuliaRegistries/General/master/G/GaussianProcesses/Versions.toml

After datasource we need to verify versioning. I checked the full list of ranges above and npm's semver implementation actually follows the same results, so looks like we can reuse it. Maybe we alias it in case one day it needs to diverge.

Finally there is the manager. We'd need to extract dependencies from Project.toml files, which should be pretty easy. I don't see any lock file in Julia, so that should be all we need.

[neomatrixcode]

Hi.

Starting with Julia version 1.4, it is also possible to use hyphens in the version specification of dependencies.

So it is recomended to also add

[compat]
julia = "1.4"

to the project file when using them.

• 0.2.3 - 4.5.6 => [0.2.3, 4.5.6]

Any unspecified trailing numbers in the first end-point are considered to be zero:

• 1.2 - 4.5.6 => [1.2.0, 4.5.6]
• 1 - 4.5.6 => [1.0.0, 4.5.6]

Any unspecified trailing numbers in the second end-point will be considered to be wildcards:

• 1.2.3 - 4.5 => 1.2.3 - 4.5.* = [1.2.3, 4.6.0)
• 1.2.3 - 4 => 1.2.3 - 4.. = [1.2.3, 5.0.0)
• 1.2 - 4.5 => 1.2.0 - 4.5.* = [1.2.0, 4.6.0)
• 1.2 - 4 => 1.2.0 - 4.. = [1.2.0, 5.0.0)
• 1 - 4.5 => 1.0.0 - 4.5.* = [1.0.0, 4.6.0)
• 1 - 4 => 1.0.0 - 4.. = [1.0.0, 5.0.0)

[rarkins]

I underestimated a little. First, the deps section in Project.toml contains a form of checksum, while also these's a Manifest.toml file that needs to be updated by pkg.

The pkg examples on https://docs.julialang.org/en/v1/stdlib/Pkg/index.html also shows that the pkg command is run from within a repl, so I'm not sure how it would be run "headless".

I have concluded that someone more familiar with Julia needs to take over the work, but I've done some of the implementation and most of the scaffolding in https://github.com/renovatebot/renovate/compare/feat/6508-julia

[neomatrixcode]

Project.toml
The project file describes the project on a high level, for example the package/project dependencies and compatibility constraints are listed in the project file. The file entries are described below.

• The name field
  The name of the package/project is determined by the name field, for example:
  name = "Example"
  The name can contain word characters [a-zA-Z0-9_], but can not start with a number.

• The uuid field
  uuid is a string with a universally unique identifier for the package/project, for example:
  uuid = "7876af07-990d-54b4-ab0e-23690620f79a"
  The uuid field is mandatory for packages.
  It is recommended that UUIDs.uuid4() is used to generate random UUIDs.

• The version field
  version is a string with the version number for the package/project. It should consist of three numbers, major version, minor version and patch number, separated with a .

• The [deps] section
  All dependencies of the package/project are listed in the [deps] section. Each dependency is listed as a name-uuid pair, for example:

[deps]
Example = "7876af07-990d-54b4-ab0e-23690620f79a"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

Typically it is not needed to manually add entries to the [deps] section; this is instead handled by Pkg operations such as add.

• The [compat] section
  Compatibility constraints for the dependencies listed under [deps] can be listed in the [compat] section. Example:

[deps]
Example = "7876af07-990d-54b4-ab0e-23690620f79a"

[compat]
Example = "1.2"

The Compatibility section describes the different possible compatibility constraints in detail. It is also possible to list constraints on julia itself, although julia is not listed as a dependency in the [deps] section:

[compat]
julia = "1.1"

Manifest.toml
The manifest file is an absolute record of the state of the packages in the environment. It includes exact information about (direct and indirect) dependencies of the project. Given a Project.toml + Manifest.toml pair, it is possible to instantiate the exact same package environment, which is very useful for reproducibility. For the details, see Pkg.instantiate.

The Manifest.toml file is generated and maintained by Pkg and, in general, this file should never be modified manually.

---

source https://julialang.github.io/Pkg.jl/v1/toml-files/

[neomatrixcode]

https://discourse.julialang.org/t/how-to-give-a-valid-uuid-to-an-existing-package-and-how-to-desactivate-an-environment/19236/5

[neomatrixcode]

https://stackoverflow.com/questions/55320875/how-to-get-julia-package-uuid

[neomatrixcode]

https://discourse.julialang.org/t/how-can-i-know-the-uuid-of-a-registered-package/14081/4

[neomatrixcode]

https://discourse.julialang.org/t/pkg-api-for-getting-uuid-of-another-package/15061/3

[rarkins]

Thanks for the pointers. It seems like the deps section should not need updating by Renovate, even if the compat values do.

The next question is:

• If we manually update the Project.toml file with a new compat value, what is the most lightweight pkg command to run to update the corresponding Manifest.toml. I'm also still confused by the Manifest.toml is empty in the example provided

[neomatrixcode]

Note: currently REQUIRE is no longer used

https://discourse.julialang.org/t/whats-the-use-a-project-toml-file-and-the-the-use-of-a-manifest-toml-in-a-julia-project/22524/2

[neomatrixcode]

https://julialang.github.io/Pkg.jl/v1.4/api/#Pkg.instantiate

[neomatrixcode]

https://julialang.github.io/Pkg.jl/v1.4/api/#Pkg.update

[giordano]

Is there a way to make this happen? Do you need help with something?

[rarkins]

Yes, you can submit a PR

[giordano]

Well, that escalated quickly 🙂 I don't even know where to start from as I'm not familiar at all with renovate code, maybe some guidance on what has to be done and where to look at?

[github-actions]

Hi there,

You're asking us to support a new package manager. We need to know some basic information about this package manager first. Please copy/paste the new package manager questionnaire, and fill it out in full.

Once the questionnaire is filled out we'll decide if we want to support this new manager.

Good luck,

The Renovate team

[rarkins]

Actually, the best next step would be for someone to fill out the questionnaire linked above. After that we can advise on the steps after that

[bauglir]

I'm sure I'm missing some details, but I gave filling out the form a stab.

New package manager questionnaire

Did you read our documentation on adding a package manager?

• I've read the adding a package manager documentation.

Basics

What's the name of the package manager?

Pkg, or Pkg.jl (but Julia package are always referenced without the extension).

What language(s) does this package manager support?

Julia.

How popular is this package manager?

It is the standard package manager for the Julia language and considered one of the languages strengths.

Does this language have other (competing?) package managers?

• Yes (give names).
• No.

What are the big selling points for this package manager?

It is the primary (and only?) package manager that supports the Julia language.

Detecting package files

What kind of package files, and names, does this package manager use?

• (Julia)Project.toml is the equivalent of a package.json file. It declares the dependencies of a project by name and matches them to the UUIDs that are used to reference them ("names" of packages do not have to be unique across "package registries", but UUIDs do have to be). A dependency manager should therefore likely work directly on the UUIDs as it is allowed to enable multiple package registries that may serve packages with the same name (but differing UUIDs) at the same time. Dependencies are recorded in a [deps] key and the corresponding compatibility under a [compat] key. Note that specifying a [compat] entry for a dependency is not mandatory, but recommended practice.
• (Julia)Manifest.toml is the equivalent of a package-lock.json file. It records all (transitive) dependencies and their versions (including the Julia version). This is intended to support reproducibility and can be used to instantiate a project's dependencies by itself. Committing this file to a project is optional.

These files are typically located at the root of a project (with a "package" being a specific type of project). But the files may be located elsewhere as well. For instance, in the case of monorepos or to add use case specific dependencies, e.g. when running tests or creating documentation. It is important to note that each (Julia)?Project.toml declares its own unique set of dependencies, they do not have to be "merged" (e.g. a test environment would depend on the package itself to pull in the main dependencies of the package).

There are multiple ways of recording test dependencies. One is to record them in a dedicated test/Project.toml file and the other is using extras and targets keys in the Project.toml. In the latter case it is not possible, as far as I know given that I personally don't use the feature, to restrict versions (although there may be some interaction with the [compat] key). As far as I know the extras and targets declarations also have not been used outside of the context of specifying test dependencies although originally intended otherwise (as documented).

A (Julia)?Project.toml can also specify "weak dependencies". This is a new Julia feature (as of version 1.9) that allows specifying dependencies that when installed will activate "package extensions". Compatible versions for weak dependencies are also managed using the [compat] key in the (Julia)?Project.toml. This is a more advanced feature intended for experienced package maintainers.

See the documentation for details on these files.

Which fileMatch pattern(s) should Renovate use?

[ "(Julia)?Project.toml", "(Julia)?Manifest.toml" ]

Do many users need to extend the fileMatch pattern for custom file names?

• Yes, provide details.
• No.

Is the fileMatch pattern going to get many "false hits" for files that have nothing to do with package management?

I don't believe so. A (Julia)?Project.toml file may opt to not specify any dependencies, but that is its primary use case. The unprefixed name may also be more common, but I have personally not run into it elsewhere.

Parsing and Extraction

Can package files have "local" links to each other that need to be resolved?

Only (Julia)?Manifest.toml files can resolve to local paths to aid with local development. A (Julia)?Project.toml file always requires a package to be registered with a package registry. There have been some proposals to change this, JuliaLang/Pkg.jl#492, JuliaLang/Pkg.jl#3263, but these have not been finalized/implemented.

Package file parsing method

The package files should be:

• Parsed together (in serial).
• Parsed independently.

I'm not 100% sure on this, but I believe they could be parsed independently.

Which format/syntax does the package file use?

• JSON
• TOML
• YAML
• Custom (explain below)

How should we parse the package files?

• Off the shelf parser.
• Using regex.
• Custom-parsed line by line.
• Other.

Julia's package manager could be used to parse the package files directly. But it'd add a dependency on Julia.

Does the package file have different "types" of dependencies?

• Yes, production and development use case-specific dependencies.
• No, all dependencies are treated the same.

As mentioned above, along the "primary" dependencies declared at the root of a project, test (and other use case) specific dependencies, etc. may be defined in either separate (Julia)?Project.toml files or in certain cases within the primary (Julia)?Project.toml.

"Weak" dependencies are another type of dependency, as also mentioned, but these declare their versions in the same location as "regular" dependencies within a (Julia)?Project.toml.

List all the sources/syntaxes of dependencies that can be extracted

In a (Julia)?Project.toml file:

• The [deps] key,
• The [weakdeps] key,
• The [extras] key (not support for versioning).

In a (Julia)?Manifest.toml file:

• The [[deps]] key.

A (Julia)?Artifacts.toml file may specify additional dependencies. However, these dependencies are content-addressed and I'm not certain whether those could be managed by Renovate.

Describe which types of dependencies above are supported and which will be implemented in future.

• "Regular" dependencies.
• Use-case specific dependencies through separate (Julia)?Project.toml and/or (Julia)?Manifest.toml files.
• "Weak" dependencies.

Towards the future support for "relative" dependencies may be added to the (Julia)?Project.toml files.

Versioning

What versioning scheme does the package file(s) use?

By convention, Semantic versioning is used.

Does this versioning scheme support range constraints, like ^1.0.0 or 1.x?

• Supports range constraints (for example: ^1.0.0 or 1.x), provide details.
• No.

See the documentation for a full explanation of the semantics. The semantics used are very similar to those used by NPM.

Lookup

Is a new datasource required?

• Yes, provide details.
• No.

A (Julia)?Project.toml file is expected to refer to dependencies registered in a "package registry". A General registry is available that records all open-source packages. Additional registries may be created either open- or closed-source. Registries can be stacked (provided some restrictions).

A package registry can be interacted with directly through a Git repository, but is typically interacted with through a package server. Package servers may be public (such as the one hosted on https://juliahub.com) or private.

It is important to note that neither the (Julia)?Project.toml nor the (Julia)?Manifest.toml track which registries package are available from. Which registries are active is an external configuration. Hence, it may be the case that a dependency cannot be found if package servers/registries are not properly configured when resolving package dependencies.

So likely two data sources would have to be added: one for package servers and one for package registries hosted on Git directly. I'd say this could be done incrementally though primarily relying on a package server data source and using the other for more advanced use cases.

Will users want (or need to) set a custom host or custom registry for Renovate's lookup?

• Yes, provide details.
• No.

As mentioned above, although the General registry is typically directly available within a Julia session, additional package servers/registries need to be explicitly configured.

Where can Renovate find the custom host/registry?

• No custom host or registry is needed.
• In the package file(s), provide details.
• In some other file inside the repository, provide details.
• User needs to configure Renovate where to find the information, provide details.

Package servers and/or registries are configured using URIs and/or UUIDs. A single package server may host multiple registries and it is possible to either enable a single registry from a package server or all of them at once. Only a single package server can be active at the same time (although I do believe it'd be possible to change the package server and initialize registries from different ones, but that seems like a very advanced use case). In case a registry is represented directly by a Git repository it can be referenced directly by URI.

See the Registry API reference for details on how registries can be specified using the RegistrySpec.

For private registries, whether hosted through a package server or a private Git repository, it would be necessary to be able to configure appropriate credentials.

Are there any constraints in the package files that Renovate should use in the lookup procedure?

• Yes, there are constraints on the parent language (for example: supports only Python v3.x), provide details.
• Yes, there are constraints on the parent platform (for example: only supports Linux, Windows, etc.), provide details.
• Yes, some other kind of constraint, provide details.
• No constraints.

A special julia dependency may be added to a [compat] section restricting the Julia versions that a package is compatible with. This applies the same version semantics as apply to packages as discussed in the documentation.

There are not restrictions on the parent platform logged in the (Julia)?Project.toml. The artifact system provided by the package manager is typically used to resolve platform specific dependencies of a package.

Will users need the ability to configure language or other constraints using Renovate config?

• Yes, provide details.
• No.

It may be useful to be able to configure available registries through Renovate's configuration.

Artifacts

Does the package manager use a lock file or checksum file?

• Yes, uses lock file.
• Yes, uses checksum file.
• Yes, uses lock file and checksum file.
• No lock file or checksum.

Is the locksum or checksum mandatory?

• Yes, locksum is mandatory.
• Yes, checksum is mandatory.
• Yes, lock file and checksum are mandatory.
• No mandatory locksum or checksum.
• Package manager does not use locksums or checksums.

If lockfiles or checksums are used: what tool and exact commands should Renovate use to update one (or more) package versions in a dependency file?

The Pkg package manager provides facilities to manipulate the lockfile. The Pkg.update function can be used to manipulate both the (Julia)?Project.toml and (Julia)?Manifest.toml files depending on the "package mode" it is configured to use. The function can be used to update a single dependency, a group of dependencies or all dependencies.

The Pkg.resolve function can be used to generate a lockfile from scratch.

Package manager cache

Does the package manager use a cache?

• Yes, provide details.
• No.

Julia package operations are relative to a (stack of) "depots". These depots may contain caches of available registries and their contents, as well as available packages. The package manager can be set to work in an "offline" mode which will only take versions and packages into consideration that are already available within these depots.

If the package manager uses a cache, how can Renovate control the cache?

• Package manager does not use a cache.
• Controlled via command line interface, provide details.
• Controlled via environment variables, provide details.

The JULIA_DEPOT_PATH environment variable can be used to control the depot path (either fully replacing it, or stacking values on top of or below the default stack).

The DEPOT_PATH can also be controlled through Julia itself by manipulating the global variable.

Should Renovate keep a cache?

• Yes, ignore/disable the cache.
• No.

Given the package manager's capabilities it might be sufficient for Renovate to rely on a "depot" as a cache.

Generating a lockfile from scratch

Renovate can perform "lock file maintenance" by getting the package manager to generate a lockfile from scratch.
Can the package manager generate a lockfile from scratch?

• Yes, explain which command Renovate should use to generate the lockfile.
• No, the package manager does not generate a lockfile from scratch.
• No, the package manager does not use lockfiles.

Remove an existing Manifest.toml and run Pkg.resolve.

Other

What else should we know about this package manager?

The Pkg.update function exposes control over whether only the package itself is allowed to be updated, or whether its transitive dependencies may also be updated (depending on whether they are also direct dependencies of the project) using the preserve keyword argument.

[rarkins]

Thanks! Skimming through this, it looks like we need a new datasource, a new manager, and probably a new versioning to wrap around npm if the syntax isn't exactly the same as npm.

Usually the best way to start is with the datasource, then versioning, then manager. These can be done in separate PRs one by one. However if the versioning is 90%+ close to npm's it may be ok to skip the versioning until a later stage and just document that the initial release doesn't cover ranges perfectly.

Datasources are added here: https://github.com/renovatebot/renovate/tree/main/lib/modules/datasource

These days we like to use Zod for validation. I think that the existing hex datasource is probably a good one to copy the concepts from

[bauglir]

Thanks for the pointers. This confirms my expectations for how to best deal with this.

Trying to address this has been on my radar for a while, but I haven't been able to prioritize it yet. Given the renewed interest, now may be the time. I think we can coordinate within the Julia community to try and move this forward.