Decentralized package management

[certik]

Rust allows the packages to be decentralized, they do not have to be listed at https://crates.io/.

The Go language does not even have a centralized package registry at all, see this blog post that explains it in the section "Package Management":

https://nullprogram.com/blog/2020/01/21/

Modules are named by a module path that includes its network location. This means there’s no land grab for popular module names.

I don't like the Go's model that the name of the package is the url, I prefer the Rust model where a package has a simple name, but you specify where it can be downloaded. The end result is the same. (Overall, the Rust package management seems much more thought out, and it seems they fixed the issues that Go is tackling long time ago.)

I want to implement the same approach for fpm. That will allow us to build an ecosystem of packages for Fortran, without a "goldrush" to reserve a popular package name in a centralized registry. Then, as we have a healthy ecosystem of packages, we can add a centralized registry later.

[milancurcic]

Overall I like this direction. However I'm confused about how this works in practice. Perhaps it's just a terminology issue.

Like PyPI, Crates serves both as a registry (metadata), and code distribution.

I agree that it's not feasible for fpm to host and distribute code. It's a huge ordeal and technical challenge.

However, do you also mean that fpm wouldn't have a centralized metadata registry? If yes, how would a newcomer to Fortran and fpm search for packages? It seems to me that if you don't have some centralized metadata registry, the user would need to know what package they need and how to download it.

[certik]

I think it's a step by step process, we start with the step 1. below, then we progress to step 2. and then to step 3.:

1. This issue Decentralized package management #33: indeed there is no centralized metadata registry. The advantage is that there is no "goldrush" to reserve popular names in a centralized registry. Initially there is no package search (newcomers must know the package name), but I can imagine actually creating a search --- all we need for people to submit urls of packages out there, and since each is using fpm.toml, we can then create a search of those decentralized packages.

2. Issue Minimal metadata registry #34: minimal centralized registry. Allows a natural fpm search capability.

3. Issue Full package registry #35: both registry and code distribution.

If step 1. is fully developed, there can actually be a "decentralized package registry". It would work just like described in step 2. (see the issue #34 for details), except that there would not be a centralized name for a package, so people could submit various packages (as url) that can have the same name. Then fpm search would give you all the packages with urls to put into your fpm.toml. We can curate this list as a community to prevent spam and malicious code.

[milancurcic]

Coming from #34, let's discuss how the step 1 (no centralized registry) works. This means we don't need fpm search and fpm list.

Let's say I just want to fpm install with datetime-fortran as a dependency. I know the name of the package. I don't (and shouldn't) know the URL of the package.

@certik How does this work? It seems to me that we need some metadata even at this stage.

Or is the user expected to enter something like:

[dependencies]
name = "datetime"
url = "https://github.com/wavebitscientific/datetime-fortran"

to their fpm.toml?

Both approaches seem okay to me for now (latter is simpler), I'm just not clear what's your idea how this works.

[certik]

The step 2 in #34 is what we should eventually do, and then step 3 in #35. It looks like we got agreement on both.

You can only install datetime-fortran (btw, it should probably just be called datetime) without knowing the url at the step 2., because you need the registry to get the url.

So let's start with step 1 in this issue #33. In here I propose to do exactly what Cargo does. Say I want to use datetime in my program. Here are details about Cargo's syntax: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html, so let's do what they do:

I would have this in my fpm.toml:

[dependencies]
datetime = "1.7.0"

Once step 2 is implemented, then fpm would have a default registry, and you can also specify your own registry either on a command line or explicitly:

[dependencies]
datetime = { version = "1.7.0", registry = "my-registry" }

Until then, in this step 1, we do not have a registry, so you have to also tell it where it can download the datetime package directly. As explained at the webpage, Cargo only uses the version field if it downloads from a registry, otherwise it ignores it if it downloads it directly from a git repository or a local path. We can start with that approach, if we need more, we can modify that behavior. So here is how you would do it:

[dependencies]
datetime = { git = "https://github.com/wavebitscientific/datetime-fortran", tag = "v1.7.0" }

However, since we want to move to a registry, and since most projects follow the convention of prefixing the version by v in the git tag (I do in all my projects, and I noticed you do too), I am proposing this natural extension of what Cargo does:

• If git and version is specified, it will checkout a tag named v + version from that git repository

So with this extension, you would do:

[dependencies]
datetime = { version = "1.7.0", git = "https://github.com/wavebitscientific/datetime-fortran" }

And then later once we move to a registry, you just remove the git field, but keep the version and it would do the right thing.

[everythingfunctional]

Cargo actually has a different meaning for this:

[dependencies]
datetime = { version = "1.7.0", git = "https://github.com/wavebitscientific/datetime-fortran" }

This means that when working in this package, it will download the latest master version of datetime, but for a package that depends on this one, the transitive dependency is version 1.7.0 of datetime from the default registry.

I'm not saying we definitely shouldn't do what you're suggesting, but if we do it will preclude us from ending up with the same functionality as Cargo. It would also impose a tagging convention on people's repositories that will be unnecessary once we have a proper registry. Also, relying on git tags for official release version identifiers is wide open for abuse.

[milancurcic]

Perfect, this is what we're after as a first step:

[dependencies]
datetime = { git = "https://github.com/wavebitscientific/datetime-fortran", tag = "v1.7.0" }

This alone will already be a huge improvement over the current state of things.

We can work out the nuances of version and registry behavior as we go.

[certik]

@everythingfunctional I am glad you noticed that. Let's not do my proposal then, just use tag. I think it is good enough and we will have a minimal registry soon anyway.

[certik]

@everythingfunctional, @milancurcic I think this issue is fixed now?

[milancurcic]

Yes, IMO this is good to close.

[everythingfunctional]

Yep, go ahead and close it.