Supporting dependencies on packages in specific remotes

[lpw25]

There have been various comments and requests regarding opam's workflow along a similar theme. This feature request is aimed at addressing these comments.

At the moment, you can share packages in two ways:

1. Create a remote containing the metadata for the packages
2. Put an opam file in the package so that people can pin it directly.

Managing a remote is intended for stable released packages, whilst pins are intended for use during development.

However, pins can fall a little short sometimes. In particular, if Alice's package foo depends on using Bob's unreleased package bar then it becomes quite difficult to use pins. Alice cannot specify in the description of foo that it depends on the unreleased version of bar at "github.com/bob/bar#next" -- which she had pinned whilst developing foo. This has lead to people e-mailing around list of pins required to build their project which is not ideal.

Of course, Alice could create a new opam remote containing metadata for foo and bar which would enable people to build foo. But this becomes more painful as the size and depth the dependency tree of unreleased packages increases -- you must create a single repository containing metadata for the the whole tree.

The theme of the aforementioned comments and requests is roughly that they like the decentralised workflow of package managers like npm but, due to the above issues, they are unable to replicate that workflow with opam.

As a possible solution, I propose the following: allow dependencies on a package from a specific remote rather than from the remotes installed on the users opam root. So, in the metadata for foo, Alice can write the dependency "github.com/bob/bobs-remote::bar" to express a dependency on the bar package in Bob's personal repo (at github.com/bob/bobs-remote).

I do not have a working knowledge of the semantics of dependencies, remotes and universes in opam -- so I do not know how feasible this is or what would be required -- but assuming it is feasible to give such things a reasonable semantics then I think it would help to address the requests for an "npm-style" model.

An alternative to this suggestion would be to allow direct dependence on packages at a given address rather than via a remote. I think that would work in the small, but keeping a package's metadata separate from it makes the system more robust and remotes are a good way of doing this. Remotes also provide a good focal point for tooling (e.g. automatic generation of documentation containing cross-references between all the packages on the remote).

[AltGr]

#2762 has some ideas for handling the same use-case.
I'll examine the possibility more carefully, bit in hindsight it seems to me that allowing packages to refer to repositories would defeat part of the purpose of configurable repositories — wouldn't package definitions and repositories become mutually recursive ?

[lpw25]

Yeah, possibly it's not the right approach. Although I don't see any obvious issues with it.

I'm very keen on promoting people having personal or organisation-based remotes rather than the more ad hoc pinning approaches because I think the separation of packages from their metadata makes things more stable, and because they are a good focus for tooling (e.g. it shouldn't be long before we can automatically generate a documentation page for a whole remote).

I'm not sure I will be able to convince people to sacrifice the convenience of the "pin to a git commit" approaches for these benefits, but I think it is worth investing some effort in making remotes easier to maintain in order to encourage their use.

An alternative to the above proposal might be to have more tools for easily managing a remote. For example, a command for "pulling" from one remote into another might work. So you could just do:

cd my-remote
opam admin pull http://github.com/janestreet/opam-repo
git commit -m 'Pull from janestreet'
git push

to bring the latest janestreet versions into my remote and allow me to depend on them.