Try install without depext at least once

[toots]

We recently had to fix a depext on libopus and noticed that opam 2.1.0 will keep failing even if the depext is installed with the fixed name. This is not really robust as depext do change over OS release. Maybe --assume-depext should be implicitly tried at least once?

[rjbou]

opam 2.1.0 will keep failing even if the depext is installed with the fixed name

You mean that conf-libopus is up-to-date with libopusenc and when opam tries to install it, it fails?

[toots]

No, conf-libopus was referring to the wrong OS package but the local system had the right depext already installed.

[dra27]

I think you'd want --no-depexts here? If the depext has changed name, then --assume-depext wouldn't be enough (because opam would still be checking to see if that package exists)?

I don't think we should automatically do things which wallpaper over issues on error - it makes opam more surprising. In this case, it also makes it slower - opam's always going to be checking for a non-existent depext and then carrying on until someone is annoyed enough to investigate why and fix the actual issue.

We'll discuss it further, but I think there are three things we could/should do instead:

• If it's not already, the error message when failing to install should clearly mention the --no-depexts option (but I think it does?)
• if opam is in interactive mode then it could offer to re-run with --no-depexts (but as I said, I don't think this should be automated - the underlying problem here is that the metadata are wrong, and the metadata should be fixed).
• The testing carried out on opam-repository is always expanding - we're on the cusp of doing regular health-check runs on macOS (and then Windows), so I would hope that we'll be able to catch these depext changes either just as, or even before, our users

[toots]

I understand all that but, at the end of the day, there are multiple ways of installing external dependencies. Compiling manually, installing via a package manager, potentially one of many.

On macos, typically, one can install via brew or macports.

Also, some dependencies can have different package names, for instance libsrt on debian has two variants, one with ssl and one with gnutls.

You also have cases where two copies of the same external dependencies are installed, for instance one for x86 and one for x64 or a cross-compiled one.

There is no way of knowing in advance which package manager and package name (if any) should be used for any given situation.

It is really good that opam has support for external dependencies, it is an incredible option and makes everything so easy to use, no doubt about it.

However, one should also think about usability from the user perspective and not tie the user to a specific package manager and/or a specific package name.

The most basic situation is to try and install a OCaml package without making assumption about external dependencies. Then, if that doesn't work, it seems okay to make an educated guess and try installing an external dependency.

Even better would be if, at this stage, an interactive question is prompted to the user:

opam is not able to install your package at the moment. It might be that you need to install the following dependencies: .... Would you like us to do it for you?

That would be a great user experience.

[dra27]

Indeed - all these scenarios are supported by opam (via either using --assume-depexts for selecting a specific depext where there's a choice, --no-depexts if you're building from source, manual configuration of the global variables to select the correct depext system to flip between macports and homebrew, and even just turning off automated depext support entirely). The thing we disagree on is that opam should not iterate through these configurations automatically - it should carry out the instruction it's been configured for, and then aim to display helpful error messages if it doesn't work. The user knows (or really needs to know!) which of these paths will be the correct - opam has a good default which the user can choose to override but opam should otherwise do as it was told and just aim to be as helpful as possible in reporting what to do when what it was told to do doesn't work.

[toots]

Well, I just think that, again, from a user perspective, expectations like:

until someone is annoyed enough

or:

The user knows (or really needs to know!)

usually lend themselves to bad users experience.

Not all opam users are OCaml developers or developers at all.

With liquidsoap, typically, we direct our users to install the application via opam precisely because it can handle their external dependencies.

However, some, if not a lot of them, usually do not have a deep understanding of the functioning of their OSes. That's totally normal, they are trying to run a media stream not be devops engineers.

For those users, I would advocate that opam tries first to do things assuming everything is in order, including assuming depexts are already present.

Then, if anything is missing, this is the right opportunity to display a relevant and informative message to the users, something like:

We were not able to install the following package(s): xxxx. This(these) package(s) has(have) external
dependency(ies).

Based on the detected OS and system as: <OS>, <system>, we believe that installing this(these)
external package(s) should fix the issue:
  <packages>

Opam can do it for you using the following command:
  <insert command>

Should we proceed?

This will definitely enhance the user experience as it tells them what has been tried, what seems to be missing and what opam is trying to do to fix the issue.

[toots]

Then, of course, you can make a --with-depext or similar option that advanced users can use to automatically install external dependencies when they know what they are doing.

[dbuenzli]

Pesronally I think that from a user perspective an expectation like:

Not all opam users are OCaml developers or developers at all.
[...]
With liquidsoap, typically, we direct our users to install the application via opam precisely because it can handle their external dependencies.

is bad user experience. If I was a liquidsoap user I would rather be annoyed to have to master yet another language specific package manager to install it since AFAIK OCaml is purely an implementation detail of liquidsoap.

[toots]

Are we considering that opam is a package manager only dedicated to OCaml developers?

[dbuenzli]

Some other communities like coq use it like, so I would rather say it is a developer tool. I don't think it's suitable to install applications, especially for people who are not developers at all.

I think the best experience for your users is to provide them the ability to install liquidsoap with their system package manager. (Having to learn something entirely new just to be able to install liquidsoap is a barrier to your software)

[dbuenzli]

Did you have a look into things like https://github.com/AltGr/opam-bundle ?

[toots]

Some other communities like coq use it like, so I would rather say it is a developer tool. I don't think it's suitable to install applications, especially for people who are not developers at all.

I think the best experience for your users is to provide them the ability to install liquidsoap with their system package manager. (Having to learn something entirely new just to be able to install liquidsoap is a barrier to your software)

I have packaged for Debian for years and was involved closely with the OCaml packaging there so I can say with some degree of confidence that system level packaging is a poor solution to help users install complex OCaml applications.

opam has proven to be a fantastic tool. It handles the complexity of:

• Resolving ocaml module dependencies and upgrading them to the latest compatible version while rebuilding all transitive dependencies
• Installing optional dependencies and supporting optional features
• Installing external dependencies

Going back to system-level packaging for each of these features is a step back in user experience and also in developers burden since this means packaging, updating and rebuilding every single ocaml dependency and their dependencies for each different OS-specific package manager.

That's not to say that we don't do it, we provide Debian, Ubuntu, Docker and Alpine packages for a plethora of versions and architectures but, when it comes to giving users a single, unified way of installing the application with the latest ocaml dependencies and optional features, opam clearly provides a much, much better user experience.

At the end of the day, I think @dra27 and myself are not far appart, we both want opam to do what it's told by default, my point is that, to be more inclusive, I believe we should assume that the user installing a opam package has no idea about external depencies and should prompted about what opam is doing when handling so.

Otherwise, this keels akind to having your tech savvy friend seating next to you telling you: "don't worry dude, let me handle this", leaving you with a total sense of confusion about what just happened.

I believe that we should take all opportunities to explain what is being done internally when specific decisions or assumptions are being made by any software, give the user a friendly, explanatory message about what and why this is being done and use this opportunity to give more clarity and give the user a chance to learn and understand better what is going on.

[dbuenzli]

I think that most of what you mention comes from someone who knows and uses opam – which is indeed a fantastic tool.

But don't underestimate how off-putting it can be. If I'm interested in setting up a internet radio, I couldn't care less about opam. And let's not pretend it's not going to be a weird install, living in ~/.opam, with runes added to my shell rc or invocations of eval $(opam env). If I was not an opam user I would just hate that.