depexts-plugins

Proposal for a plugin architecture for supporting checking and resolving
of external dependencies (depexts) in OPAM > 1.2
========================================================================

Rationale
---------

The opam package metadata now contains a specific field for declaring
dependencies on external packages handled through external package managers
(typically, distribution and OS dependent, but may in general be any other
package manager).

There are two main functionalities that are needed:

 - checking whether external dependencies are satisified at a given moment; this
   is an operation that can be implemented in linear time (we are just checking
   whether a boolean formula is true or false); since external packages are
   managed outside opam, this check needs to be performed at the beginning of
   each opam run, to discover packages that are no longer functional, and report
   the issue to the user.
   With proper, OS specific integration, this operation can be made blazingly fast;
   a simple hack, calling an external command each time, may be functionally equivalent,
   but quite slow.

 - finding a way of satisfying external dependencies required by a set of opam
   packages; this is potentially much more expensive, it involves not only a
   dependency solving phase, but also the fetch and installation phase, and
   requires proper interfacing with the existing OS specific package manager.
   This should be done only when modifying or fixing an opam configuration and
   after asking user confirmation.

Making things work smoothly and efficiently requires OS specific knowledge that
is best found among experienced users of each OS, which may be different people,
with different knowledge of Opam internals: a well designed plugin
infrastructure can separate concerns and facilitate contributions.


Proposal
--------

It is proposed to create a plugin architecture for OS specific external
dependencies, extending the following module signature for the modules
implementing a plugin

    module type OSDependencyLayer =
    sig
      type depexts   (* external dependencies, a CNF involvin OS-specific stuff *)
      type actions   (* an abstract token corresponding to actions, and a textual
                        representation of them to present to the user *)
      type explanations (* in case the depexts cannot be satisfied, explain why *)
    
      type result = Can of actions | Cannot of explanations

      type outcome   (* result of the execution of the OS-specific actions *)
    
      val satisfied : depexts -> bool    (* are the depexts already satisfied ? *)
      val cansatisfydepexts : depexts -> result
      val perform : actions -> outcome
    end

Notice that there are two distinct sets of functions for the very different
cases outlined above: 

 - satisfied performs just a check to see whether depexts are already satisfied
  
 - cansatisfydepexts tries to solve the external dependencies, and returns a proposed action,
   or an explanation for the failure, while perform executes the actions (typically
   after user confirmation)

The proposed module interface is intentionally incomplete, as it makes no assumption
on the way in which plugins are identified, and registered, which is an orthogonal
issue.


Note on OCaml detection
-----------------------

The OCaml compiler itself is an external dependency when using "system"
switches. It's currently handled by a specific, dynamically generated compiler
definition, with some ad-hoc code to treat it specifically, or check that it
didn't change at startup time.

With the current trend to move compiler handling to packages, the above won't
work anymore, because "system" would now need to be a specific, dynamic package.
While re-implementing the system switch hacks in this context would certainly be
possible, having the depexts mechanism flexible enough to handle all this
consistently would certainly be more consistent and easier to maintain.

Here is a possibility: having an 'ocaml-system' package (that would "provide"
ocaml) with depext on the system ocaml.
* the package needs to be able to export some environment variables that are
  currently exported by the switch (`CAML_LD_LIBRARY_PATH`).
* a change of this package should be detected at OPAM startup -- like for any
  depexts
* "system" compilers currently don't have to be managed by the OS, they are just
  looked for in the PATH. Keeping this would probably require a specific (lower
  level) "depext" plugin, that wouldn't have the functionalities to install the
  depext.
* this raises a new, but valid, concern: the above handles a binary state for
  depexts, while for this, we'd need to detect changes also. Creating one
  'ocaml-system' package version for each possible compiler version may be an
  answer: on system compiler change, the installed 'ocaml-system' becomes
  invalid, and you'll need to replace it by the fitting version (recompiling all
  dependent packages as you go).
* However, it sounds quite difficult to hold a middle ground between
  - "resolve with all OPAM packages installable, then check and handle their
    depexts", and
  - "check depexts, and then resolve with OPAM packages that can be installed
    with what's currently on the system;" (don't install them, except on
    conflict (how exactly?))
    
  and the above won't play well with first option here, second option raising
  many more questions.

Maybe this doesn't fit well with depexts, but it's worth considering