Variant selection mechanism for virtual libraries

[TheLortex]

The issue that was tracking this is #1859. I've been able to implement the variant selection mechanism along with default implementations in lib.ml, within closure construction.

It adds some syntax in dune files: for libraries, an implementation can specify its variant and a virtual library can specify its default_implementation. For executables, it's possible to give a set of variants to match.

What it does is the following: when the closure is computed, a map from virtual libraries to implementations is also built. Then variants are resolved and after that there's only virtual libraries with default implementations remaining unsolved: here we need to check for potential ambiguity while finding out which default implementation we can safely choose to add in the closure (and it loops until everything is resolved.).

This is not ready for a merge but we can start reviewing code and thinking about nice error messages !

[rgrinberg]

Will try to give this a review today.

[rgrinberg]

Looks good so far. Great work. I think that we'll be able to get this in for 1.8. From my brief look so far:

The test suite needs a bit more work. We need to add a test that witnesses the changes to the dune-package format. We also need some tests for the case where variants are loaded from an external package. Also, I regret the pattern of stuffing all the variant tests into a single run.t. It really slows down the test suite, so we should not do it. I propose that we split the test suite into:

• virtual libraries (basically my old tests)
• default implementations
• variants

I will do a more thorough review tomorrow

[rgrinberg]

Also: please rebase the PR when you have the time. The documentation and change log need to be updated as well

[rgrinberg]

Okay, I refactored the tests myself. It's a mess that I caused anyway.

[rgrinberg]

I've added some formatting tweaks:

• Reformatted everything with ocp-indent (please use it as well)
• Wrapped to 80 chars
• Remove most spurious uses of let .. and. If there is no recursion, let's just stick to normal let bindings.

[rgrinberg]

This can't be right I think. db is None in some cases. Why can't it be None here?

[TheLortex]

I agree this is weird. I assumed the database was accessible when virtual libraries resolution was needed. We just need to be sure that variants is empty when db is None. It's correct for every case, except in lib.ml: resolve_user_written_deps_for_exe, where db is (Option.some_if (not allow_overlaps) t). I don't really know about that allow_overlap feature: when is it set to true ?

[rgrinberg]

Could you comment on the necessity of this new field? Why can't we just use all and iterate over all the public libraries this way for example? Is it because we want to avoid resolving them?

[TheLortex]

The goal was to avoid resolving them. Before coming to this solution I wanted to put possible implementations in the Lib.t type but as you mentioned after this introduces a dependency cycle on instantiation.

[rgrinberg]

Yes, indeed that your first attempt would have been ideal. Unfortunately, we will need forward references before it's possible.

Why is resolving the implementations a problem however? Are there situations where we just need the Lib_info.t? We can't really do much with this value apart from resolving it into a proper library.

[TheLortex]

It's not really a problem, I just tried to do a minimal amount of computation when evaluating variants: we only need to resolve the implementations that are actually selected.

[rgrinberg]

The issue with these Option.value_exn should really be solved by having a field such as:

; default_implementation : Lib.t Or_exn.t option

in the Lib.t type.

However, this isn't really possible to add in instantiate as it would introduce a dependency cycle when resolving the libraries.

@diml Would it be possible to create some sort of a forward reference in Lib.t to address this issue?

[ghost]

Ah, that's annoying. A forward reference is ok if it is filled immediately after the Lib.t value is created, otherwise it is going to be hard to reason about when it is safe to reuse these values between runs.

BTW, I've been wondering if instantiate wasn't too complicated. Instead of storing direct Lib.t values in Lib.t, we would just store a new Resolved_lib_name.t type. And the idea would be that going from Resolved_lib_name.t to Lib.t would be very fast so it would be almost like having a Lib.t directly, but on the other hand it would simplify this code.

[rgrinberg]

BTW, I've been wondering if instantiate wasn't too complicated. Instead of storing direct Lib.t values in Lib.t, we would just store a new Resolved_lib_name.t type. And the idea would be that going from Resolved_lib_name.t to Lib.t would be very fast so it would be almost like having a Lib.t directly, but on the other hand it would simplify this code.

That might work, but then we have to allow for cycles when resolving the Resolved_lib_name.t, right? Seems like it's the same problem. What's the semantic difference between Resolved_lib_name.t and Lib.t?

[ghost]

Nope, we wouldn't need to recurse for getting a Resolved_lib_name.t. The only information attached to a Resolved_lib_name.t would be that it exist. We wouldn't know whether it is hidden or not for instance. It's pretty much the same as storing Lib_name.t values in Lib.t, and doing a resolution everytime we want to access one of those, except that it is faster because we already did the job of resolving the name.

[ghost]

To be clear, the distinction is as follow:

• Lib_name.t: just a string
• Resolved_lib_name.t: a unique identification of a library, i.e. a lib name + a path
• Lib.t: information about a library

[TheLortex]

So where are we now ? Shoud I try to implement that Resolved_lib_name module and refactor the code ?

By the way regarding libraries forcing implementations, I was suggested that it might not be a sane behavior, so we could simply forbid that. What do you think ?

[rgrinberg]

So where are we now ? Shoud I try to implement that Resolved_lib_name module and refactor the code ?

I need to think about this change a bit more. @diml would this require storing the Lib.DB.t in the Lib.t? Otherwise, I'm not really sure what a resolved name means if we don't know which database it came from.

By the way regarding libraries forcing implementations, I was suggested that it might not be a sane behavior, so we could simply forbid that. What do you think ?

I think it's fine to forbid it.

@TheLortex Let me have a look at this patch again today. Sorry about the delay, but we really need some simplifications here. The code was already too complex even before this PR.

[ghost]

Well, resolved library names would be globally unique, so we would have a single db going from resolved library names to Lib.t values. In fact, it could be just a memoized function. It is basically the function that takes a unique way to identify a library and returns its definition by reading dune, dune-package or META files. Breaking things down this way would also help having more fine grained memoization: the library databases would just become string -> resolved named maps, which are much smaller than the current string -> lib maps. And individual library definition would have less dependency in memory.

[rgrinberg]

Why do we need the arguments to the Implemented_by and Too_many_impl constructors? I don't see them being used anywhere.

[TheLortex]

Indeed they are not useful. They were built for potential error messages but we can remove that actually.

[rgrinberg]

I've squashed the commits into one as it makes rebasing simpler. @TheLortex I've removed a few cases where libraries were compared by name. This isn't correct in dune and names aren't unique. In fact, I agree with Jeremie's current idea of adding an intermediate step for name resolution. So even though there are still some problematic parts in the code that rely on concrete library names too much, I think this is good enough for now. I will look into implementing Jeremie's proposal in a subsequent PR.

[rgrinberg]

@TheLortex Have another look at my code. I've tried to simplify some things.

For your Vlib_status type:

module Vlib_status = struct
  type nonrec t =
    | No_implementation
    | Implemented_by of t
    | Too_many_impl of t list
end

I'm wondering if it overlaps, a little too much with our:

      type vlib_status =
        | No_impl of Dep_stack.t
        | Impl of lib * Dep_stack.t
      type t = vlib_status Map.t

So instead of a Vlib_status.t Map.t, we could use vlib_status Map.t Or_exn.t.

Yes, our error handling is a little less precise, but at least it will be consistent everywhere. Moreover, cases like this are confusing to me:

        Map.foldi !virtual_status ~init:(Ok ([], []))
          ~f:(fun lib status acc ->
            match status with
            | Implemented_by _
            | Too_many_impl _ -> acc

Shouldn't we be treating Too_many_impl _ as an error there? It probably doesn't matter, but using the type that I've proposed should simplify some things I think.

Using Lib_info.t for the dependency is something that I wish we'd avoid. We already have a Dep.Stack.t type. This is just a minor point and doesn't prevent the PR from being merged, but is there a reason why you did it this way?

[rgrinberg]

@TheLortex Could you add a comment documenting vlib_default_parent by the way? I think we should have a comment for every single one of these refs that we use while traversing the closure.

[TheLortex]

Thank you for your work on this ! I took a look, it's indeed more readable than the first version.
Moreover, if we forbid libraries to depend (and thus force) implementations, we can actually remove the default implementation dependency cycle check.

Shouldn't we be treating Too_many_impl _ as an error there? It probably doesn't matter, but using the type that I've proposed should simplify some things I think.

The error is handled later, in the Virtual_libs module it throws Double_implementation so I didn't want to make a duplicate. I guess we could use vlib_status instead.

Using Lib_info.t for the dependency is something that I wish we'd avoid.

Could you add a comment documenting vlib_default_parent by the way?

If we forbid libraries to depend on implementations we can actually remove the whole resolve_default_libraries function.

@rgrinberg I'll update the PR accordingly

[rgrinberg]

I'm not sure we should forbid to depend on implementations. The reason being is that it makes it hard to turn existing libs into implementations.

Also non dune users can only consume implementations so I think we have to live with the check for now.

[rgrinberg]

@TheLortex The last commit replaces the custom map with Vlib.Table.Partial. Things almost seem to work except for the resolution_priority issue. Could you explain that test a little more?

[TheLortex]

Yes you're right. Let's keep the check then.

resolution_priority was a case where the three kinds of implementation selection were used:

• for direct it's an user defined dependency to direct.ocaml
• for variant it's selected by using variant c
• for test it uses a default implementation

It also checks that an user defined dependency overrides variant selection. Thus even if the c variant is given, direct.c should not be selected because direct is already implemented by direct.ocaml.

This allows to override variant selection in case of ambiguity.

[rgrinberg]

I've removed the Vlib_status stuff in favor of just keeping unimplemented libraries in an abstract type. I think this is good enough to merge now.