feat(oxcaml): instantiate parameterised libraries

[art-w]

Follow up on #12392 and #12336: This PR adds the ability to instantiate parameterized libraries in library/executable dependencies. This feature requires compiler support which is only available in OxCaml atm.

In stanza dependencies, the new form (my_lib arg1 arg2 ... :as new_name) specifies that the parameterized library my_lib should be instantiated with a list of arguments (which must implement the parameters requested by the lib). The arguments are identified by the parameter interface they implement, such that the order in which they are provided doesn't matter. (It's nominal not positional. We sort them internally in dune, but only to ensure we don't duplicate efforts to instantiate both (f x y) and (f y x), since they are the same.)

The optional :as new_name allow renaming the instantiation in user code, which is useful when the same library is instantiated multiple times (otherwise if missing, by default My_lib is used as the name of the instantiation). In the alias file, each instantiation is translated to a module New_name = My_lib(Param1)(Arg1)(Param2)(Arg2)(* ... *) [@jane.non_erasable.instances]. (Note that each ArgN implements ParamN, i.e. (Arg1 : Param1), so this is not a real functor application, hence the OxCaml attribute.)

While libraries can do a partial application of their parameterized dependencies (as long as their list of parameters is a superset of missing dependency parameters), executables must fully instantiate their dependencies. Computing the transitively instantiated libraries requires passing the applied arguments to each of parameterized dependencies that requires this parameter. This impacts the existing transitive closures performed by dune, since now the same library may be required multiple times but with different arguments.

To link the executables, all the (transitively) instantiated libraries must also be built by the compiler with the new OxCaml command ocamlopt -instantiate mylib.cmx arg1.cmx arg2.cmx ... -o mylib-Arg1-Arg2...cmx. Here a partial instantiation is not allowed (all arguments must be specified). Furthermore, this should be done for all modules of mylib, in their topological order. We create an archive at the end to package everything.

Instantiated modules use dashes in their filenames to "parenthesize" their arguments, which can themselves contain dashes if they are the result of another instantiation. The number of dashes indicates the depth of the application. For example, a filename f-g--h---x-y corresponds to the application F(G(H(X)))(Y). We reuse this idea for the name of the folders where we instantiate each module of a fully instantiated parameterized dependency. A dependency (mylib arg1 arg2) (fully applied) is instantiated in the _build subfolder .parameterized/mylib!arg1!arg2. We can't use dash separators here because it's already allowed in library names and would be ambiguous, so instead we arbitrarily picked the exclamation mark instead... but let us know if there's a better choice!

Finally for opam installations, we export all the dependencies arguments (but not their concrete instantiation, since this will done by user executables when needed).

While I still have a couple of TODOs to address minor issues (or so I hope), this PR seems to be working... so I would appreciate an early review to check I haven't done anything stupid :)

Fix #12088

[rgrinberg]

This ~dir is always the root of the workspace, in which case the scope is just the current project. Is that correct?

[art-w]

I believe so yes, the parameterised libraries are always instantiated at the root _build/<profile>/.parameterised so ~dir is always the root. Should I be using something else to get the scope? (or is there an issue with using the root dir?)

[rgrinberg]

The scope at the root will only be able to look up public libraries and private libraries of the project root. I think that's probably incorrect if some instances are private?

You should be able to demonstrate that this doesn't work in a test where:

$ ls
dune-project
a/
$ ls b/
dune-project

And project b/ needs some private instance instantiated.

In the jsoo rules, this isn't a problem because we only generate global rules for public libraries. But if you look at the ppx rules (ppx_driver.ml), you will see that we are careful to encode the project id into the identifier in the path.

[art-w]

Oh thanks for the clarifications and the ppx_driver pointer! I've extended the test of vendored parameterised libraries with private libraries and private instantiations, to show that the scope is correctly handled :)

[rgrinberg]

Looks like you're going in the right direction. There's a massive amount of code here, and I couldn't go thoroughly through everything in one go. I'll try do another pass soon.

I'm a bit concerned about all the somewhat similar but not quite the same rules in parameterized_rules.ml. Do you intend to factor this out in a better way with the rest of the rules?

[art-w]

Thanks for the review @rgrinberg! Would you have time for another round? :) I believe all the major issues have been fixed, there's more tests and documentation, and I've integrated feedback from @Alizter too.

The only pending issues I know of is with unwrapped libraries, which currently work but support could be better. The pre-existing bug #6148 is easy to trigger, and the automatic generation of module X = Lib(Param)(Impl)[@jane...] is missing due to a lack of alias file. Unless I find a short fix or support is critical, I think it's better to follow-up in another PR specifically for unwrapped libraries than grow this one further.

I'm a bit concerned about all the somewhat similar but not quite the same rules in parameterized_rules.ml. Do you intend to factor this out in a better way with the rest of the rules?

Are you thinking of the build archive rule or something else? We looked at it with @Alizter and it's unclear that this can be simplified easily without refactoring Module for instantiated artifacts (and at the PR is already big, I would again prefer not to introduce more changes here!)
Did you have something else in mind to better fit with the existing rules?

[rgrinberg]

Sure, will do another round this week.

[shonfeder]

@rgrinberg is that a request for particular work on this PR, or for followup documentation work? Just asking because it is not obvious to me whether this PR needs additional work, is currently still under review, or is in an inbetween state. :)

[rgrinberg]

I'd like to see the following in this pr:

1. Tests and comments for the current set of limitations. They would make it much easier to pick up the work for anybody in the future.

2. Address the comment about the scope that is used for looking up instances.

Sorry if I wasn't clear.

[art-w]

Thanks! I believe the scope issue is now fixed, with a test to show that private instantiations work.
Regarding unwrapped libraries, I've extended the documentation to explain the limitations and this test has a TODO item associated with lifting them.

[rgrinberg]

By the way please user your full name to sign commits. See https://github.com/ocaml/dune/blob/main/CONTRIBUTING.md

[art-w]

Thanks @Alizter for the review! The PR should be all clean now, rebased and squashed with my full name for the sign off :)

[Alizter]

I will merge this later today. One thing I've noticed is that there is slippage between the spelling of parameterised and parameterized. It doesn't bother me too much but it's better to be consistent. I think in dune we have tended for American English in the code and British English in the docs.

Do you know what the correct one should be? We don't have to fix it in this PR.

[art-w]

We decided to follow the British spelling used by OxCaml, with an S, so I've corrected the few instances left of parameteriZed to keep everything uniform:

$ ocamlc -config | grep param
parameterised_modules: true