Private libraries attached to a package

[rgrinberg]

This PR implements the (package ..) field on libraries that lack a public name. For example:

(library
 (name foo)
 (package bar))

This makes foo available to the public libraries in bar but not to public libraries in any other package. foo will be installed as bar.__private__.foo. The name mangling encourages users to avoid using this library name directly.

I have a some questions about the implementation to @emillon and @jeremiedimino:

Initially, we agreed that the cmi's of such libraries will not be installed. I'm not entirely sure this is workable however. Consider this setup with 2 libraries and an executable in a single project

(library
 (name foo)
 (package bar))

(library
 (libraries foo)
 (public_name bar))

(executable
 (name baz)
 (libraries bar)
 (package baz))

Suppose that (implicit_transitive_deps true) is set. Do you expect the executable baz to see the cmi's of foo? If you think the answer is negative, then consider the situation if the executable was now:

(executable
 (name baz)
 (libraries bar foo)
 (package baz))

This definition is perfectly reasonable with the private libraries we have today, but to make it work with a package-private foo, we need to make sure the installed cmi's of foo are used. That's because bar is installed and we must build against the same version of foo that bar used.

Alternatively, we could simply disallow the usage of such libraries by executables from other packages. Though i think this is a serious limitation.

Progress

• Mangle names of libraries when installing them
• Make private packge names visible when they are hidden by -p.
• Allow libraries in the same scope to refer to such libraries.
• Hide cmi's of such package private libraries from findlib users
• Install cmi's of package private libraries into a separate directory
• Hide cmi's of such package private libraries from users outside of the scope.
• Document package private libraries
• Restrict it to 2.8 and above.
• Fix overlap checks
• Implement name mangling for package private libraries.

[ghost]

Alternatively, we could simply disallow the usage of such libraries by executables from other packages. Though i think this is a serious limitation.

We agreed to keep this limitation on slack the other day. So let's do that unless there is a new argument in favour of lifting this restriction.

[rgrinberg]

We agreed to keep this limitation on slack the other day. So let's do that unless there is a new argument in favour of lifting this restriction.

Not exactly. We talked about the limitation for libraries. I.e. we agreed that library x cannot depend on a private library y in package foo, unless x is also in foo. That seems fine because it was completely impossible for public libraries to depend on private libs, so there was no limitation introduced. However, for executables, private libraries that belong to a package are now strictly less powerful. As normal private libraries can be used in executables freely.

Anyway, I'll introduce the restriction for executables as well then. I don't think it's such a big deal for v1.

[rgrinberg]

So let's do that unless there is a new argument in favour of lifting this restriction.

Actually, there's a new argument: I don't know how to lift this restriction.

The issue with executables is that it's not easy at all to know which package they belong to. The following case is trivial:

(executable (name foo) (package bar))

But what about this one:

(executable (name foo))
(install (section bin) (package bar) (files foo.exe))

[ghost]

The executable case is more convincing. Agreed that this is not ideal.

If we install the cmi files, then adding (package foo) to private library lib becomes exactly the same as adding (public_name foo.__private__.<libname>), right?

[rgrinberg]

Almost. The cmi's for the private library aren't going to be visible if you use them in an executable/library that lives in another project. Since these cmi's are going to live in a separate dir.

In fact we should probably disallow the mangled names from being written by the user altogether. We really only need them for compatibility with findlib anyway.

[ghost]

The cmi's for the private library aren't going to be visible if you use them in an executable/library that lives in another project.

I can see how that would work in a mono-repo, but how would that work with opam? If we consider this example:

(library
 (name foo)
 (package bar))

(executable
 (name baz)
 (libraries foo)
 (package baz))

Once package bar is installed and we are compiling package baz though opam, so with -p baz, how will Dune know that bar is part of the same package?

In fact, there is another problem: when building package baz, library foo will be hidden given that it is part of another package. So the library name foo will not exist. How will Dune resolve this name?

[rgrinberg]

Once package bar is installed and we are compiling package baz though opam, so with -p baz, how will Dune know that bar is part of the same package?

In fact, there is another problem: when building package baz, library foo will be hidden given that it is part of another package. So the library name foo will not exist. How will Dune resolve this name?

We could change how -p works. Before filtering any stanzas, we need to construct a map of packages to library names for every dune-project. My only concern is that it will somehow harm performance in some unexpected way. But I can't think of how because we already need to resolve all the libraries in a scope before evaluating rules.

Alternatively, we could require the user to qualify such dependencies by the package:

(executable
 (name baz)
 (libraries (package bar (foo))
 (package baz))

This will only be allowed if bar and baz share the same scope.

[ghost]

We could also do what we did for variants: instead of filtering we replace the stanza by something like (library_alias (name foo) (libraries bar.__private__.foo)).

Note that there is also a conceptual difference with the original proposal: with the original proposal the developer has a stronger guarantee that its users won't try to use the private libraries. Here, we only get the guarantee for users using Dune, but someone using ocamlfind could happily go and use such private libraries.

[rgrinberg]

We could also do what we did for variants: instead of filtering we replace the stanza by something like (library_alias (name foo) (libraries bar.private.foo)).

Yeah, that seems good.

Note that there is also a conceptual difference with the original proposal: with the original proposal the developer has a stronger guarantee that its users won't try to use the private libraries. Here, we only get the guarantee for users using Dune, but someone using ocamlfind could happily go and use such private libraries.

It wouldn't be enough because using that name in findlib would only make the object files visible. The.cmi files would live in a sub directory. They would need something hacky like ocamlfind ocamlc -I $(ocamlfind query bar.__private__.foo)/.private. If users are willing to go through this trouble, we might as well let them.

[ghost]

I see. Yes, in this case that all looks good.

[rgrinberg]

@bobot this PR is ready for review.

I will add name mangling in a separate PR, as I'd like to do a bit more refactoring + work on the (formerly named) extra_aliases first.

[bobot]

That's great. It simplifies a lot the mental model of what is possible or not and open a lot the possibilities.

[bobot]

Just why do we accept for executable of another package but in the same project to depend on a private library and not for libraries?

[rgrinberg]

Just why do we accept for executable of another package but in the same project to depend on a private library and not for libraries?

Libraries may depend on any private library inside the project as long as it has (package ..). Previously, we tried to limit it to the package, but lifted this restriction exactly for this reason. We wanted it to be symmetric for executables.