Attempt to guess correct libdir from current exe

[gridbugs]

In cases where the libdir computed by ocamlbuild -where doesn't contain the expected ocamlbuild library files, attempt to guess an alternative libdir relative to the currently running executable's path (using Sys.executable_name). If the resulting libdir does contain the expected ocamlbuild library files then a warning is printed and the guessed libdir is used.

The motivation for this change is to allow the ocamlbuild (opam) package to be built by dune's package management features. Dune builds opam packages by executing their build and install commands in a temporary sandbox environment, and then copying the resulting artifacts to a final installation path. This creates a problem for ocamlbuild as the OCAMLBUILD_LIBDIR variable set at build time will be a path inside the temporary sandbox. That path is then used to compute the value printed by ocamlbuild -where which consequently reports a non-existant path.

This is an attempt to satisfy dune's constraint that a package's installed files must work after copying them out of their original install location.

We discussed this a little in #326. This is intended to fix this issue: ocaml/dune#10290.

[gridbugs]

Closing this in favour of maintaining a fork of ocamlbuild in Dune's opam-overlays repo. The dune developers intend to maintain forks of a small number of widely used packages whose built artifacts do not work correctly after relocation from dune's build sandbox.

[kit-ty-kate]

Why close it though? Relocatable OCaml is coming anyway and it has nothing to do with dune. This PR only needed some regular pings for reviews (i personally don't have time right now but might in the future, but i'm not the only maintainer)

[hhugo]

We could also make ocamlbuild depend on findlib and no longer hardcode theses paths

[kit-ty-kate]

that sounds like a good idea to me

[gridbugs]

Ok fair enough! I'm happy to leave it open but I don't have a lot of time to commit to working on this in the near future.

[kit-ty-kate]

no problem!

[gasche]

I have not done much on this PR because I find the proposed use-case odd, if not broken. The standard software configure-build-install is to configure software with information on where it is going to be installed, and then build and install it. Dune broke this convention in its internal package-manager implementation, by providing installation locations within its own sandbox. (@gridbugs explained this well in ocaml/dune#10290 (comment) ). I would naively think that trying to support this convention is the logical next step, but in fact dune maintainers decided to special-case packages that assume it; this closed-world approach is likely to make it difficult in the future to package software on opam-repository that use different build systems than dune, which I find worrying given dune's difficulty in building non-OCaml software. I am not very motivated to spend time supporting this approach, and I was sort of hoping that dune would eventually choose a different path.

The PR itself is supporting relocatability, which (solves dune's problem and) is something that I find reasonable in general. The thing that would give me pause is that we keep adding layers and layers of complexity to this logic, and we never remove or simplify the previous layers away. (We are not motivated to simplify the code because we want maintenance-only changes and simplifications have a risk of regressions, but the result of this approach is to increase complexity and maintenance harder over time.)

Is there a "clean" approach to write relocatable OCaml programs these day, even if they perform some sort of dynamic linking? (ocamlbuild needs to locate its own libraries at runtime to build plugins.) The PR implements the approach of guessing the libdir from the bindir, going to ../lib/ocamlbuild from the location of the binary; this works for the standard opam install layout, but I would worry about other filesystem organizations (for example ocamlfind by default puts things in a /lib/ocaml/site-lib/ subdirectory). Does the dune sandbox environment respect this convention?

Calling ocamlfind query ocamlbuild at runtime may be the easiest way to find this out, because giving library directories for packages is precisely its job -- but again, is the dune sandbox environment correctly configured to make this work before ocamlbuild is actually installed?

[hhugo]

Related to dra27#2

[gasche]

That is a different PR from @dra27 that also makes ocamlbuild relocatable, on top of his work to make the OCaml compiler itself relocatable. I asked David a few days ago (during the github/ocaml triaging meeting) about this general question of how to make OCaml programs relocatable, and my worry that assuming a fixed relative path from bindir to libdir is not expressive enough to work with diverse packaging choices. For OCaml, David made the bindir->libdir relative path configurable (at configure-time), which is flexible enough to capture many use-cases. This would probably also be a fine approach for ocamlbuild -- provide a configure-time choice for the bindir->libdir relative path.

My impression is that this idea of configurable relative bindir->libdir path should be enough to support the systems that we currently support well, possibly with an explicit opt-out to set libdir at configure and then at runtime to cover weird cases (I'm not sure how things work on Nix). The difference relative to the present PR would not be very large -- but I would be interested in trying to remove some of the older stuff to simplify our lives.

(The patch of David does something different, using new primitives of the compiler introduced by the Relocatable PRs, and I think it deserves its own design explanation and discussion.)