Dune Site Plugins do not work in toplevel

[ivg]

Expected Behavior

It should be possible to load plugins in the toplevel.

Actual Behavior

There are actually two independent problems.

Problem 1

First, the dune-site.plugins library is incomplete as it misses the Dune_site_data module.

Reference to undefined global `Dune_site__Dune_site_data'

Workaround/Solution

This module is generated at link time when an executable is linked, so to make it work, we can link our own toplevel, and we just need to add dune-site.plugins to the libraries stanza that will force dune to generate this file. Once, the toplevel is linked (I am just using let () = Utop_main () but one could run the vanilla toplevel using Topmain.main ().

While this problem has a workaround it would be nice to have the ability to generate self-contained libraries that are loadable in runtime. I didn't test it, but it might be that using the shared_object mode of an executable will build such a library. I will update the report if it will work.

Problem 2

Now to the main problem. When we try loading plugins from our toplevel, every plugin fails with,

(Invalid_argument
 "The dynlink.cma library cannot be used inside the OCaml toplevel")

Indeed, dynlink is not supposed to work inside the toplevel, which provides its own dynamic loading facilities. Therefore, the plugin system of dune has to check if we are in the interactive mode (in the toplevel) and use Topdirs.dir_load instead of Dynlink.loadfile.

Solution

I have a solution that I will PR in a second (the link should appear below). It has some things to discuss. Especially from the usability point of view. But at least it allows me to load all BAP plugins in a toplevel without any issues.

[ivg]

Update on Problem 1

I was trying to build a self-contained .cma plugin that could be loaded into OCaml top-level as a solution to a more specific problem - trying to load a library that depends on link-time generated code. After hot-fixing #6086, it turned out that linking the link-time generated code into the library, which depends on a library that requires this link-time generated code, doesn't help. Obviously, since the dependency is loaded before our plugin so we fail long before we load the plugin with the missing code. I tried to build a fully self-contained plugin, but the embed_in_plugin_libraries stanza neither sorts the specified libraries in the topological order (which looks like a bug)¹ nor it walks through the transitive closure of dependencies. The latter might be the desired behavior, but in general it makes this feature hardly usable. E.g., I ended up manually discovering the depdencies, so my link list looks like this,

 (embed_in_plugin_libraries
   dynlink
   ordering
   dyn.pp
   dyn
   stdune.filesystem_stubs stdune.csexp
   stdune
   dune-private-libs.dune-section
   dune-private-libs.dune_re
   dune-private-libs.meta_parser
   dune-private-libs.ocaml-config
   dune-site.private
   dune-site dune-site.toplevel dune-site.plugins findlib.dynload
   monads
   bap-plugins
   bap-bundle bap-main bap)

Which is obviously counter-productive and should never be done in real life, so I forfeited this idea. And this brings us back to the main issue - is that libraries that require link-time code do not work in OCaml toplevel. Maybe a simpler solution would be to provide for such libraries a sublibrary <package>.top that will have the generated code already linked? And the instruct ocamlfind via meta to load this library, when toplevel is used? This is quite a killer feature, to be able to use OCaml toplevel and denying it to libraries that use dune-sites is a big problem. Or maybe I am missing some obvious solution?

---

¹⁾ So the order in which libraries are enumerated matters, which is counter-intuitive, people would expect that it is the same as in the libraries stanza, i.e., that they can specify it in the arbitrary order. If you think that this is a bug, I can open a new issue.

[nojb]

cc @bobot

[ejgallego]

@ivg not sure if that's your problem but after OCaml 4.08, the linker will reject to link twice a module, so indeed the loader itself (as in Fl_dynload) must keep track of the deps and loading order as to ensure diamonds in the dep chain are properly shared among all possible loading setups.

[ejgallego]

The classical example is plugins A B both depending on plugin C. C must be loaded once, thus you can't link C on neither of them.

[ejgallego]

Before 4.08 OCaml allowed this so a lot of people got away with doubly-linking.

[ivg]

@ejgallego, not it is not the problem. We're maintaining our own plugin system for nearly a decade, back in the times when dynlink was unsound, so am pretty aware of it). But this issue has nothing to do with Dynlink or Fl_dynload, but with a different dynamic loading facility that is provided by the top-level. Dynlink and findlib.dynload work only in native or bytecode, but not in the top-level mode.

[nojb]

Dynlink and findlib.dynload work only in native or bytecode, but not in the top-level mode.

Sorry for the naïve question, but couldn't #require be used to load a plugin into the toplevel?

[ivg]

Sorry for the naïve question, but couldn't #require be used to load a plugin into the toplevel?

Well, when a dune plugin is installed into a path that is known by ocamlfind and if the plugin name is known, then it could be manually loaded with the #require directive (which is provided by ocamlfind). Of course, it won't work from other sites (like when the plugin is not installed). And it will load only one plugin, not a hundred plugins. And this is if we will put aside problem 1. With problem 1 you can't load in top-level (using #require or #load, doesn't matter) any library that depends on link-time generated code because those libraries have unresolved symbols (that are supposed to be generated during an executable linking). This is another big problem and unlike problem 2, which is solvable (see the #6082) I don't have a solution for this problem.

[ivg]

Also, using dune ocaml top as well reveals problem 1.

$ ocaml
OCaml version 4.14.0
Enter #help;; for help.

# #use_output "dune ocaml top";;
File "(command-output)", line 1:          
Error: Reference to undefined global `Dune_site__Dune_site_data'