Add a test about dune-sites where plugin links extra libary.

[Kakadu]

For now this test gives a dynlink error about missiing library. See also
#4320

@bobot ^^

[Kakadu]

@bobot Any change this can be added to 2.9 milestone together with #4212 ?

[Kakadu]

@bobot Did you manage to find some free time to look at it?

diff --git a/../../../default/test/blackbox-tests/test-cases/sites.t/run.t b/test/blackbox-tests/test-cases/sites.t/run.t.corrected
index 174ce491e..c0da368a0 100644
--- a/../../../default/test/blackbox-tests/test-cases/sites.t/run.t
+++ b/test/blackbox-tests/test-cases/sites.t/run.t.corrected
@@ -1,5 +1,7 @@
 
   $ dune build ./app.exe @install
   $ dune exec ./app.exe
+  Fatal error: exception Dynlink.Error (Dynlink.Cannot_open_dll "Dynlink.Error (Dynlink.Cannot_open_dll \"Failure(\\\"/home/kakadu/.opam/4.12.0+options/lib/ocaml/unix.cmxs: undefined symbol: camlStdlib__callback__register_exception_16\\\")\")")
+  [2]
 

[bobot]

The problem is that a plugin depends on threads and not the main program. It is a limitation of the OCaml runtime. The only fix currently is at runtime to not try to load threads, and raise a good error instead.

Sorry for the delay.

[bobot]

I made some modifications directly to the branch: moving the tests, adding a working version

[ejgallego]

Umm, does this mean that plugins are now always linked with -linkall?

Isn't that a problem in OCaml >= 4.08?

[bobot]

Indeed I still added the -linkall as for findlib.dynload. @ejgallego Why do you think it is a problem for OCaml >= 4.08?

[ejgallego]

Indeed I still added the -linkall as for findlib.dynload. @ejgallego Why do you think it is a problem for OCaml >= 4.08?

I always get confused with the meaning of -linkall in different contexts [such as -shared].

Does the use of -linkall here mean that .cmxs files will include all the libraries they refer to?

[ejgallego]

To be clear, the particular case I worry about is:

• library Lib
• plugin A, using Lib
• plugin B, using Lib

Now, application App loads A and B; if Lib is linked in both, dynlink will fail in OCaml >= 4.08 as doubly linking modules was forbidden in that version. IMHO there should be a test for this.

[Kakadu]

To be clear, the particular case I worry about is:

* library `Lib`

* plugin `A`, using `Lib`

* plugin `B`, using `Lib`

Now, application App loads A and B; if Lib is linked in both, dynlink will fail in OCaml >= 4.08 as doubly linking modules was forbidden in that version. IMHO there should be a test for this.

@ejgallego I fixed a test in a manner I understood your concerns. Are you less worried now?

[ejgallego]

@ejgallego I fixed a test in a manner I understood your concerns. Are you less worried now?

Thanks a lot, it is great we are testing this case. I will have a detailed look tomorrow.

[bobot]

To be clear, -linkall is only used for the static linking of the binary (EDIT:executable) because we must be sure that all the libraries that are dependencies of the binaries have all their modules linked. Thanks @Kakadu for the additional tests !

[ejgallego]

To be clear, -linkall is only used for the static linking of the binary because we must be sure that all the libraries that are dependencies of the binaries have all their modules linked. Thanks @Kakadu for the additional tests !

Oh, I see; thanks for the comment @bobot ; on the other hand I think that silently adding -linkall (even in the findlib case) is a bug.

I find it pretty weird that just by adding findlib.dynload to my libraries stanza, then all modules in unrelated libraries [with their side effects] are now linked. WDYT?

[bobot]

I think it is weirder to add findlib.dynload or dune-sites.plugin as dependencies and being ok that it is not usable. I don't see any use case for that.

[ejgallego]

I think it is weirder to add findlib.dynload or dune-sites.plugin as dependencies and being ok that it is not usable.

Why they would be not usable? Adding -linkall seems to me like a brute-force way to implement them, but it has problems for sure with libraries that may not like -linkall. It should be possible to implement the plugin support without using the big -linkall hammer.

I don't see any use case for that.

A use case for what?

[bobot]

Adding -linkall seems to me like a brute-force way to implement them, but it has problems for sure with libraries that may not like -linkall.

Which libraries don't like -linkall ? In a way -linkall only simulate that the code use (i.e open) all the modules provided by the libraries. A library that break because you are using too much of it seems weird.

It should be possible to implement the plugin support without using the big -linkall hammer.

Perhaps but I don't know it. There are many difficulties to implement a plugin engine without linking all the modules of the static libraries:

• It is difficult to know which modules are statically linked
• It is currently not possible to dynlink a .cmxs where part of the modules are already linked
• Libraries that does side effect, without using for themselves -linkall, will have to take into account different loading orders.

The library granularity is clearer and simpler.

[ejgallego]

Which libraries don't like -linkall ?

I can't come up with an example now, but I imagine with large things such as core.async etc..., you don't want to link to all modules unless you know what you are doing.

My point is that we are using -linkall in the executable, so, I have a project, then I add findlib.dynload to the libraries field, now my executable gets huge, and maybe is doing stuff it shouldn't.

Perhaps but I don't know it. There are many difficulties to implement a plugin engine without linking all the modules of the static libraries

What's the problem with just referencing the modules that are used?

[ejgallego]

I moved the discussion to its own issue #4957

[bobot]

@Kakadu I rebased the commits and changed the order in which the tests are added in order to see that there is 2 fixes.

@ejgallego Can you review? even if the current state of the art of plugins in OCaml is not statisfying you 😸

[Kakadu]

I'm having issues with running my old example about sites. The manual mentions a library dune-site.plugins. Where should I get it?

[bobot]

It is in dune, how are you installing the development version of dune? Perhaps the pin is not pinning every package the dune repository provides.

[Kakadu]

It is in dune, how are you installing the development version of dune? Perhaps the pin is not pinning every package the dune repository provides.

https://gist.github.com/Kakadu/92c8181f89b13c3e584893c173bd34cb#file-log-txt-L75

Am I doing something wrong? It could be separate issue though...

P.S. I moved my complains to separate issue #4966

[snowleopard]

This test fails with a crash for me locally:

   Fatal error: exception Dune_site_plugins__Plugins.Thread_library_required_by_plugin_but_not_required_by_main_executable
+  Raised at Dune_site_plugins__Plugins.load_gen in file "otherlibs/site/src/plugins/plugins.ml", line 200, characters 6-89
+  Called from Stdlib__list.iter in file "list.ml", line 110, characters 12-15
+  Called from Dune_site_plugins__Plugins.load_gen in file "otherlibs/site/src/plugins/plugins.ml", line 204, characters 4-36
+  Called from Stdlib__list.iter in file "list.ml", line 110, characters 12-15
+  Called from Dune_site_plugins__Plugins.load_gen in file "otherlibs/site/src/plugins/plugins.ml", line 204, characters 4-36
+  Called from Stdlib__list.iter in file "list.ml",   [2] line 110, characters 12-15
+  Called from Dune__exe__App in file "app.ml", line 2, characters 9-42
   [2]

Could you do something about it?

[bobot]

Of course

[bobot]

Do you have something like "backtrace printing is always activated"?

[snowleopard]

Thanks! I'm not sure what "backtrace printing is always activated" means.

[Kakadu]

Maybe something like export OCAMLRUNPARAM=b in ~/.bashrc...

For my your problem is not easily reproducible too.

[snowleopard]

@Kakadu I see. That may well be it: my ~/.bashrc collects settings from various sources, and I wouldn't be surprised if one of them does what you say.

[bobot]

@ejgallego Sorry the changes is not well placed, are you going to move it?