install cmxs files

[nilsbecker]

hi, i was trying to load owl into a native ocamlnat toplevel, using dbuenzli's omod package. this fails when trying to load eigen. (see e.g. https://discuss.ocaml.org/t/interim-report-on-experience-with-ocamlnat/833/8)

# #use "omod.nattop";;
# Omod.load "Eigen";;
[OBJ] /Users/nbecker/.opam/410nattop/lib/eigen/eigen.cmxs
[ERROR] load /Users/nbecker/.opam/410nattop/lib/eigen/eigen.cmxs:
File not found: /Users/nbecker/.opam/410nattop/lib/eigen/eigen.cmxs

It would be great if Eigen could install its cmxs libaries in order to be able to explore usage of owl/eigen with the native toplevel.

[mseri]

I am surprised that they are not installed. The build (besides generating the c++ stubs) is a rather standard dune build. Does it require a special dune rule?

[nilsbecker]

i'm certainly no expert so i might have done something incorrectly. my $OPAMROOT/410nattop/lib/eigen/ has these files:

META
cpp
dune-package
eigen.a
eigen.cma
eigen.cmi
eigen.cmt
eigen.cmx
eigen.cmxa
eigen.cmxs.startup.o
eigen.ml
eigen__.cmi
eigen__.cmt
eigen__.cmx
eigen__.ml
eigen__Eigen_dsmat_c.cmi
eigen__Eigen_dsmat_c.cmt
eigen__Eigen_dsmat_c.cmx
eigen__Eigen_dsmat_d.cmi
eigen__Eigen_dsmat_d.cmt
eigen__Eigen_dsmat_d.cmx
eigen__Eigen_dsmat_s.cmi
eigen__Eigen_dsmat_s.cmt
eigen__Eigen_dsmat_s.cmx
eigen__Eigen_dsmat_z.cmi
eigen__Eigen_dsmat_z.cmt
eigen__Eigen_dsmat_z.cmx
eigen__Eigen_spmat_c.cmi
eigen__Eigen_spmat_c.cmt
eigen__Eigen_spmat_c.cmx
eigen__Eigen_spmat_d.cmi
eigen__Eigen_spmat_d.cmt
eigen__Eigen_spmat_d.cmx
eigen__Eigen_spmat_s.cmi
eigen__Eigen_spmat_s.cmt
eigen__Eigen_spmat_s.cmx
eigen__Eigen_spmat_z.cmi
eigen__Eigen_spmat_z.cmt
eigen__Eigen_spmat_z.cmx
eigen__Eigen_tensor_d.cmi
eigen__Eigen_tensor_d.cmt
eigen__Eigen_tensor_d.cmx
eigen__Eigen_tensor_s.cmi
eigen__Eigen_tensor_s.cmt
eigen__Eigen_tensor_s.cmx
eigen__Eigen_types.cmi
eigen__Eigen_types.cmt
eigen__Eigen_types.cmx
eigen__Eigen_utils.cmi
eigen__Eigen_utils.cmt
eigen__Eigen_utils.cmx
eigen__Ffi_eigen_bindings.cmi
eigen__Ffi_eigen_bindings.cmt
eigen__Ffi_eigen_bindings.cmx
eigen__Ffi_eigen_generated.cmi
eigen__Ffi_eigen_generated.cmt
eigen__Ffi_eigen_generated.cmx
eigen_dsmat_c.ml
eigen_dsmat_d.ml
eigen_dsmat_s.ml
eigen_dsmat_z.ml
eigen_spmat_c.ml
eigen_spmat_d.ml
eigen_spmat_s.ml
eigen_spmat_z.ml
eigen_tensor_d.ml
eigen_tensor_s.ml
eigen_types.ml
eigen_utils.ml
ffi_eigen_bindings.ml
ffi_eigen_generated.ml
libeigen_stubs.a
opam

[mseri]

I don’t know, maybe you can ask on dune repository, in the thread in the forum or on the discord channel. I have never dealt with cmxs or this issue before.

If you use the trick suggested in the thread to manually create the cmxs, ocamlopt -shared -linkall xxx.cmxa -o xxx.cmxs, does it work?

In case we need to change anything to add support here, feel free to send a pr

[nilsbecker]

no, that did not work.

ocamlopt -shared -linkall eigen.cmxa -o eigen.cmxs
ld: library not found for -leigen_stubs
clang: error: linker command failed with exit code 1 (use -v to see invocation)
File "caml_startup", line 1:
Error: Error during linking

[nilsbecker]

i can try asking elsewhere, thanks for now.

[nilsbecker]

ok, the missing eigen.cmxs turned out to be probably due to something going wrong at some point in the past in my opam switch. now the file is found. however, it leads to a linking failure, as reported in the issue ocaml/dune#3889 (comment) . should i open a new issue for that? any insights?

[mseri]

Mmh, that function is in the c stubs, I am surprised it does not find it. Is the ldpath added by opam pointing at the stublibs folder?

[nilsbecker]

do you mean this:

 env | grep CAML_LD
CAML_LD_LIBRARY_PATH=/Users/nbecker/.opam/410nattop/lib/stublibs:/Users/nbecker/.opam/410nattop/lib/ocaml/stublibs:/Users/nbecker/.opam/410nattop/lib/ocaml

in that folder:

dllbase_internalhash_types_stubs.so dllctypes_stubs.so.owner
dllbase_stubs.so                    dlleigen_cpp_stubs_stubs.so
dllcamlzip.so                       dlleigen_stubs.so
dllcamlzip.so.owner                 dllintegers_stubs.so
dllctypes_stubs.so                  dllowl_stubs.so

[mseri]

Yes, that looks correct and the stubs are there

[nilsbecker]

hmm. could it be a bug/limitation of the Omod loader?

[nilsbecker]

insights from @dbuenzli : the .cmxs is not generated correctly, see dbuenzli/omod#12 . some dune flags have to be added i assume, but i don't know which.

[dbuenzli]

Just to make things clear:

1. The stubs in stublibs you mention above are only ever used by bytecode executables (and ocaml). For cmxs either you put the stubs directly in the cmxs or maybe you could try to add a C linker dependency on the cmxs so that it loads that library. It's unclear to me which is best or generally recommended.
2. The way the stubs are structured means that Omod misses the dependency. This is a limitation of Omod which relies on concrete OCaml data dependencies by looking at the results of ocamlobjinfo. So as mentioned in this message you need to first load eigen_cpp_stubs and then eigen for the load to succeed. If you can mention the module Eigen_cpp_stub somewhere in eigen so that an ocamlobjinfo on eigen.cm[x]a reports it as an interface imported, this should work; but might feel artificial.

[mseri]

Thanks for the clarifications. I could open Eigen_cpp_stub in the root of eigen, after all it is an empty module just to link the stubs. @nilsbecker would you be able to check if that is enough?

[dbuenzli]

would you be able to check if that is enough?

That won't solve the problem of not having the stubs in the eigen_cpp_stub.cmxs file. That only solves the problem of omod missing the dependency.

[dbuenzli]

That won't solve the problem of not having the stubs in the eigen_cpp_stub.cmxs file.

Could someone check whether this also happens on Linux (more precisely are the symbols of libeigen_stubs_stubs.lib in the cmxs or not). If it does not then I think we are looking at a macOS issue and it would be nice to have something reported upstream asap. See the discussion on the dune issue.

[nilsbecker]

it does seem to happen on linux, see the dune issue.

[nilsbecker]

a solution for linking stubs in such a way that the library can be dynamically loaded has been found in ocaml/dune#3908 . it's in nojb/eigen@8de3f76
this seems to enable loading of eigen and then owl in the native toplevel ocamlnat on ocaml 4.10 at least.

[mseri]

Thanks for the update and sorry for the slow reactions. I’d be happy to merge the fix and draft a new release