shared library names and search path

[ViralBShah]

There are two issues that need to be resolved relating to shared libraries:

1. Search path for dependencies. Libraries in julia_home/lib often depend on other libraries in the same directory, but dlopen does not look in julia_home/lib. This can possibly be fixed by adding julia_home/lib to LD_LIBRARY_PATH during startup.
2. At least on debian, all libraries are installed as libfoo.so.n, and only the dev packages install libfoo.so. Julia's dlopen looks for libfoo.so to load packages. We need to be able to find the right version of the library installed and dlopen libfoo.so.n.

[JeffBezanson]

Which library in julia_home/lib depends on another in the same directory? I can't find one.

[ViralBShah]

We have arranged things to avoid that. For instance, liblapack should depend on libopenblas, and libarpack should also do the same. libsuitesparse will actually be a series of libraries, many of which depend on openblas and lapack. Same will hold true for Clp. Part of the reason why our memory usage is high on startup is because we are statically linking multiple copies of the same library.

-viral

On Jan 28, 2012, at 12:11 PM, JeffBezanson wrote:

Which library in julia_home/lib depends on another in the same directory? I can't find one.

---

Reply to this email directly or view it on GitHub:
#353 (comment)

[JeffBezanson]

Ah, ok. We might be able to solve it by adding -Xlinker -rpath options when linking the executable, listing any directories we might need to look in. Something different might be needed on OS X.

[JeffBezanson]

Since #551 was fixed, now we're installing way too much stuff in lib. We need to list the libraries that should be copied.

[JeffBezanson]

Here's a patch to try:

--- a/Makefile
+++ b/Makefile
@@ -22,6 +22,8 @@ sys0.ji: src/boot.jl src/dump.c jl/stage0.jl
 sys.ji: VERSION sys0.ji jl/stage1.jl jl/sysimg.jl jl/start_image.jl jl/*.jl
        $(QUIET_JULIA) ./julia `test -f sys.ji && echo stage1.jl || echo -J sys0.ji stage1.jl`

+SHLIBS = libamos libarpack libfdm libfftw3 libfftw3f libgmp libgmp_wrapper libgrisu liblapack libpcre librandom libsuitesparse libsuitesparse_wrapper
+
 install: release
        install -d $(DESTDIR)$(PREFIX)/share/julia/lib
        install -d $(DESTDIR)$(PREFIX)/share/julia/jl
@@ -35,7 +37,7 @@ install: release
        install -v jl/* $(DESTDIR)$(PREFIX)/share/julia/jl
        install -v extras/* $(DESTDIR)$(PREFIX)/share/julia/extras
        install -v examples/*.jl $(DESTDIR)$(PREFIX)/share/julia/examples
-       -install -v lib/*.$(SHLIB_EXT) $(DESTDIR)$(PREFIX)/share/julia/lib
+       -install -v $(SHLIBS:%=lib/%.$(SHLIB_EXT)) $(DESTDIR)$(PREFIX)/share/julia/lib

[ViralBShah]

Ok, finally figured this out on OS X:

Need to use gcc -install_name @rpath/libfoo.dylib for shared libraries that need to be resolved when dlopening other libraries. Alternatively install_name_tool can be used.

[ViralBShah]

@JeffBezanson Note that -Xlinker -rpath only works when loading the binary. On OS X, if you have, say, libRmath that depends on librandom, which only gets loaded when you do load("extras/Rmath.jl"), the -Xlinker method does not work. I believe linux will not be as difficult to tackle as OS X.

I'll clean up the OS X build in the local_blas branch first, and then try out linux.

[ViralBShah]

The -Xlinker approach
http://stackoverflow.com/questions/4513799/how-to-set-the-runtime-path-rpath-of-an-executable-with-gcc-under-mac-osx

The install_name approach
http://stackoverflow.com/questions/6737795/does-rpath-work-for-looking-up-dylibs-that-are-loaded-called-from-another-dyl

[ViralBShah]

https://blogs.oracle.com/dipol/entry/dynamic_libraries_rpath_and_mac

[ViralBShah]

This is now resolved as per the above pull request and later commits to stabilize the build.