Do not rely on unversioned .so files

[nalimilan]

Since Julia does not link to external libraries at compile time, it should not try to access unversioned .so files, which may not be present on the user system as they are typically only provided by -devel packages.

This problem is visible at least for openlibm and openspecfun with Julia from the Fedora RPM package, when the -devel packages are not installed:

julia> svd([1:10 1:10])
ERROR: error compiling ctranspose: error compiling transpose!: could not load module libopenlibm: /lib64/libopenlibm.so: cannot open shared object file: No such file or directory
julia> airy(1, 1+1im)
ERROR: error compiling airy: error compiling airy: could not load module libopenspecfun: /lib64/libopenspecfun.so: cannot open shared object file: No such file or directory

I'm not sure how best to fix this. The versioned file names could be saved somewhere at build time (just like a linker would do), in order to call them instead of the unversioned file.

[JeffBezanson]

Interesting; I thought this would be fixed by parsing the output of ldconfig -p as we do. What does that show for these libraries?

[nalimilan]

Glad to hear it's the intended behavior.

$ ldconfig -p | grep openspecfun
    libopenspecfun.so.0.3.0 (libc6,x86-64) => /lib64/libopenspecfun.so.0.3.0
    libopenspecfun.so (libc6,x86-64) => /lib64/libopenspecfun.so
$ ldconfig -p | grep openlibm
    libopenlibm.so.0.3.0 (libc6,x86-64) => /lib64/libopenlibm.so.0.3.0
    libopenlibm.so (libc6,x86-64) => /lib64/libopenlibm.so

[JeffBezanson]

This could be the problem: that output points us to /lib64/libopenspecfun.so but the file doesn't exist.

[nalimilan]

No, the file exists (it's the same as /usr/lib64/libopenspecfun.so). At least, it exists when package openspecfun-devel is installed.

[staticfloat]

It could be that there's something else going on here, because if I remember correctly we attempt to load the versioned library first, and if that fails, we attempt to load the unversioned library next, so if we for whatever reason can't load either, that error message might be the one that gets printed out.

[nalimilan]

I've tried debugging this, but strangely enough I'm not able to reproduce the problem with an in-tree build, only with the RPM version. More precisely, I've tried breaking on jl_read_sonames() and jl_load_dynamic_library_(), I've tried adding abort() calls there, and they don't appear to be called when running the commands from the issue description. Are libraries statically linked? Can this be related to USE_SYSTEM_{OPENLIBM,OPENSPECFUN}=1 in the RPM?

[staticfloat]

Could it be that the error message we're seeing here is deceptive? Looking at your error message again:

julia> svd([1:10 1:10])
ERROR: error compiling ctranspose: error compiling transpose!: could not load module libopenlibm: /lib64/libopenlibm.so: cannot open shared object file: No such file or directory

This sounds to me like the error is that libopenlibm is trying to open a library and failing to do so. That's what all those colons are trying to get across, I think. What does ldd say about the libopenlibm libraries? Both the versioned and unversioned .so files would be nice.

[nalimilan]

No, the error appears even when openlibm or openspecfun are not present at all, so the problem is not in their files. And the error message can be found in ccall.cpp.

Thinking about it, I think Julia really needs to store the expected SOVERSION of the libraries it wants to load. It doesn't make sense to try loading any version of a library, since API incompatibilities can arise; what would happen if the user has two versions of the same library installed? Granted, for openlibm and openspecfun at the moment it's not a problem, but we may need to break the API in the future; double-conversion may break it at some point. (For packages it would also be useful, especially since they will not be shipped as distro packages, and thus are particularly exposed to version mismatches.)

[staticfloat]

This is a part of Julia that definitely will need work in the future. Pinging @Keno since he's worked on this area before.

The way I see it; dlopen("openlibm") should always work. If there is a cross-platform way that we can use to divine the version of a library, the easiest way for this to work is to have something like dlopen("openlibm", v"0.2.3"), where dlopen("openlibm", v"0.2") would still "just work". Unfortunately, I'm not sure if disciplined methods for library versioning exist on Windows. On OSX, as long as we're talking about opensource linuxy things I'm pretty sure we can wrangle something, but I have no idea how this would work out with osx-only libraries, Frameworks and Windows dll's which often use Manifests and such builtin to the DLLs to do their versioning.

[nalimilan]

The version shouldn't be so precise, you only need the SONAME. So e.g. just dlopen("openlibm", v"0"). One could even do just dlopen("openlibm.0") since SONAME is really handled that way by the linker, i.e. either it's equal or it isn't, it's actually part of the library name; it's more like a name than a "version" since you will never compare whether one is more recent than another.

On Windows we probably don't care since the libraries wouldn't be shared with the rest of the system. So Julia could just remove everything after the dot to get unversioned requirements (or keep the version in the name). I don't know how this works on Mac OS X.

[nalimilan]

Hm, dlopen("openlibm.0") wouldn't work since the library name is libopenlibm.$(SHEXT).0. So I guess dlopen("openlibm", v"0") or just dlopen("openlibm", "0") is better.

[nalimilan]

@sebastien-villemot Maybe you want to give us your opinion here?

[svillemot]

I actually agree with @nalimilan : Julia should ideally always include a SONAME version when dlopen'ing a library. This diminishes the risk of calling an old version of a library that does not include the required symbols. For the Debian package, I carry a (quite big) patch where I add a version number to all dlopen calls.

Since SONAME can actually vary across distributions, I guess the only practical way of implementing this is to have some sort of detection of libraries at the beginning of the build process, and put the SONAMES into base/build_h.jl (as is already done for libm or BLAS/LAPACK).

[staticfloat]

I think we can make the following assumptions:

• There are times when we will want to ignore SONAMES. As such, I don't think dlopen("libfoo") will ever go away. For users developing their own libraries, or many other uses, this should work just fine.
• Many times what we're looking for is not so much a certain SONAME but rather certain functionality to be present in the shared library. For this we have the hooks in BinDeps to check whether or not a certain dynamic library is sufficient for our use. It's an arbitrary function that gets passed a handle to the library just opened, which must return true or false to determine whether this library satisfies the user's requirements. I think we should encourage users to use BinDeps for all cross-platform binary needs, since it has the machinery to do this kind of thing better than base julia

I'm not sure in what circumstances SONAMEs would be necessary given the hooks in BinDeps, (and if you're writing a package that uses binary dependencies, you'll probably need to use the hooks anyway since SONAMEs aren't the standard on other platforms) but I'd like to be enlightened so that we can implement this in the best way possible. The place I can see this being used most is in Base, where we might need to do things like only load libgmp.so.10 instead of just libgmp, but again if we are amenable to loading many different SONAMEs, it might be better to just test for the functionality we need.

[nalimilan]

We probably need to distinguish Julia itself from packages: Julia will be compiled once when making distro packages, and we can save the SONAMEs at that point; OTC, packages will be built each time a user installs them, therefore they will need to detect libraries at that point anyway.

So I'd say we should use @sebastien-villemot's solution for Julia, and @staticfloat's for packages.

[nalimilan]

For now the workaround which has been suggested while reviewing the Fedora package is to create .so symlinks in /usr/lib/julia, which point to the versioned .so.X files in /usr/lib. It works quite well, but it would be better to have an automatic mechanism to save the SOVERSIONs of the libraries during the build, creating Julia constants for them.

[petercolberg]

For the Julia Debian package we now automatically detect the library sonames at build-time and generate lib*.so symlinks in Julia’s private libdir. This is achieved by linking a dummy executable against the dlopen'd libraries, and calling dladdr() for a symbol of each library to query the corresponding soname.

https://anonscm.debian.org/cgit/pkg-julia/julia.git/tree/debian/shlibdeps.c
https://anonscm.debian.org/cgit/pkg-julia/julia.git/tree/debian/shlibdeps.mk

The use of private library symlinks ensures that external packages load the same library soversion as Base modules, e.g., for :libgmp that is also loaded in some Julia packages such as BigRationals.jl.

As a long-term solution, however, I suggest to define constants in build_h.jl with the sonames of all libraries dlopen'd in Base, queried during build time using dladdr(). This functionality would ideally also be provided for Julia packages to be used during Pkg.build().

[nalimilan]

@petercolberg That sounds interesting.

Though I wonder whether we couldn't link libjulia to these libraries instead, even if they are not used from the C++ code. That way, if I'm not mistaken, all calls to dlopen (including those from packages) would use these library versions. As an additional benefit, distro package build tools would automatically detect the version requirements, removing a potential source of mistakes when the packager forgets to update them manually.

[nalimilan]

My proposal above about actually linking against the libraries would fix issues reported with RHEL packages in #12741 (comment). It would allow the package to explicitly depend on the version that was used during the build, which isn't currently possible to do automatically and leads to less clear errors at runtime.

[tkelman]

I think only Linux distribution packagers should ever link libjulia against ccall-only dependencies. Otherwise it's bad for embedding and solves a problem that doesn't really exist in other environments.

Could you create the current libjulia as libjulia-runtime and make a new libjulia for your purposes that just links against that plus the ccall deps? That way users of the distro packages who want to embed julia in some other application still have an option comparable to the way the tarball binaries work.

[nalimilan]

Of course, we could link to ccall libraries only when some environment variable is passed to make.

But I don't understand what's so specific about embedding: if we want ccall to work in this case, we still need to find the right version of the libraries, isn't it? And distro packages will be kept in sync (that's the whole point), so packaged embedding apps should use the same versions anyway.

[tkelman]

Embedding use cases would often want to leave out any libraries they do not use.

[petercolberg]

@nalimilan Please take a detailed look at the Debian packaging, it generates both private symlinks and package dependencies automatically. We also disable the fallback to ldconfig -p, which ensures that an error is produced during automated package testing (in a minimal environment with .so.VERSION but without .so symlinks) if any library dependencies are missing.

git clone https://anonscm.debian.org/git/pkg-julia/julia.git

Symlinking libjulia against all libraries needed by Base is not good solution, since it produces warnings during package building about useless dependencies due to not using any symbols of those libraries. Note that dlopen() would have to be avoided entirely. ccall() would have to be invoked without a library argument to use symbols from the compile-time linked libraries.

@tkelman, @nalimilan If you think it is an idea worthwhile exploring, I will prepare a pull request that integrates shlibdeps.c into the Julia build process for Linux and FreeBSD. This step would query the sonames at build time and store them as constants in base/build_h.jl.

[nalimilan]

Symlinking libjulia against all libraries needed by Base is not good solution, since it produces warnings during package building about useless dependencies due to not using any symbols of those libraries.

@petercolberg Warnings that are clearly wrong are a small annoyance, and we could likely get rid of them with appropriate flags (remember, those would only be used to build packages, so any legitimate warning would still be caught during normal builds).

Note that dlopen() would have to be avoided entirely. ccall() would have to be invoked without a library argument to use symbols from the compile-time linked libraries.

I'm not an expert about this, so others will have to confirm, but it may well be possible to call linked libraries with a higher priority. Libdl.dlopen already uses a hash table of sonames to find out what library to load.

If linking really couldn't work, then adding a mechanism to list required libraries sounds reasonable. The advantage of linking would have been that packagers wouldn't need to do anything special, while with your solution they need to generate dependencies manually (e.g. it doesn't sound super easy, though doable, for RPM).

Regarding how to get the list of required libraries, I don't understand why a creating a specific program is needed. Couldn't we simply list the libraries that Julia needs in a static file, then use jl_lookup_soname to get the SONAME? Anyway, with the approach in the Debian package, you still need to list symbols to resolve in the makefile, which isn't fully automated either.

[petercolberg]

On Fri, Nov 20, 2015 at 08:50:39AM -0800, Milan Bouchet-Valat wrote:

If linking really couldn't work, then adding a mechanism to list required libraries sounds reasonable. The advantage of linking would have been that packagers wouldn't need to do anything special, while with your solution they need to generate dependencies manually (e.g. it doesn't sound super easy, though doable, for RPM).

The solution uses the same linking of all libraries that you propose for libjulia, but for a dummy executable. The packaging tool inspects the dummy executable to generate automatic dependencies.

Regarding how to get the list of required libraries, I don't understand why a creating a specific program is needed. Couldn't we simply list the libraries that Julia needs in a static file, then use jl_lookup_soname to get the SONAME?

jl_lookup_soname uses ldconfig to match the library name against a soname. This is problematic since the order is undefined and depends on the library versions installed on the machine. The library link name might not even match the library soname (e.g., libdSFMT.so pointing to libdSFMT-19937.so.MAJOR). A side goal of this proposal is to avoid the non-deterministic fallback altogether.

Anyway, with the approach in the Debian package, you still need to list symbols to resolve in the makefile, which isn't fully automated either.

Yes, you still need to provide one symbol per library, as found in the Base modules.

I have a fully automatic solution in mind, too, but that requires more work on the side of the distributions. The GNU dynamic linker provides an interface to audit the loading of libraries (man rtld-audit), which could be used to record the run-time dependencies of julia, e.g., by running the test suite. Until such a tool has been developed, querying the sonames using dladdr() is a reliable solution.

[nalimilan]

The solution uses the same linking of all libraries that you propose for libjulia, but for a dummy executable. The packaging tool inspects the dummy executable to generate automatic dependencies.

Yes, but the point of my proposal is that package building tools already take care of detecting linked libraries automatically, without all this custom code to be written for each distribution.

jl_lookup_soname uses ldconfig to match the library name against a soname. This is problematic since the order is undefined and depends on the library versions installed on the machine.

I wouldn't care too much about that, since the goal is to build packages in clean environments where the only installed packages are those shipped by distributions and required by the package description. What matters is the robustness on users' systems.

The library link name might not even match the library soname (e.g., libdSFMT.so pointing to libdSFMT-19937.so.MAJOR). A side goal of this proposal is to avoid the non-deterministic fallback altogether.

Looks like we can handle this easily too by calling objdump to get the SONAME.

[petercolberg]

On Fri, Nov 20, 2015 at 10:41:08AM -0800, Milan Bouchet-Valat wrote:

Yes, but the point of my proposal is that package building tools already take care of detecting linked libraries automatically, without all this custom code to be written for each distribution.

I am not sure what you mean with custom code. All that is required for the distributor is to pass the dummy executable to the tool that detects shlibs dependencies.

Looks like we can handle this easily too by calling objdump to get the SONAME.

I have been through that thought process before. You cannot simply dump the linked libraries, because the SONAME might not match the link name.

[nalimilan]

Solution 2. is what I'm currently doing in Fedora. It indeed works well, with two drawbacks:

• the dependencies are not automatically versioned when building the package (as you noted)
• a change in SONAME requires manual intervention from the packager

To generate dependencies, RPM needs an executable taking file names via stdin and writing requirements to stdout: http://www.rpm.org/wiki/PackagerDocs/DependencyGenerator But that executable could be a very simple shell script, or even a call to cat when passed a file with a list of SONAMES.

[petercolberg]

Good to know that solution 2 works for Fedora, too. Then I will prepare a pull request that (a) creates symlinks in the private libdir for system libs and (b) generates a text file with the sonames, one per line.

A change in soname should in fact require manual intervention from the packager. This way Julia is rebuilt and tested when a dependent library undergoes an incompatible ABI change. In Debian we schedule a binNMU when a library changes ABI, which is an automated rebuild of the same source package using the updated dependencies.

[nalimilan]

@opoplawski Could you confirm that using RPM dependency generators (or another solution if it exists) would allow turning a text file listing the sonames into RPM Requires?

[opoplawski]

I believe I can write a RPM dependency generator for this case, yes. @petercolberg - have you made any progress on generating the list/links?

[kkofler]

Instead of producing all these symlinks, why can't you add a hash table (generated at compile time) into the julia binary that encodes the mapping? It would not (by itself) solve the dependency generator issue, but at least it would avoid creating all those ugly symlinks.

[petercolberg]

@kkofler can you explain why you think the symlinks are ugly?

On the contrary, I think it’s a transparent and maintainable solution from the point of view of distributions. lib*.so symlinks have been in use since a long time to provide development libraries for linking during static compilation.

[petercolberg]

To provide more detail on the maintainability aspect: It is immediately apparent to a non-Julia developer which library soversions julia loads, by inspecting the julia binaries (for compile-time linked libraries, e.g., using readelf -d) and the symlinks in the private libdir (for run-time linked libraries).

This non-Julia developer could for example be a distribution maintainer who has triggered the recompilation of a collection of dependent packages for a library soversion transition.

[nalimilan]

I wouldn't consider the use of symlinks as an issue. What matters is that 1) distributions can extract a list of versioned library dependencies automatically, and 2) that any change in version requirements immediately leads to a build/test failure so that the package maintainer notices it.

[nalimilan]

@petercolberg I've just discovered that RPM automatically added SONAMES as dependencies when the package contains a symlink to the library .so file. So I've added a series of commands like this:

    ln -s $(realpath -e %{_libdir}/libarpack.so) libarpack.so

This hardcodes the SONAME (including SOVERSION) at build time, so that any API/ABI break in one of the dependencies is detected by the package manager.

This simple solution could be moved upstream if you'd also like to use it. I don't even need a file with the list of SONAMES, but if that's useful to you, we could create it.

Are you still interested in making a PR?

[petercolberg]

On Thu, Mar 03, 2016 at 10:21:02AM -0800, Milan Bouchet-Valat wrote:

@petercolberg I've just discovered that RPM automatically added SONAMES as dependencies when the package contains a symlink to the library .so file. So I've added a series of commands like this:

    ln -s $(realpath -e %{_libdir}/libarpack.so) libarpack.so

This hardcodes the SONAME (including SOVERSION) at build time, so
that any API/ABI break in one of the dependencies is detected by the
package manager.

That won’t produce the correct symlinks, see for example libpcre2 on Debian:

/usr/lib/x86_64-linux-gnu/libpcre2-8.so -> libpcre2-8.so.0.2.0
/usr/lib/x86_64-linux-gnu/libpcre2-8.so.0 -> libpcre2-8.so.0.2.0
/usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.2.0

The .so symlink points to .so.0.2.0, which means the private symlink
for julia would break when the minor/patch version of the library is
updated while the major version stays the same (= same ABI).

The correct private symlink based on the DT_SONAME field is

/usr/lib/x86_64-linux-gnu/julia/libpcre2-8.so -> ../libpcre2-8.so.0

This simple solution could be moved upstream if you'd also like to use it. I don't even need a file with the list of SONAMES, but if that's useful to you, we could create it.

Are you still interested in making a PR?

Yes, I will submit a PR soon. Knowning that rpm can derive the library
dependencies from the symlinks themselves makes it much easier though.

[nalimilan]

Of course you're right. I knew it couldn't be so simple... I guess as a temporary hack until your PR I can switch to find /usr/lib64 -name libpcre2-8.so.?.

[petercolberg]

On Thu, Mar 03, 2016 at 10:59:49AM -0800, Milan Bouchet-Valat wrote:

Of course you're right. I knew it couldn't be so simple... I guess as a temporary hack until your PR I can switch to find /usr/lib64 -name libpcre2-8.so.?.

You could replace realpath with readelf, e.g.,

readelf -d /usr/lib/x86_64-linux-gnu/libpcre2-8.so.0 | sed -n '/SONAME/s/.(lib[^ ].so.[0-9])./\1/p'

[nalimilan]

Thanks, that indeed sounds more robust.

[nalimilan]

For reference, openlibm needs to be handled separately since sed only supports greedy matching, which gives libm.so.2. While the regex could certainly be fixed, it is easy to add libopen manually.

[jirutka]

I have the same problem, as @nalimilan described in this issue, on Alpine Linux – Julia tries to load unversioned so libs which are provided by -dev packages.

Moreover, ldconfig provided by musl libc doesn’t know the option -p.

[petercolberg]

Indeed, parsing ldconfig is at best a temporary work around.

I would like to implement a reliable query of the library soname based on #6742 (comment). I have yet to figure out how to best integrate this as a substitute for the existing jl_lookup_soname on Linux/FreeBSD.

[nalimilan]

@jirutka Have you tried the realpath solution from http://pkgs.fedoraproject.org/cgit/rpms/julia.git/tree/julia.spec#n156? It works great for me.

[jirutka]

@nalimilan No, but I found another solution/workaround, see alpinelinux/aports#112 (comment). 😺

[nalimilan]

AFAICT that workaround won't automatically add the loaded libraries as package dependencies. That's the real interest of the realpath approach, as breakage is quite common in my experience without an automated mechanism.

[jirutka]

The workaround in Fedora package does not automatically add the loaded libraries either, the libraries are explicitly named here: arpack cholmod dSFMT git2 fftw3 gmp mpfr openspecfun pcre2-8 umfpack (line 163). So it’s no more dynamic, actually quite opposite.

The workaround I’ve used allows to discover any library, not just those specifically named and not just those available in build time. Its behaviour is almost the same as the original Julia’s solution which relies on ldconfig -p. If I understand ldconfig -p correctly, it needs .pc files that we have separated in -dev subpackages; scanelf doesn’t need .pc files.

[nalimilan]

That's true, but that's exactly the same in your case, via the list of dependencies. Anyway, the list of libraries that Julia uses is relatively stable, so while not perfect it's not a big deal.

OTOH, distribution packages are often updated in a backward-incompatible way, so it's best to depend on a specific SONAME so that distribution build systems notice the package needs to be rebuilt. Since the test suite is run during the build, we get a build failure if the ABI break affects Julia and I get notified. Believe me, getting silent failures until a user files a bug days later was really terrible until I found that solution.