[build] Add `symlink_system_library` calls for CSL #40217

staticfloat · 2021-03-26T18:08:28Z

We handle USE_SYSTEM_CSL=1 the same as USE_BINARYBUILDER_CSL=0, so
we should always install the libraries locally. Arguably we could
symlink instead of copying, but the benefit of isolation where the
user can then run make binary-dist is still evident.

Fixes #40201

staticfloat · 2021-03-26T18:10:33Z

Fixes #40201

yuyichao

System libraries must not be copied and must be symlinked instead.
If you want to support creating a portable build out of system library, that should be handled with a different option.

vtjnash · 2021-03-26T18:34:09Z

Previously, we've been handling these cases in base/Makefile. Perhaps not really the most consistent place to handle it, since many of those are actually present for stdlib _jll packages, but we should at least keep the handling consistent and together, where ever that may be.

staticfloat · 2021-03-26T19:42:40Z

Alright, new implementation that is a little bit more work (mostly because we can't find libgcc_s without adding the SOVERSION to the end, which required re-architecting some Makefile functions) but what I have now gives us three separate configurations:

USE_BINARYBUILDER_CSL=1: behavior unchanged, downloads and bundles the CSL from Yggdrasil. This is the default.
USE_BINARYBUILDER_CSL=0: Copies the libraries from the current compiler suite into usr/lib, as this is the closest we can get to a "from source" build without actually building GCC and friends.
USE_SYSTEM_CSL=1: Symlinks the libraries into usr/lib/julia using the same logic as the rest of the USE_SYSTEM_* flags.

I think this should satisfy all use cases. :)

yuyichao · 2021-03-26T19:46:32Z

The revised behavior SGTM.

We differentiate between a `USE_BINARYBUILDER_CSL=0` configuration (where we copy the CSL libraries into the Julia build tree) versus a `USE_SYSTEM_CSL=1` configuration (where we symlink them in a manner similar to other system libararies)

Without this, we fail to install things like `libgcc_s.so.1` when installed into the private libdir by `symlink_system_libraries`.

ronisbr · 2021-04-09T03:01:02Z

I am not sure if this is related, but using this patch in 1.6 and building Julia with USE_SYSTEM_CSL=1, the system tries to build the documentation even if it is already present from the tarball I downloaded:

[ 2343s] /home/abuild/rpmbuild/BUILD/julia-1.6.0/deps/tools/jldownload "/home/abuild/rpmbuild/BUILD/julia-1.6.0/deps/srccache/UnicodeData-13.0.0.txt" http://www.unicode.org/Public/13.0.0/ucd/UnicodeData.txt
[ 2343s] Could not find working curl, wget, or fetch.
[ 2343s] You need to install one of these to download dependencies.
[ 2343s] make[2]: *** [Makefile:30: /home/abuild/rpmbuild/BUILD/julia-1.6.0/deps/srccache/UnicodeData-13.0.0.txt] Error 1
[ 2343s] make[2]: Leaving directory '/home/abuild/rpmbuild/BUILD/julia-1.6.0/doc'
[ 2343s] make[1]: *** [Makefile:95: docs] Error 2
[ 2343s] make[1]: Leaving directory '/home/abuild/rpmbuild/BUILD/julia-1.6.0'
[ 2343s] make: *** [Makefile:47: /home/abuild/rpmbuild/BUILD/julia-1.6.0/doc/_build/html/en/index.html] Error 2
[ 2343s] error: Bad exit status from /var/tmp/rpm-tmp.R86M1y (%install)

The build works fine without this patch and USE_SYSTEM_CSL=0.

Note: I cannot build in the server farm Julia 1.6.0 with USE_SYSTEM_CSL=1 and without this patch. That's why I did not test.

* [build] Add `symlink_system_library` calls for CSL We differentiate between a `USE_BINARYBUILDER_CSL=0` configuration (where we copy the CSL libraries into the Julia build tree) versus a `USE_SYSTEM_CSL=1` configuration (where we symlink them in a manner similar to other system libararies) * Expand `JL_PRIVATE_LIBS-1` for versioned library names as well Without this, we fail to install things like `libgcc_s.so.1` when installed into the private libdir by `symlink_system_libraries`. (cherry picked from commit 336cdf8)

* [build] Add `symlink_system_library` calls for CSL We differentiate between a `USE_BINARYBUILDER_CSL=0` configuration (where we copy the CSL libraries into the Julia build tree) versus a `USE_SYSTEM_CSL=1` configuration (where we symlink them in a manner similar to other system libararies) * Expand `JL_PRIVATE_LIBS-1` for versioned library names as well Without this, we fail to install things like `libgcc_s.so.1` when installed into the private libdir by `symlink_system_libraries`.

* [build] Add `symlink_system_library` calls for CSL We differentiate between a `USE_BINARYBUILDER_CSL=0` configuration (where we copy the CSL libraries into the Julia build tree) versus a `USE_SYSTEM_CSL=1` configuration (where we symlink them in a manner similar to other system libararies) * Expand `JL_PRIVATE_LIBS-1` for versioned library names as well Without this, we fail to install things like `libgcc_s.so.1` when installed into the private libdir by `symlink_system_libraries`. (cherry picked from commit 336cdf8)

staticfloat added the backport 1.6 Change should be backported to release-1.6 label Mar 26, 2021

staticfloat mentioned this pull request Mar 26, 2021

Build fails with USE_SYSTEM_CSL=1 #40201

Closed

yuyichao requested changes Mar 26, 2021

View reviewed changes

staticfloat force-pushed the sf/use_system_csl branch from f8d0457 to 71d7a8a Compare March 26, 2021 19:39

staticfloat changed the title ~~[build] Always install csl in deps/Makefile~~ [build] Add symlink_system_library calls for CSL Mar 26, 2021

yuyichao approved these changes Mar 26, 2021

View reviewed changes

staticfloat added 2 commits April 2, 2021 18:57

Expand JL_PRIVATE_LIBS-1 for versioned library names as well

d11c86c

Without this, we fail to install things like `libgcc_s.so.1` when installed into the private libdir by `symlink_system_libraries`.

staticfloat force-pushed the sf/use_system_csl branch from 6536b85 to d11c86c Compare April 3, 2021 01:57

staticfloat mentioned this pull request Apr 3, 2021

make USE_BINARYBUILDER_CSL=0 fails with static libraries #40234

Closed

KristofferC mentioned this pull request Apr 4, 2021

Backports release 1.6.1 #40209

Merged

33 tasks

staticfloat merged commit 336cdf8 into master Apr 6, 2021

staticfloat deleted the sf/use_system_csl branch April 6, 2021 21:10

KristofferC removed the backport 1.6 Change should be backported to release-1.6 label May 4, 2021

KristofferC mentioned this pull request May 4, 2021

Backports for Julia-1.6.2 #40702

Merged

45 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[build] Add `symlink_system_library` calls for CSL #40217

[build] Add `symlink_system_library` calls for CSL #40217

staticfloat commented Mar 26, 2021 •

edited by yuyichao

Loading

staticfloat commented Mar 26, 2021

yuyichao left a comment

vtjnash commented Mar 26, 2021 •

edited

Loading

staticfloat commented Mar 26, 2021

yuyichao commented Mar 26, 2021

ronisbr commented Apr 9, 2021 •

edited

Loading

[build] Add symlink_system_library calls for CSL #40217

[build] Add symlink_system_library calls for CSL #40217

Conversation

staticfloat commented Mar 26, 2021 • edited by yuyichao Loading

staticfloat commented Mar 26, 2021

yuyichao left a comment

Choose a reason for hiding this comment

vtjnash commented Mar 26, 2021 • edited Loading

staticfloat commented Mar 26, 2021

yuyichao commented Mar 26, 2021

ronisbr commented Apr 9, 2021 • edited Loading

[build] Add `symlink_system_library` calls for CSL #40217

[build] Add `symlink_system_library` calls for CSL #40217

staticfloat commented Mar 26, 2021 •

edited by yuyichao

Loading

vtjnash commented Mar 26, 2021 •

edited

Loading

ronisbr commented Apr 9, 2021 •

edited

Loading