nix-shell -p julia doesnt work on osx

[tyoc213]

Describe the bug

On osx 12.6 (21G115) executingnix-shell -p julia will throw the error

error: Package ‘julia-1.5.4’ in /nix/store/qn9krwzkk4z9bkbvdd9mqn6an3yskx2y-nixpkgs/nixpkgs/pkgs/development/compilers/julia/1.5.nix:138 is marked as broken, refusing to evaluate.

       a) To temporarily allow broken packages, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_BROKEN=1

        Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
        (Flake) command, `--impure` must be passed in order to read this
        environment variable.

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowBroken = true; }
       in configuration.nix to override this.

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowBroken = true; }
       to ~/.config/nixpkgs/config.nix.
(use '--show-trace' to show detailed location information)

Expected behavior

I want to use julia via nix-shell

nix-env --version
nix-env (Nix) 2.11.1

[SuperSandro2000]

dupe of #114766 #164424 #123394

[wentasah]

Does nix-shell -p julia_18 work for you? Since 4 days ago, we have a new version packaged. I see that darwin is not officially supported, but you can try adding the darwin platform and build it. If it happens to work, let us know.

[Sean1708]

@wentasah It's still failing, at least it is for me:

$ NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1 nix profile install --impure 'nixpkgs#julia_18'
error: builder for '/nix/store/h689slw5gi0simnynqhdrv2mqxkkmz1s-julia-1.8.2.drv' failed with exit code 2;
       last 10 log lines:
       > /private/tmp/nix-build-julia-1.8.2.drv-0/julia-1.8.2/contrib/install.sh 755 /private/tmp/nix-build-julia-1.8.2.drv-0/julia-1.8.2/contrib/julia-config.jl /private/tmp/nix-build-julia-1.8.2.drv-0/julia-1.8.2/usr/share/julia/
       > Warning: git information unavailable; versioning information limited
       > /nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/bash: line 1: xcode-select: command not found
       > /nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/bash: line 1: xcode-select: command not found
       > rm: cannot remove 'scratch/objconv/': No such file or directory
       > /nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/bash: line 1: unzip: command not found
       > make[1]: *** [/private/tmp/nix-build-julia-1.8.2.drv-0/julia-1.8.2/deps/objconv.mk:11: scratch/objconv/source-extracted] Error 127
       > make[1]: *** Waiting for unfinished jobs....
       > rm: cannot remove 'scratch/dsfmt-2.2.4/': No such file or directory
       > make: *** [Makefile:60: julia-deps] Error 2
       For full logs, run 'nix log /nix/store/h689slw5gi0simnynqhdrv2mqxkkmz1s-julia-1.8.2.drv'.

$ nix log /nix/store/h689slw5gi0simnynqhdrv2mqxkkmz1s-julia-1.8.2.drv
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/bapalpix5agr1442plsbgrjn90k9dny0-julia-1.8.2-full.tar.gz
source root is julia-1.8.2
setting SOURCE_DATE_EPOCH to timestamp 1664488064 of file julia-1.8.2/deps/srccache/objconv.zip
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
applying patch /nix/store/75p50vzzy0199hqmhkmv400xw524ldwn-julia-hardcoded-libs.patch
patching file stdlib/MbedTLS_jll/src/MbedTLS_jll.jl
patching file stdlib/libLLVM_jll/src/libLLVM_jll.jl
patching file stdlib/nghttp2_jll/src/nghttp2_jll.jl
patching file stdlib/LibGit2_jll/src/LibGit2_jll.jl
patching file stdlib/OpenLibm_jll/src/OpenLibm_jll.jl
applying patch /nix/store/jl74akw7jnzgga6m2g7yk5l6i5j599yv-julia-libunwind-1.6.patch
patching file src/debuginfo.cpp
Hunk #1 succeeded at 292 (offset 2 lines).
applying patch /nix/store/8ms0pr3zm2s51dxax7l30b6mf7zgbh0k-0001-skip-symlink-system-libraries.patch
patching file base/Makefile
applying patch /nix/store/jrw6l4p7z2nd32ymdibwhda8161b5d17-0002-skip-building-doc.patch
patching file Makefile
applying patch /nix/store/qidv3kszm3x9yxy0fj5idfhdj3wc55vh-0003-skip-failing-tests.patch
patching file test/Makefile
applying patch /nix/store/f20bhd1kbv20b77wx6j8ghnyc6h8c85p-0004-ignore-absolute-path-when-loading-library.patch
patching file cli/loader_lib.c
patching script interpreter paths in .
./contrib/prepare_release.sh: interpreter directive changed from "#!/bin/sh" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/sh"
./contrib/delete-all-rpaths.sh: interpreter directive changed from "#!/bin/sh" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/sh"
./contrib/tsan/build.sh: interpreter directive changed from "#!/bin/bash" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/bash"
./contrib/check-whitespace.sh: interpreter directive changed from "#!/bin/sh" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/sh"
./contrib/fixup-rpath.sh: interpreter directive changed from "#!/bin/sh" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/sh"
./contrib/install.sh: interpreter directive changed from "#!/bin/sh" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/sh"
./contrib/new-stdlib.sh: interpreter directive changed from "#!/bin/sh" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/sh"
./contrib/mac/app/renotarize_dmg.sh: interpreter directive changed from "#!/bin/bash" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/bash"
./contrib/mac/app/notarize_check.sh: interpreter directive changed from "#!/bin/bash" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/bash"
./contrib/asan/build.sh: interpreter directive changed from "#!/bin/bash" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/bash"
./contrib/asan/check.jl: interpreter directive changed from "#!/bin/bash" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/bash"
./contrib/fixup-libgfortran.sh: interpreter directive changed from "#!/bin/sh" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/sh"
./contrib/normalize_triplet.py: interpreter directive changed from "#!/usr/bin/env python" to "/nix/store/w4agbx5b17d57jij2i4klv67r8qnl6g3-python3-3.10.7/bin/python"
./contrib/commit-name.sh: interpreter directive changed from "#!/bin/sh" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/sh"
./contrib/codesign.sh: interpreter directive changed from "#!/bin/sh" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/sh"
./contrib/download_cmake.sh: interpreter directive changed from "#!/bin/sh" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/sh"
./contrib/fixup-libstdc++.sh: interpreter directive changed from "#!/bin/sh" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/sh"
./deps/tools/jldownload: interpreter directive changed from "#!/bin/sh" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/sh"
./deps/tools/jlchecksum: interpreter directive changed from "#!/bin/sh" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/sh"
./deps/patches/config.sub: interpreter directive changed from "#! /bin/sh" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/sh"
./src/flisp/bootstrap.sh: interpreter directive changed from "#!/bin/sh" to "/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/sh"
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
updateAutotoolsGnuConfigScriptsPhase
Updating Autotools / GNU config script to a newer upstream version: ./deps/patches/config.sub
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
no configure script, doing nothing
@nix { "action": "setPhase", "phase": "buildPhase" }
building
build flags: -j8 -l8 SHELL=/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/bash prefix=\$\(out\) USE_BINARYBUILDER=0 USE_SYSTEM_CSL=1 USE_SYSTEM_LLVM=0 USE_SYSTEM_LIBUNWIND=1 USE_SYSTEM_PCRE=1 USE_SYSTEM_LIBM=0 USE_SYSTEM_OPENLIBM=0 USE_SYSTEM_DSFMT=0 USE_SYSTEM_LIBBLASTRAMPOLINE=0 USE_SYSTEM_BLAS=0 USE_SYSTEM_LAPACK=0 USE_SYSTEM_GMP=1 USE_SYSTEM_MPFR=1 USE_SYSTEM_LIBSUITESPARSE=0 USE_SYSTEM_LIBUV=0 USE_SYSTEM_UTF8PROC=1 USE_SYSTEM_MBEDTLS=1 USE_SYSTEM_LIBSSH2=1 USE_SYSTEM_NGHTTP2=1 USE_SYSTEM_CURL=1 USE_SYSTEM_LIBGIT2=1 USE_SYSTEM_PATCHELF=1 USE_SYSTEM_LIBWHICH=1 USE_SYSTEM_ZLIB=1 USE_SYSTEM_P7ZIP=1 PCRE_INCL_PATH=/nix/store/zxx67fsa1rkacj0rkfqyy0gjrlkbv6qa-pcre2-10.40-dev/include/pcre2.h
Warning: git information unavailable; versioning information limited
/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/bash: line 1: xcode-select: command not found
/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/bash: line 1: xcode-select: command not found
Copying in usr/share/man/man1/julia.1
Creating usr/etc/julia/startup.jl
/private/tmp/nix-build-julia-1.8.2.drv-0/julia-1.8.2/contrib/install.sh 755 /private/tmp/nix-build-julia-1.8.2.drv-0/julia-1.8.2/contrib/julia-config.jl /private/tmp/nix-build-julia-1.8.2.drv-0/julia-1.8.2/usr/share/julia/
Warning: git information unavailable; versioning information limited
/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/bash: line 1: xcode-select: command not found
/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/bash: line 1: xcode-select: command not found
rm: cannot remove 'scratch/objconv/': No such file or directory
/nix/store/a21jngr2a6gxln6nifn4xw9k5l8kda3l-bash-5.1-p16/bin/bash: line 1: unzip: command not found
make[1]: *** [/private/tmp/nix-build-julia-1.8.2.drv-0/julia-1.8.2/deps/objconv.mk:11: scratch/objconv/source-extracted] Error 127
make[1]: *** Waiting for unfinished jobs....
rm: cannot remove 'scratch/dsfmt-2.2.4/': No such file or directory
make: *** [Makefile:60: julia-deps] Error 2

$ nix run 'nixpkgs#nix-info'
system: "aarch64-darwin", multi-user?: yes, version: nix-env (Nix) 2.11.1, channels(root): "nixpkgs", nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixpkgs

$ sw_vers
ProductName:	macOS
ProductVersion:	12.6
BuildVersion:	21G115

Seems like there's some differences in the commands that are needed on Darwin (specifically xcode-select and unzip) that aren't being provided by the derivation.

[wentasah]

Yes, these are definitely missing. Can you try adding these to the derivation?

I don't use Macs at all, but xcode-select seems to be provided by xcbuild and other derivations add it to nativeBuildInputs with:

  ++ optionals stdenv.isDarwin [ xcbuild ];

Unzip should probably be added just next to xcbuild:

  ++ optionals stdenv.isDarwin [ unzip xcbuild ];

[Sean1708]

That gets it further along in the process, but eventually the build reaches a point where it needs to call julia and when it does it can't locate libgcc_s.dylib:

JULIA usr/lib/julia/corecompiler.ji
ERROR: Unable to load dependent library libgcc_s.1.1.dylib
Message:dlopen(libgcc_s.1.1.dylib, 0x000A): tried: '/private/tmp/nix-build-julia-1.8.2.drv-0/julia-1.8.2/usr/lib/libgcc_s.1.1.dylib' (no such file), '/nix/store/xdc73dwdxz1v1llz0kc7vdhlfzbykiii-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/libgcc_s.1.1.dylib' (no such file), '/private/tmp/nix-build-julia-1.8.2.drv-0/julia-1.8.2/usr/bin/../lib/libgcc_s.1.1.dylib' (no such file), '/nix/store/xdc73dwdxz1v1llz0kc7vdhlfzbykiii-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/libgcc_s.1.1.dylib' (no such file), 'libgcc_s.1.1.dylib' (no such file), '/usr/local/lib/libgcc_s.1.1.dylib' (no such file), '/usr/lib/libgcc_s.1.1.dylib' (no such file), '/private/tmp/nix-build-julia-1.8.2.drv-0/julia-1.8.2/base/libgcc_s.1.1.dylib' (no such file)
make[1]: *** [sysimage.mk:61: /private/tmp/nix-build-julia-1.8.2.drv-0/julia-1.8.2/usr/lib/julia/corecompiler.ji] Error 1
make: *** [Makefile:82: julia-sysimg-ji] Error 2

This library does exist on my machine:

$ find /nix/store -iname '*libgcc_s*'
/nix/store/bhlcyrh0zx99d84a0pvyil82cnp4mzfh-libSystem-11.0.0/lib/libgcc_s.1.tbd
/nix/store/i34pl7nbci48xxchr5shk2a62s9jj1x2-gfortran-11.2.0-lib/lib/libgcc_s.1.1.dylib
/nix/store/23k6r5z47xi4czn32qcvvrbimd1nx8dh-gcc-11.2.0-lib/lib/libgcc_s.1.1.dylib

Just not where it's trying to look:

$ ls /nix/store/xdc73dwdxz1v1llz0kc7vdhlfzbykiii-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/
CoreFoundation.framework
$ find /nix/store/xdc73dwdxz1v1llz0kc7vdhlfzbykiii-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/ -iname '*libgcc_s*'

I've tried a couple of different things to try to fix that (mostly adding things to the JULIA_RPATH variable), but my knowledge of build processes in general is fairly limited and on a Mac it's completely non-existent.

[wentasah]

Hmm, I see. The build process on MacOS is probably quite different from Linux. Moreover, it seems that MacOS build tries to download libraries at build time, which will probably not work in Nix.

Anyway, I'm pinging @NickCao so that he is aware of your attempts. I'm not sure whether he will be able to help, though.

[NickCao]

Julia does something unusual in it's building process, namely symlinking the dynamic libraries to it's private folder under /lib/julia, to ensure dlopening the exact same copy of libraries at runtime. However the tool it uses to locate the path of the shared libraries, libwhich, is not that reliable, thus I patched that part out and let it find libraries in RPATH in stead. If there is an equivalent of RPATH on darwin, you may use that instead, or you can drop that one patch on darwin and hope it works.

[Sean1708]

So dropping the patch did not work... It ends up failing when trying to call flisp at some point (I think) earlier in the build process.

Regarding an equivalent of RPATH: it seems that Darwin has something called an install_name that needs to be set and it looks like other packages use install_name_tool to set this but I think Julia's build process might be a bit too special for this to work... at least my attempts have thus far failed (I've only tried adding a snippet, like the one I linked to above, in the postFixup or postInstall phases but I think the problem occurs as part of the build phase).

Maybe there's someone more knowledgable in the Darwin build process that we can pester?

[wentasah]

Maybe there's someone more knowledgable in the Darwin build process that we can pester?

If I recall correctly, there is a new team/initiative for improving Nix experience on Darwin. Not sure where they are, but asking on Matrix at https://matrix.to/#/#macos:nixos.org might help.