Cannot run under NixOS

[jonahbron]

Running workerd-linux-64 under linux should be possible with just installing glibc according to the readme. However when I try to execute it, I get this error:

node_modules/@cloudflare/workerd-linux-64/bin$ ls -la
total 86052
-rwxr-xr-x 2 jonah users 88115856 Dec  8 05:51 workerd
node_modules/@cloudflare/workerd-linux-64/bin$ ./workerd
bash: line 1: ./workerd: cannot execute: required file not found

This is running in a Nix shell with nixpkgs.glibc present (and verified by running the binaries it provides)

[jonahbron]

This is also reported by someone else in the workers-cli repo

cloudflare/workers-sdk#4483

[mikea]

what does ldd workerd say for you?

[jonahbron]

$ ldd node_modules/.bin/workerd
        linux-vdso.so.1 (0x00007ffcbb7c1000)
        libdl.so.2 => /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib/libdl.so.2 (0x00007f324fb34000)
        libpthread.so.0 => /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib/libpthread.so.0 (0x00007f324fb2f000)
        libm.so.6 => /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib/libm.so.6 (0x00007f324fa4f000)
        libc.so.6 => /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib/libc.so.6 (0x00007f324f867000)
        /lib64/ld-linux-x86-64.so.2 => /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib64/ld-linux-x86-64.so.2 (0x00007f3254111000)
        librt.so.1 => /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib/librt.so.1 (0x00007f324f862000)

[kentonv]

Hmm it sure looks like the libraries are there. I am not sure what would cause bash to produce the error cannot execute: required file not found. But when I Googled this error a bit I came across this stack exchange question which implies that non-Nix binaries cannot directly be executed on NixOS? Can this be the problem?

https://unix.stackexchange.com/questions/522822/different-methods-to-run-a-non-nixos-executable-on-nixos

[jonahbron]

In that answer under solution one (manual), part of what's happening is that a not-found library is being resolved, while in our case here all of the binaries are present and have known paths.

[jonahbron]

Following along further with that SE answer, I'm trying to run it with steam-run but encountering another issue first lol

nix-community/NixOS-WSL#328

[ckiee]

The problem can be located with the file command:

$ file ………/bin/workerd
node_modules/@cloudflare/workerd-linux-64/bin/workerd: ELF 64-bit LSB pie executable, x86-64, 
version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[xxHash]=bb2f4709f5fab1f7, not stripped

/lib64/ld-linux-x86-64.so.2 is not valid on NixOS. Workaround: wrap workerd with a shell script:

#!/bin/sh
export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt # https://github.com/cloudflare/workers-sdk/issues/3264
exec steam-run /path/to/workerd.orig "$@"

Cloudflare should compile workerd as a static binary, as it has no interesting dependencies anyway.

[PsychoLlama]

@ckiee your comment led me to figure out what happened. Thank you!

/lib64/ld-linux-x86-64.so.2 is not valid on NixOS

Checking the statically linked interpreter of hello gives me almost the exact same interpreter path that ldd ./workerd resolves:

$ file $(which hello)
# (output truncated)
# /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib/ld-linux-x86-64.so.2

vs

$ ldd ./workerd
# (output truncated)
# /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib64/ld-linux-x86-64.so.2

The difference between those two interpreter paths is /lib64 vs /lib, but /lib64 is just a symlink back to /lib. The paths are identical. I wasn't sure how execve felt about symlinks in the interpreter path so I tried patching the ELF:

# swaps `/lib64` to the resolved `/lib`
patchelf --set-interpreter /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib/ld-linux-x86-64.so.2 ./node_modules/@cloudflare/workerd-linux-64/bin/workerd

The binary becomes executable 🎉

I'm able to run wrangler dev now and serve requests.

[kentonv]

Cloudflare should compile workerd as a static binary, as it has no interesting dependencies anyway.

Unfortunately, linking statically against glibc does not work, because glibc uses dlopen() to dynamically load some of its own functionality, e.g. for DNS resolution. The dlopen()ed components must be from exactly the same version of glibc. But if you statically link glibc and then try to run the binary on some other distribution, the dynamically loaded components will almost certainly not match. This is a well-known problem with static linking on Linux, it's frustrating, but it's not something we can do much about.

Therefore, we statically link everything except glibc. But it seems that NixOS isn't happy with such binaries, unless you use an extra wrapper.

If there's something we could do that would make things easier on Nix without harming usability on other distros, we're happy to accept PRs. Unfortunately our team doesn't have the bandwidth to work on Nix support directly.

[jonahbron]

@PsychoLlama Would you mind clarifying the solution a bit? Is patchelf a command that you run once to fix the linking, then workerd starts working properly? Where do I get the interpreter path for my machine, does that come from the file command or somewhere else?

[PsychoLlama]

TL;DR: I only need to patch it once. I got the expected path from ldd ./workerd.

Longer answer

Sure, I'll tell you what I can, but I think I'm still missing parts of the picture. If my first post was correct then patching the interpreter to use /lib/ld-linux-x86-64.so.2 should have fixed the problem, but it did not. I have to provide the full nix store path or it does not work.

I also tried overriding the glibc package to copy the /lib64 files instead of symlinking them, as my original assumption was that symlinks were the problem. That would've been easily committed to a flake file. Nope, no luck.

I no longer believe symlinking is the problem. I don't know why it fails. I'm still a noob when it comes to native stuff like this, so I might be missing something obvious.

EDIT: While writing this response it occurs to me that /lib64 is not part of the search path. It's a hard-coded path. Running ldd in a ubuntu container shows that path actually exists on disk. Maybe libraries can query a search path, but the interpreter must be static? I queried every command in my OS and they all provide a static path for the ELF interpreter... I dunno. I'm out of my depth 🤷

Is patchelf a command that you run once to fix the linking [...]?

Yes, I only need to run it once after the first npm install, or any time the executable changes (e.g. updating the wrangler package).

Where do I get the interpreter path for my machine [...]?

I pulled it from ldd ./workerd. It prints the linked interpreter, but it doesn't actually seem to use it.

# ...
libc.so.6 => /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib/libc.so.6
/lib64/ld-linux-x86-64.so.2 => /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib64/ld-linux-x86-64.so.2
librt.so.1 => /nix/store/qn3ggz5sf3hkjs2c797x.........

I just took the path that it printed and used that in the patchelf command. Then, because I'm afraid I'd forget what happened the next time around, I plugged it into a shell hook in my flake file:

{
  description = "Development environment";

  outputs = { self, nixpkgs }:
    let inherit (nixpkgs) lib;

    in {
      devShell = lib.genAttrs lib.systems.flakeExposed (system:
        let pkgs = import nixpkgs { inherit system; };
        in pkgs.mkShell {
          nativeBuildInputs = [ pkgs.nodejs ];

          # See: https://github.com/cloudflare/workerd/issues/1482
          shellHook = ''
            __patchTarget="./node_modules/@cloudflare/workerd-linux-64/bin/workerd"
            if [[ -f "$__patchTarget" ]]; then
              ${pkgs.patchelf}/bin/patchelf --set-interpreter ${pkgs.glibc}/lib/ld-linux-x86-64.so.2 "$__patchTarget"
            fi
          '';
        });
    };
}

Now every time I enter my dev shell it will auto-correct the workerd interpreter path. You may need to change __patchTarget if your package manager installs it somewhere else.

[PsychoLlama]

@jonahbron Ha, while poking around, I discovered a simpler alternative: remove wrangler from your package.json and install it through pkgs.wrangler instead.

I think through troubleshooting this issue, I rediscovered why the fixup phase of stdenv exists 😅

[jonahbron]

Oh my god you're a life saver @PsychoLlama, installing with pkgs.nodePackages.wrangler worked great! Thank you thank you. The only hitch I ran into was I had the flake in a parent directory of the package.json, and it didn't like not having a package.json around.

error: builder for '/nix/store/lvb0wayv3z0qly8jkql3ya0rb0nrffym-wrangler-3.16.0.drv' failed with exit code 1;
       last 10 log lines:
       > npm ERR!               ^
       > npm ERR!
       > npm ERR! Error: Failed to install package "@cloudflare/workerd-linux-64"
       > npm ERR!     at checkAndPreparePackage (/nix/store/rpywngw6pvm8nd4iny8iidsh2nmdqrwp-wrangler-3.16.0/lib/node_modules/wrangler/node_modules/workerd/install.js:242:15)
       > npm ERR!     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
       > npm ERR!
       > npm ERR! Node.js v18.18.2
       >
       > npm ERR! A complete log of this run can be found in: /build/.npm/_logs/2023-12-29T07_01_26_669Z-debug-0.log

Moving the flake down into the child directory fixed that issue. Only drawback is I have to invoke wrangler directly (can't have it in a package.json) script because it will search in node_modules/.bin first.

[jasnell]

I'm going to move this issue to a discussion as it does not appear as if this is actually an issue to resolve in workerd itself.

[CMCDragonkai]

NixOS uses a non-standard location for the glibc loader. Therefore compiled binaries that assume the Linux FHS won't run on NixOS. This is a well-known problem in the NixOS community, and it is by-design.

Binaries compiled inside Nix will of course pick up the right glibc loader from the /nix/store.

To run unpatched binaries you need to use steam-run or an FHS simulator, there's a bunch workarounds too here https://github.com/Mic92/nix-ld.

Although ideally the whole thing would be a static binary to avoid problems.

[CMCDragonkai]

So this is the best way to do it:

    patchelf \
      --set-interpreter $(cat $NIX_CC/nix-support/dynamic-linker) \
      $(which workerd)

Just make sure that patchelf exists in your mkShell.

[mikea]

On a nix-related note: I have tried to produce shell.nix configuration to build workerd from sources. Unfortunately I can't figure out how to use clang and libc++ together:

{ pkgs ? import <nixpkgs> {} }:

let
    stdenv = pkgs.llvmPackages_17.libcxxStdenv;
    clang = pkgs.llvmPackages_17.libcxxClang;
in
  stdenv.mkDerivation rec {
    name = "workerd-env";
    buildInputs = with pkgs; [
      bazelisk clang
    ];
}

nix-shell --command 'bazel build //...' gives me various libc++ related errors:

clang: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
external/com_google_absl/absl/base/internal/strerror.h:18:10: fatal error: 'string' file not found

Nix help is appreciated.

[patonw]

My work-around is to hook into npm's postinstall phase with:

"postinstall": "command -v patchelf && patchelf --set-interpreter $(ldd node_modules/@cloudflare/workerd-linux-64/bin/workerd | grep -oh '/nix/store\\S*ld-linux\\S*') node_modules/@cloudflare/workerd-linux-64/bin/workerd"

It'd be cleaner to stuff this into shell.nix using writeShellScript if you're sufficiently motivated.

Benefit of this method is that it runs every time a new package is added. Otherwise yarn overwrites the local node_modules with the bad copy and I have to manually trigger patchelf. I'll try nix-ld next time.

[dbaynard]

Thanks for this — it's the most elegant solution. (I use pnpm so I was able to run pnpm patch '@cloudflare/workerd-linux-64' and add this to the resulting package.json.)

I've prepended the equivalent of node_modules/@cloudflare/workerd-linux-64/bin/workerd --help || before the command so that patchelf is only required when the binary doesn't work. This way it should work on non-NixOS systems (such as github actions) — I haven't tested it on non-NixOS yet, though.

[unlux]

@patonw did you find the correct nix-ld config? i want to use that but am unable to find which pkg to put in it.

[patonw]

I ended up putting nix-ld into my system-wide configuration.nix for other NPM dependencies and it fixed workerd as a side-effect.

Not all of these libraries might be needed for workerd, but they don't break anything as far as I've experienced.

  programs.nix-ld.enable = true;
  programs.nix-ld.libraries = with pkgs; [
    stdenv.cc.cc
    zlib
    fuse3
    icu
    nss
    openssl
    curl
    expat
  ];

I haven't tried it via just mkShell, but it should still work.

[patonw]

Just tried via mkShell (still need to enable the nix-ld feature system-wide but without defining libraries).

It works without any libraries as long as NIX_LD is defined:

    NIX_LD = lib.fileContents "${stdenv.cc}/nix-support/dynamic-linker";

[unlux]

  programs.nix-ld.enable = true;
  programs.nix-ld.libraries = with pkgs; [
    stdenv.cc.cc
    zlib
    fuse3
    icu
    nss
    openssl
    curl
    expat
  ];

this worked like a charm, just added this to config and rebuilt, and it worked! didnt even have to make a dev shell for it, thanks man.

[jkxyz]

Only programs.nix-ld.enable = true was required for me. Logging out and back in sets the NIX_LD environment variable.

[unlux]

it didnt work for me without the pkgs added initially, so i asked for the config. i've kept it in #2382 to be on the safer side and avoid future issues.