native-deps.org

Native Dependencies and Packages with Installs

While the vast majority of JavaScript projects “just work” out of the box when you unpack them from a registry tarball, a few require [pre|post]install scripts to be run to make themselves ready for use.

By default floco will run this install phase in an isolated sandbox, providing only declared dependencies and optionalDependencies, and a few common tools such as node, python3, jq, node-gyp, and stdenv ( providing a C/C++ compiler, make, and other common utilities ). This is often sufficient for most installs but some may also require “native” libraries that aren’t declared in manifests in a standard way, and instead get mentioned in a README.

Unlike other JavaScript package managers - floco has a way to declare these dependencies so that these builds can be run in a reproducible manner across platforms; all we need to do is add a few fields to our config files.

Packaging zeromq@5.3.1 and zeromq@6.0.0.0-beta.16

For our example we’re going to package zeromq@5.3.1 and zeromq@6.0.0-beta.16 which run node-gyp compilation stages to complete their installs.

I’ve chosen 5.3.1 because it’s relatively straightforward with a small snag that we can deal with easily - a great intro.

Next I’ve chosen 6.0.0-beta.16 to cover some practical techniques of locating native deps in nixpkgs. Its binding.gyp ( build recipe ) is written in a way that we can configure to build libzmq “from scratch”, or link against a shared libzmq from the host system. It’s possible to package the static build for this module, and I did get it to work with some effort; but because it attempts to use curl inside of its Makefile it required a heavy handed approach that is “out of scope” for this guide. We’ll just cover the shared library case here.

Preparing a Workspace

For both of the versions we’ll use a common base project. You can initialize a workspace by running the following. Note that this initialization process can be used for any registry package.

mkdir -p ./zeromq-5.3.1 ./zeromq-6.0.0-beta.16;
cd ./zeromq-5.3.1;
nix run floco#fromRegistry -- -tp zeromq@5.3.1;
# Aggregates `floco' configuration modules.
echo '{ imports = [./pdefs.nix ./foverrides.nix]; }' > floco-cfg.nix;
# We'll define a real `foverrides.nix' in later sections, this is a stub.
echo '{}' > foverrides.nix;
# Provides a CLI frontend to module system.
echo '
{ floco  ? builtins.getFlake "github:aakropotkin/floco"
, lib    ? floco.lib
, system ? builtins.currentSystem
, ...
}: let
  mod = lib.evalModules {
    modules = [
      floco.nixosModules.default
      ./floco-cfg.nix
      { floco.settings = { inherit system; }; }
    ];
  };
in mod.config.floco.packages.zeromq."5.3.1".global
' > ./default.nix;

# Onto the next workspace
cd ../zeromq-6.0.0-beta.16;
nix run floco#fromRegistry -- -tp zeromq@6.0.0-beta.16;
cp ../zeromq-5.3.1/{default,floco-cfg,foverrides}.nix .;
sed -i 's/5\.3\.1/6.0.0-beta.16/' ./default.nix;
cd ..;

Naviely Compiling a Static libzmq ( 5.3.1 )

The default behavior defined by this projects’ node-gyp builder is to build from source to create a static library unless the user has set some special environment flags to indicate that a shared library should be used.

Like I said before, the process given above will work for the vast majority of packages, so we’ll just give this a shot as is.

nix build -f ./zeromq-5.3.1 -L --no-link 2>&1;

...SNIP...
zeromq-installed> make: Leaving directory '/private/tmp/nix-build-zeromq-installed-5.3.1.drv-0/source/build'
zeromq-installed> gyp info ok
zeromq-installed> buildPhase completed in 30 seconds
zeromq-installed> installing
zeromq-installed> post-installation fixup
zeromq-installed> checking for references to /private/tmp/nix-build-zeromq-installed-5.3.1.drv-0/ in /nix/store/wrzjzdm0sq3kfq0bddc2w9f680ydkcs4-zeromq-installed-5.3.1...
zeromq-installed> /nix/store/g0bmb8iq70gf1yk6pr50i2bp7gxfz77i-audit-tmpdir.sh: line 23: patchelf: command not found
zeromq-installed> /nix/store/g0bmb8iq70gf1yk6pr50i2bp7gxfz77i-audit-tmpdir.sh: line 23: patchelf: command not found
zeromq-installed> /nix/store/g0bmb8iq70gf1yk6pr50i2bp7gxfz77i-audit-tmpdir.sh: line 23: patchelf: command not found
zeromq-installed> /nix/store/g0bmb8iq70gf1yk6pr50i2bp7gxfz77i-audit-tmpdir.sh: line 23: patchelf: command not found
zeromq-installed> /nix/store/g0bmb8iq70gf1yk6pr50i2bp7gxfz77i-audit-tmpdir.sh: line 23: patchelf: command not found
zeromq-installed> /nix/store/g0bmb8iq70gf1yk6pr50i2bp7gxfz77i-audit-tmpdir.sh: line 23: patchelf: command not found
zeromq-installed> patching script interpreter paths in /nix/store/wrzjzdm0sq3kfq0bddc2w9f680ydkcs4-zeromq-installed-5.3.1
zeromq-installed> strip is /nix/store/8sigdq7hayfxqkxahjs70s6ny42wfwgh-Toolchains/XcodeDefault.xctoolchain/bin/strip
zeromq-installed> stripping (with command strip and flags -S) in  /nix/store/wrzjzdm0sq3kfq0bddc2w9f680ydkcs4-zeromq-installed-5.3.1/lib

And would ya look at that - it worked right out of the box!

Well, sort of… If we take a look at the end of our log there you’ll notice a few warnings that say patchelf: command not found. For context this is log was created by a aarch64-darwin machine, and will not produce this error for linux boxes.

While this isn’t technically an error ( and in this specific instance the warnings are completely benign ), in general it leaves the door open for potentially hard to debug issues for consumers of the library. Luckily there’s an easy fix.

Static libzmq with patchelf ( 5.3.1 )

The utility patchelf won’t be covered in length here, but suffice to say that it fixes up binaries so that they can link dynamic libraries without LD_LIBRARY_PATH, helping to purify them. This tool is only used to patch ELF binary formats and in general isn’t needed on Darwin; but in this case the zeromq package ships out with some pre-compiled ELF artifacts that Nix has detected and is trying to patch.

The problem here is that on Darwin stdenv doesn’t provide patchelf, so we’ll need to add it to the build sandbox explicitly.

We’ll do this using the foverrides.nix file I mentioned before to add some extra config to this build recipe. This file is a module just like any other, in this case we’ll want to make it a function which takes pkgs as an argument so that we can reference the patchelf derivation defined by nixpkgs.

# zeromq-5.3.1/foverrides.nix
{
  # The `packages' records are created automatically from `pdefs' and
  # hold `derivations' associated with the package, representing stages of
  # its preparation.
  # In this case we'll configure the `installed' "target" to add a
  # native dependency.
  config.floco.packages.zeromq."5.3.1".installed = { pkgs, ... }: {
    config.extraBuildInputs = [pkgs.patchelf];
  };
}

That’s all we have to do. floco provides the options extra[Native]BuildInputs, override, and overrideAttrs for both the installed and built targets for handling common tasks like this.

A notable behavior of extra[Native]BuildInputs and override is that they may be defined multiple times in multiple files/modules. These definitions will be merged together so that definitions with the same priority are joined using ++ for lists, and // for attrsets. This merging behavior can be leveraged to a great degree when organizing overrides and extensions in large codebases. See the NixOS Manual for more details on merging behaviors.

Providing a Shared Library for libzmq ( 6.0.0-beta.16 )

Now we’ll ramp up the difficulty by trying to build a later release of zeromq@6.0.0-beta.16 with a shared library pulled from nixpkgs.

I’m writing this guide without having packaged this before so for all we know this might not work because this project is a beta release or a compatibility issue with the nixpkgs libs; but this is probably a good thing since it’ll allow me to cover some practical debugging techniques.

In this example we’ll use override to set some extra environment variables, and we’ll use extraBuildInputs again to add a shared libzmq. We’ll also conditionally add libsodium if the package is being built for Darwin.

A Naive Attempt

Just like before lets just give the naive recipe a shot. As a reminder this build is run on aarch64-darwin, and this backtrace will not appear on linux ( more on that caveat later ).

nix build -f ./zeromq-6.0.0-beta.16 -L --no-link;

zeromq-installed> /nix/store/n0k8njvgg6yjapkl81rm821s9vx0qrwb-bash-5.2-p15/bin/sh: line 1: pkg-config: command not found
zeromq-installed> gyp: Call to 'pkg-config libsodium --libs' returned exit status 127 while in binding.gyp. while trying to load binding.gyp
zeromq-installed> gyp ERR! configure error
zeromq-installed> gyp ERR! stack Error: `gyp` failed with exit code: 1
zeromq-installed> gyp ERR! stack     at ChildProcess.onCpExit (/nix/store/pjrp2b9c0kj2v98nn8fmmnq5gxp38aq1-node-gyp-9.3.1/lib/node_modules/node-gyp/lib/configure.js:325:16)
zeromq-installed> gyp ERR! stack     at ChildProcess.emit (events.js:400:28)
zeromq-installed> gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:285:12)
zeromq-installed> gyp ERR! System Darwin 21.4.0
zeromq-installed> gyp ERR! command "/nix/store/7fs3x8nji7msymvlw1dxs1bf34d6hwc7-nodejs-14.21.2/bin/node" "/nix/store/pjrp2b9c0kj2v98nn8fmmnq5gxp38aq1-node-gyp-9.3.1/bin/.node-gyp-wrapped" "rebuild"
zeromq-installed> gyp ERR! cwd /private/tmp/nix-build-zeromq-installed-6.0.0-beta.16.drv-0/source
zeromq-installed> gyp ERR! node -v v14.21.2
zeromq-installed> gyp ERR! node-gyp -v v9.3.1
zeromq-installed> gyp ERR! not ok
zeromq-installed> /nix/store/w2krpzg514ffrpsk2flf8bbkw7dy463c-floco-hooks/nix-support/setup-hook: line 43: pop_var_context: head of shell_variables not a function context
zeromq-installed> /nix/store/11kqdpgbaj7d3vp6kn5d35jspg5isjzv-stdenv-darwin/setup: line 1594: pop_var_context: head of shell_variables not a function context
error: builder for '/nix/store/kf6p1wv3v78ff0p9nj9wf0xjn1i7x0ar-zeromq-installed-6.0.0-beta.16.drv' failed with exit code 1;

Alright lets dive into the backtrace. Looks like pkg-config is missing, and line two also shows us what it was searching for; this tells us we’ll need a libsodium.pc file.

First lets do some homework and read the binding.gyp file so we can look for any platform dependenant quirks to watch out for. We’d like to avoid accidentally adding/missing native deps or configuration options that are only applicable to some systems.

Context From zeromq Tree

For a bit of context I’ll include a snippet from a few files in the distributed tarball for the package:

package.json

We can get a look at the install they’ve defined.

This script does not need to be defined when binding.gyp is present; but if it is the package.json script is what gets run, otherwise projects just run node-gyp rebuild.

"install": "(shx test -f ./script/build.js || run-s build.js) && cross-env npm_config_build_from_source=true node-gyp-build",

I haven’t got a clue what shx is, but I recon it’s some sort of portability wrapper used to run the script ./script/build.js.

It’s very common for projects to execute something like postinstall.js in their install script; in this case it looks like the authors decided to go with the name build.js which is somewhat misleading if you subscribe the the conventional npm and yarn terminology for “builds” and “installs”; but I digress.

binding.gyp

This is the build recipe run by node-gyp. The format is some bastard child born of JSON + Python3 object syntax. These are declarative wrappers around an underlying Makefile, often produced by CMake which adds yet another layer of indirection between the developer and CC / LD.

The declared variables are effectively arguments, and you can set them using environment variables by adding the prefix npm_config_<NAME>. Don’t forget the prefix. While writing this guide I forgot the prefix and spent like 30 minutes accidentally debugging the static build because node-gyp ignored my environment variables that lacked the prefix.

{
  'variables': {
    'zmq_shared%': 'false',
    'zmq_draft%': 'false',
    'zmq_no_sync_resolve%': 'false',
    'sanitizers%': 'false',
    'openssl_fips': '',
    'runtime%': 'node',
  },

# ...<SNIP>...

        ["zmq_shared == 'true'", {
          'link_settings': {
            'libraries': ['-lzmq'],
          },
        }, {
          'conditions': [
            ['OS == "mac"', {
              'libraries': [
                '<(module_root_dir)/build/libzmq/lib/libzmq.a',
                "<!@(pkg-config libsodium --libs)",
              ],
            }],

# ...<SNIP>...

}

This snippet indicates that the builder is sensitive to an environment variable npm_config_zmq_shared ( among others ) which has a default value of false, and that when building on Darwin with zmq_shared = true, it will use pkg-config to locate libsodium.

It’s a good thing we checked the binding.gyp because if I hadn’t I’d have assumed libsodium was required for all platforms. The research paid off.

scripts/build.js

Next lets take a look at the script they call from their install routine.

It’s just a JS file, but at the bottom I noticed they have a block that seems to add some addition CMake flags for certain platforms, and they do so by checking the ARCH environment variable.

I’m pointing this out now because we have to set this ourselves because ( spoiler alert ) an issue we run into later requires us to set this manually.

// ...<SNIP>...

function archCMakeOptions() {
  const arch = (process.env.ARCH || process.arch).toLowerCase()
  // ...<SNIP>...
  if (process.platform === "darwin") {
    // handle MacOS Arm
    switch (arch) {
      case "x64":
      case "x86_64": {
        return ""
      }
      case "arm64": {
        return ` -DCMAKE_OSX_ARCHITECTURES=${arch}`
      }
      default: {
        return ""
      }
    }
  }
}

Finding libsodium in Nixpkgs

Since we know that the build is going to look for libsodium on Darwin, we need to make sure that we have pkg-config AND that libsodium.pc is available in the build sandbox. To provide these lets search in Nixpkgs a bit:

nix search nixpkgs '\.libsodium';

* legacyPackages.aarch64-darwin.libsodium (1.0.18)
  A modern and easy-to-use crypto library

Easy enough. Now lets make see if libsodium.pc is provided in the default output, or if we need to use a secondary output such as lib or dev to get the pkg-config metadata.

# Lets try the default output ( comes back empty )
find "$( nix build nixpkgs#libsodium --no-link --print-out-paths; )"  \
     -name '*.pc'|grep .||echo NONE;

# Lets look for alternative outputs.
nix eval nixpkgs#libsodium.outputs;

# Lets try `dev' ( BINGO! )
find "$( nix build nixpkgs#libsodium.dev --no-link --print-out-paths; )" \
     -name '*.pc'|grep .||echo NONE;

NONE
[ "out" "dev" ]
/nix/store/820s23l9i9lqksg1dsxyxjgcsi2q3gp0-libsodium-1.0.18-dev/lib/pkgconfig/libsodium.pc

This tells us we need to add pkgs.libsodium.dev for pkg-config to resolve our library.

Next lets look for a shared library form of libzmq, being libzmq.so on linux, or libzmq.dylib on Darwin.

nix search nixpkgs '\.(libzmq|zeromq)';

* legacyPackages.aarch64-darwin.lispPackages_new.sbclPackages.zeromq (20160318-git)
* legacyPackages.aarch64-darwin.lispPackages_new.sbclPackages.zeromq_dot_tests (20160318-git)
* legacyPackages.aarch64-darwin.octavePackages.zeromq (7.3.0-zeromq-1.5.3)
  ZeroMQ bindings for GNU Octave
* legacyPackages.aarch64-darwin.zeromq (4.3.4)
  The Intelligent Transport Layer
* legacyPackages.aarch64-darwin.zeromq4 (4.3.4)
  The Intelligent Transport Layer

The last two look right to me since the earlier results appear to be modules/packages for octave and LISP. Because both of the final results have the same version number and description, my bet is that they’re aliases of one another.

I have some concerns about the 4.x major version number though. I’ll cross my fingers and hope that the version number used by the JS module doesn’t necessarily correspond to the C library. Like I said, I haven’t packaged this before so this type of hiccup was always a risk.

Writing the Recipe

We’ll extend our foverrides.nix file from before:

Lets start with these additions based on what learned in our research above.

# zeromq-6.0.0-beta.16/foverrides.nix
{
  config.floco.packages.zeromq."6.0.0-beta.16" = {
    installed = { pkgs, ... }: {

      config.extraBuildInputs = [
        # Always add these.
        pkgs.zeromq
      ] ++ ( if ! pkgs.stdenv.hostPlatform.isDarwin then [] else [
        # Only add these for when the host system is `darwin'.
        pkgs.pkg-config
        pkgs.libsodium.dev
      ] );

      # Setting `override' attrs causes them to be set on the underlying
      # derivation, which then get set as environment variables in the
      # sandbox where we run out install.
      # We want to tell `node-gyp' to look for the shared `libzmq', so
      # we'll set the variable we found in their `binding.gyp' file.
      # XXX: You must quote "true" because `binding.gyp' expects a
      # string, and a Nix boolean of `false' gets stringized as the
      # empty string.
      config.override.npm_config_zmq_shared = "true";

    };
  };
}

And if we run another build, we survive past our previous crash, but we’ve got a new one.

zeromq-installed> gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
zeromq-installed> make: Entering directory '/private/tmp/nix-build-zeromq-installed-6.0.0-beta.16.drv-0/source/build'
zeromq-installed>   TOUCH Release/obj.target/libzmq.stamp
zeromq-installed>   CXX(target) Release/obj.target/zeromq/src/context.o
zeromq-installed> error: unknown target CPU 'armv8.3-a+crypto+sha2+aes+crc+fp16+lse+simd+ras+rdm+rcpc'
zeromq-installed> note: valid target CPU values are: nocona, core2, penryn, bonnell, atom, silvermont, slm, goldmont, goldmont-plus, tremont, nehalem, corei7, westmere, sandybridge, corei7-avx, ivybridge, core-avx-i, haswell, core-avx2, broadwell, skylake, skylake-avx512, skx, cascadelake, cooperlake, cannonlake, icelake-client, icelake-server, tigerlake, knl, knm, k8, athlon64, athlon-fx, opteron, k8-sse3, athlon64-sse3, opteron-sse3, amdfam10, barcelona, btver1, btver2, bdver1, bdver2, bdver3, bdver4, znver1, znver2, x86-64

This backtrace looks like a failure to detect the system’s architecture. I can’t say why it failed, but experience tells me that the conflicting output people get from arch and uname CLI commands between various implementations is usaully the root cause. In any case, we noticed before that the build.js script checks an environment variable ARCH, so we might try setting that. In that file we’ll find the exact patterns they expect which are “x86_64”, and “arm64”, which we can set based on info pulled out of stdenv.

Here’s another draft of foverrides.nix:

# zeromq-6.0.0-beta.16/foverrides.nix
{
  config.floco.packages.zeromq."6.0.0-beta.16" = {
    installed = { pkgs, ... }: {

      config.extraBuildInputs = [
        # Always add this one.
        pkgs.zeromq
      ] ++ ( if ! pkgs.stdenv.hostPlatform.isDarwin then [] else [
        # Only add these for when the host system is `darwin'.
        pkgs.pkg-config
        pkgs.libsodium.dev
      ] );

      # Setting `override' attrs causes them to be set on the underlying
      # derivation, which then get set as environment variables in the
      # sandbox where we run out install.
      # We want to tell `node-gyp' to look for the shared `libzmq', so
      # we'll set the variable we found in their `binding.gyp' file.
      # XXX: You must quote "true" because `binding.gyp' expects a string,
      # and a Nix boolean of `false' gets stringized as the empty string.
      config.override.npm_config_zmq_shared = "true";
      config.override.ARCH                  =
        if pkgs.stdenv.hostPlatform.isx86_64 then "x86_64" else "arm64";

    };
  };
}

Lets see how we did:

nix build -f ./zeromq-6.0.0-beta.16 -L --no-link;

...<SNIP>...
zeromq-installed> make: Leaving directory '/private/tmp/nix-build-zeromq-installed-6.0.0-beta.16.drv-0/source/build'
zeromq-installed> gyp info ok
zeromq-installed> @nix { "action": "setPhase", "phase": "installPhase" }
zeromq-installed> installing
zeromq-installed> post-installation fixup
zeromq-installed> checking for references to /private/tmp/nix-build-zeromq-installed-6.0.0-beta.16.drv-0/ in /nix/store/2ra6949ynpbs3y3l57y0wa69mhdyr7il-zeromq-installed-6.0.0-beta.16...
zeromq-installed> /nix/store/g0bmb8iq70gf1yk6pr50i2bp7gxfz77i-audit-tmpdir.sh: line 23: patchelf: command not found
zeromq-installed> /nix/store/g0bmb8iq70gf1yk6pr50i2bp7gxfz77i-audit-tmpdir.sh: line 23: patchelf: command not found
zeromq-installed> patching script interpreter paths in /nix/store/2ra6949ynpbs3y3l57y0wa69mhdyr7il-zeromq-installed-6.0.0-beta.16
zeromq-installed> strip is /nix/store/8sigdq7hayfxqkxahjs70s6ny42wfwgh-Toolchains/XcodeDefault.xctoolchain/bin/strip
zeromq-installed> stripping (with command strip and flags -S) in  /nix/store/2ra6949ynpbs3y3l57y0wa69mhdyr7il-zeromq-installed-6.0.0-beta.16/lib

And we have a winner!