resolve-system-dependencies.pl segfaults if Nix is used on Mac OS X in combination with Hydra's queue runner

[svanderburg]

When using Nix 1.11.x, and Hydra Mac OS X builds refuse to build. In the queue runner log, I observed the following:

possibly transient failure building ‘/nix/store/11hdvirhq6xzn26b81ykm97jchpjs3n2-package.drv’ on ‘sander@192.168.88.63’: program ‘/nix/store/zg7r946ndp3fag3bxy0v5xi3vw624jvi-nix-1.11.1/libexec/nix/resolve-system-dependencies.pl’ failed due to signal 11 (Segmentation fault: 11)
Jan 29 17:28:40 milou hydra-queue-runner[17848]: will retry ‘/nix/store/11hdvirhq6xzn26b81ykm97jchpjs3n2-package.drv’ after 180s
Jan 29 17:28:41 milou hydra-queue-runner[17848]: performing step ‘/nix/store/44f7jw17h3bwpvrccy1dy0c879qdcj98-package.drv’ on ‘sander@192.168.88.63’ (needed by build 123896 and 1 others)
Jan 29 17:28:41 milou hydra-queue-runner[17848]: killing process 19024
Jan 29 17:28:41 milou hydra-queue-runner[17848]: possibly transient failure building ‘/nix/store/44f7jw17h3bwpvrccy1dy0c879qdcj98-package.drv’ on ‘sander@192.168.88.63’: program ‘/nix/store/zg7r946ndp3fag3bxy0v5xi3vw624jvi-nix-1.11.1/libexec/nix/resolve-system-dependencies.pl’ failed due to signal 11 (Segmentation fault: 11)
Jan 29 17:28:41 milou hydra-queue-runner[17848]: will retry ‘/nix/store/44f7jw17h3bwpvrccy1dy0c879qdcj98-package.drv’ after 181s
Jan 29 17:28:42 milou hydra-queue-runner[17848]: performing step ‘/nix/store/klkj7l2gn7ms9xqbn1hwi4jk8sxjdrg0-package.drv’ on ‘sander@192.168.88.63’ (needed by build 123899 and 2 others)
Jan 29 17:28:42 milou hydra-queue-runner[17848]: killing process 19034
Jan 29 17:28:42 milou hydra-queue-runner[17848]: possibly transient failure building ‘/nix/store/klkj7l2gn7ms9xqbn1hwi4jk8sxjdrg0-package.drv’ on ‘sander@192.168.88.63’: program ‘/nix/store/zg7r946ndp3fag3bxy0v5xi3vw624jvi-nix-1.11.1/libexec/nix/resolve-system-dependencies.pl’ failed due to signal 11 (Segmentation fault: 11)
Jan 29 17:28:42 milou hydra-queue-runner[17848]: will retry ‘/nix/store/klkj7l2gn7ms9xqbn1hwi4jk8sxjdrg0-package.drv’ after 180s

As far as I know, the issue is caused by the fact that the script tries the open the corresponding store derivation file, which does not seem to exists.

The reason why this drv-file does not exist is that hydra-queue-runner transfers an in-memory representation of a derivation closure from the coordinator machine to each target (using the Nix daemon protocol), instead of actual files, for performance reasons. As a result, this script consistently fails for all builds.

This makes it in impossbile to use Hydra for building anything Mac OS X related.

[svanderburg]

Downgrading to nix-1.10 on the Mac OS X build slave, seems to have solved this issue for me atm...

[ghost]

Any updates on this? I seem to be hitting other issues with 1.10 and would like to upgrade.

[edolstra]

The workaround is to add pre-build-hook = to /etc/nix/nix.conf. At least that's what I do on our build machines: https://github.com/NixOS/nixos-org-configurations/blob/master/delft/scripts/nix-mac-installer.sh#L36

[matthewbauer]

This has been coming up again lately:

https://hydra.nixos.org/build/51897213

Any ideas what's going on?

[copumpkin]

@pikajude ?

[LnL7]

I think mac2 was offline for a while and wasn't configured with pre-build-hook =