Random segfaults on the MacOS CI

[thufschmitt]

Describe the bug

The MacOS CI randomly segfaults on nix flake check. Some other Nix-based CIs also seems to encounter that (https://github.com/tweag/nickel/actions/runs/4532944265/jobs/7985404911#step:5:24 for instance).

Steps To Reproduce

https://github.com/NixOS/nix/actions/runs/4616918102/jobs/8162538381

Additional context

• According to the Nickel team, disabling the GC seems to fix it
• This happens on Nix 2.13.3 (which has been here for some time), but started triggering only a few weeks ago, probably because of some changes in the runners environment
• I tried debugging this by adding -vvvvv, but I couldn't manage to reproduce it then (https://github.com/tweag/nix/actions/runs/4627359431/jobs/8191562562)

Priorities

Add 👍 to issues you find important.

[thufschmitt]

Next step is trying to get code dumps.
https://gist.github.com/kudaba/53f68340bd28cd1e8824ac1e6ba8cf88 and https://stackoverflow.com/questions/9412156/how-to-generate-core-dumps-in-mac-os-x seems to give the right way of getting and exporting them from GHA

[abathur]

Crash dumps are usually visible in console.app, though SO tells me they might also be in ~/Library/Logs/DiagnosticReports

@grahamc @cole-h has detsys seen this on the macs you run? Wondering since you might be able to search for and eject crash reports as part of your wipe process?

[thufschmitt]

Crash dumps are usually visible in console.app, though SO tells me they might also be in ~/Library/Logs/DiagnosticReports

Oh, I didn't know about these Crash dumps. Looks like this would have been much more lightweight than a full core dump. Good to know for a next time.

At any rate, I eventually managed to get a core dump at https://github.com/tweag/nix/suites/12088097593/artifacts/636572529. I'll download it and see what I can make of it.

[thufschmitt]

Unsurprisingly the stack trace mentions a mix of boehm and boost coroutines.
I had totally forgotten about that, but https://github.com/NixOS/nix/pull/7725/files never got backported (and got reverted on master). I wonder whether it could fix this issue

[thufschmitt]

#7725 seems to fix it indeed (at least I couldn't reproduce it on tweag@15ce51f). I've just opened #8179 to add the non-tests part of #7725 back

[thufschmitt]

Closing since 2.15 go relased, and hopefully fixes this.

Anyone, feel free to reopen if the problem arises again