Improvements to the NixOS Hardened Profile

[kmcopper]

Motivation for this change

Use scudo allocator for the hardened profile by default. Includes changes to compiler-rt to make it more compatible by default with things like man see #65000 for some details. Turning these two options off by default will prevent issues in programs from crashing the malloc, which is critical if we are replacing it as the system malloc. These options are all able to be overridden by simple environment variables even when compiled in so this should be a very safe change. I enabled the allocator zeroing by default in the hardened profile only. The buildSandbox option is currently incompatible with the hardened profile so it should be defaulted to off unless you manually re-enable both it and user namespaces.

I tested in a VM using the master branch. I used the hardened profile for the guest. I used both standard llvmPackages.compiler_rt and llvmPackages_9.compiler_rt for testing and both work. I mostly tested these options affects on programs by using man but I've also noticed needing these options as an environment variable for qemu to function well.

I feel like this should be backported to 19.09 so that the hardened profile can take advantage of it as was intended in #65000 for 19.09 (albeit with the graphene_malloc allocator). I feel the clang allocator has a much more stable history and should be the mkDefault here. I was unsure if and how to make the environment variable depend on the scudo allocator being the default but I do feel like it's safe to set in to profile anyway since it can be overridden easily.

I wasn't sure if maintainers wanted me to apply the compatibility all the way back to 4 which does still have scudo. I included 8 because I'm not sure what is going to land as the default for 20.03

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• [] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

Notify maintainers

cc @joachifm @lovek323 @dtzWill @7c6f434c

[joachifm]

Hi, this LGTM

I think it's inappropriate to backport using scudo by default to 19.09. Non-essential changes that have some chance of producing observable side-effects violate the implied stable "contract", as I understand it.

If the other changes are deemed safe, we can backport those, however.

The compiler-rt changes should go via staging, due to the darwin mass rebuild.

[kmcopper]

Depends on #73766 now as per comments.

[kmcopper]

I was hoping to get this merged and in 20.03 is it too late now to get this into that?

[joachifm]

Sorry I completely forgot about this

[Mic92]

If @joachifm is too busy, maybe @emilazy or @yegortimoshenko can have a look.

[joachifm]

Thank you. Please open PRs against the release branch for backports.

[flokli]

Urgh - disabling the nix sandbox by default, just because Nix doesn't play well with disabled user namespaces sounds like a very scary hack to me. Not really a fan of 00ac71a TBH.

[joachifm]

Please elaborate.

[flokli]

I consider disabling the Nix sandbox by default dangerous and a huge impurity. Also, see NixOS/nix#3006 (review) why disabling the sandbox alltogether isn't really nice. With this PR merged, shouldn't Nix degrade gracefully?

[flokli]

I need to admit, I don't exactly know how and when Nix can enable (some) sandboxing these times, but disabling it alltogether sounds like a bad idea.

[emilazy]

I agree, it would be far preferable to merge the kernel patch to allow disabling unprivileged userns creation than it is for the hardened profile to reduce security by silently disabling build sandboxing entirely.

[dasJ]

Just found this, I think it would have been better to allow user namespaces rather than disabling sandboxing.
Ever since we switched to the hardened profile, we use boot.kernel.sysctl."user.max_user_namespaces" = 512; which sounds a lot less frightening than disabling a sandbox.

[joachifm]

Alright, it seems prevailing consensus is to revert that commit.

IMO, allowing arbitrary code execution on a machine is anathema to "hardening" to begin with. I consider build sandboxing to be mainly a development aide. I'm unconvinced that disabling the sandbox makes you materially worse off than you already are by handing out access to a C toolchain and all the tools you'd ever need for exploiting bugs in the host kernel. If you're serious about the security of your deployment, do not allow it to run build jobs.

[emilazy]

I run the hardened profile on all my NixOS machines and do development on them; I'm sure I'm not alone. The defence in depth hardening features are more useful when running untrusted code, not less. Meanwhile, there's no good reason to use unsandboxed builds at all unless you're in a platform that doesn't support them.

[flokli]

Reverted that commit in a8989b3.

[emilazy]

For what it's worth, though I support using a hardened allocator, this currently makes Firefox crash on startup. This doesn't seem to be scudo specific – I get the same results with jemalloc too – but it's unfortunately challenging to work around (seemingly requires a new mount namespace).

It's well-documented that Firefox uses its own jemalloc allocator which can't be overridden without additional effort (see e.g. the Whonix wiki), but a hard SIGSEGV at startup (with the firefox package; the binary packages start the crash reporter) seems somewhat unfortunate. Will continue poking, but for now, anyone wanting to use Firefox with the hardened profile can revert to:

environment.memoryAllocator.provider = "libc";

[Flakebi]

Thanks for working on hardening NixOS @kmcopper and @emilazy.
As I was struck by segfaults in the last few days and it took a while to find this issue, I want to add this here:
The freshly enabled allocator leads to segfaults in firefox (as you mentioned), thunderbird, rustc (yeah, all the mozilla related tools) and some background services that get started by evolution.

The easiest way to get a segfault that I found is running rustc --crate-name ___ --print=file-names --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg /dev/null (in case it is useful for someone).

[kmcopper]

@Flakebi Does environment.variables.SCUDO_OPTIONS = "" or environment.variables.SCUDO_OPTIONS = "ZeroContents=0"

Change the behavior or does it remain? I'm beginning to think made an additional way to scope the preload especially if we end up switching to graphene in the future as it's much less compatible with various applications than scudo is and I already needed to apply some compatibility fixes to the clang builds in nixpkgs.

For the record I am more pro graphene malloc then I am pro scudo malloc. It's more that I found the performance and hardening trade off to me more reasonable than the graphene malloc on it's own so I decided to go with that one since the small compatibility issues with man and some other stuff were easily resolved when I changed the LLVM stuff to default to non-aggressive faulting on certain kinds of program issues.

[Flakebi]

Setting these variables does not change anything.

env SCUDO_OPTIONS= rustc --crate-name ___ --print=file-names --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=s
ysroot --print=cfg /dev/null

-> Segfault

env SCUDO_OPTIONS=ZeroContents=0 rustc --crate-name ___ --print=file-names --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=s
ysroot --print=cfg /dev/null

-> Segfault

[kmcopper]

Have you tested at all with environment.memoryAllocator.provider = "graphene-hardened"; on rust as well?

[Flakebi]

With environment.memoryAllocator.provider = "graphene-hardened";

• rustc works,
• thunderbird works,
• firefox does not work,
• nix stopped working:

$ nixos-rebuild test
building Nix...
building the system configuration...
error: out of memory

[emilazy]

IIRC the Firefox segfault is because it passes something to scudo free that was allocated with jemalloc or vice versa; I don't think there's going to be an easier fix than implementing the replace-malloc API in a separate library.