-
-
Notifications
You must be signed in to change notification settings - Fork 14.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RFC: Harden(ed) NixOS #7220
Comments
From previous discussions, some of the flags (with negligible performance disadvantages) should be easily accepted to be in the defaults, perhaps even without an explicit option to disable them. Some regular distros also do have some of these for all packages IIRC. Side note: I've seen that generating non-deterministic binaries is also used to improve security. I've seen that elsewhere, but wiki also mentions randomization of control flow https://en.wikipedia.org/wiki/Binary_hardening. (I'm no security guy, but I expect such techniques aren't available with unmodified toolchain, and non-determinism is a bit difficult with nix anyway.) |
👍 👍 👍 @thoughtpolice in addition to all your excellent points, I've also wondered about reliable "auditability" of the system. The recent work on #7092 helps figure out what @vcunat I think 99% of the security benefits you'd get from generating nondeterministic binaries are attained by ASLR on deterministic binaries. That is, unless your threat model involves an attacker using tooling that assumes binaries are laid out in a specific manner (and doesn't otherwise interpret ELF or your object format), what matters is runtime memory layout, not static binary layout. That's not to say that other aspects of binary hardening aren't applicable. If we have no source for e.g., Spotify, there might be instances in which the binary can be modified and made more secure, but that seems like a much less common case for a distro that is almost 100% source-based. |
FWIW, I am in general agreement with @thoughtpolice's suggestions. I would add that it should be a no-brainer to allow the implementation of the security enhancements that have already been done by any of the mainstream, general-purpose distros that rely on a large community of contributors (e.g. Debian, Ubuntu, Arch Linux, non-hardened Gentoo, ...). The last time I looked, Ubuntu in particular implemented a significant number of these enhancements. These have already been battle-tested and shouldn't have a significant impact in maintaining most packages. Also, I think it would be good to prioritize the enhancements @arno01 has been working on in #7212 and related PRs (although, I guess we should try applying them to most of NixOS instead of on a package-by-package basis, assuming that's what we want). This is obviously a priority to him and it can be frustrating and discouraging if we don't accept his contributions (or at least, help / guide him) for a long time. The same applies to any other already-implemented contributions / contributors. |
I assume this has already been thought of, but: if there will be that kind of penalty, we need to allow per-package opt-out. We package numerical computing libraries for which that kind of penalty is unacceptable, whatever the security implications. The same thing goes for PIE. I'm not asking for an unhardened system, just some variable for |
Some thoughts: Yes, I fully envision this will be controllable via a flag for The granularity of these options is up for debate; for example RELRO/BIND_NOW protection can probably always be enforced. Disabling PIE and stack protection will be the biggest wins for most applications. Note that PIE is very efficient on x86_64 in comparison to i386; in fact, stack protection will be the biggest loss on 64bit, and PIE should have little effect. It's only 32bit where they both hurt. Numerical notes: note that PIE is only going to hurt applications - this is a very important distinction for numerical libraries, because PIE is basically just a dynamically shared object disguised as ELF. Meaning, if you already had a numerical library on 32bit machines that offered a However, it's not clear to me how high the demand for 32bit numerical libraries are anyway in this scenario anyway - almost anyone seriously using them is going to opt for a 64bit instruction set which will offer wider register sets and more features. Finally, things like dynamically linked copies of I think where this is really going to be a bigger impact is things like databases, which aren't hand written assembly, but a very large performance sensitive code base. I suppose for this we'll have to extend the Hardened GCC patches if we decide to go that route, because then Thanks for the feedback, let me know if you think this sounds all wrong. :) |
Actually, is it likely that someone would want to run 32-bit stuff on a security-sensitive machine? (Numerical-intensive stuff also doesn't sound like going well with the register limitations inherent to standard i686.) |
I think we need something like smart-flexi-hardening. It should be relatively easy to enable hardening global-wise or per-package-wise including the levels of hardening, for example: |
On Tuesday, April 07, 2015 14:10:20 Austin Seipp wrote:
|
Binary hardening: this doesn't sound desirable if the performance impact is that bad. Could be an option for network-facing services. Service hardening: 👍 Kernel enhancements: I'm reluctant to depend too much on non-standard kernel patches, since they can create a situation where we can't update our kernel to the latest upstream version because the patches no longer apply. E.g. the discussion at #6858 seems to imply that grsecurity does not support 3.18 kernels, meaning we can't use that (LTS) branch. Enabling AppArmor by default: Sounds good. |
The problem is that this does not scale nicely for users - they really want binary packages, and every single one of these different flags implies a complete rebuild of pretty much the entire transitive closure of your system environment, because it will imply hash changes all the way through the graph your system closure. Selecting appropriate defaults for users is extremely important so they get A) good improvements with B) minimal friction and pain. The current default is essentially equivalent to your L0, which I find unoptimal. :) In general I'm OK with leaving judgement to maintainers of packages/services about what hardening steps are necessary, but I'm much more hesitant about leaving that decision up to users, because the trade offs are much less clear than they may seem.
The impact is really only PIE/stack protection. The others are very cheap and increase security with few - if any - complications. Again, I think the correct thing to do is to have services that absolutely need the extra performance opt-out of these two features. Just looking at my system closure for example, there are a lot of applications for which this would only increase security with few downsides - systemd, coreutils, tcpdump/libpcap, etc etc. In fact, I'd say network services are the ones that want to opt out, not in, precisely because they're performance sensitive and tend to have a large amount of security scrutiny anyway. I also like to keep in mind almost every other distribution enforces many of these enhancements anyway. This performance impact may seem bad, but it's a loss that a very high majority of distribution maintainers see as appropriate. In fact, PIE is the only truly controversial one. An alternative would be to instead default on PIE is something special on its own, and perhaps worth not enabling by default on 32bit (which is a bit weird and inconsistent), but on 64bit, should absolutely be enforced IMO as it can significantly complicate code-reuse exploits and RIP-relative addressing is cheap in comparison to stealing a base register.
Just to be clear: I do not ever plan on making NixOS require things like grsecurity* - which, on its own, would be a bigger discussion than this IMO - which is an extensive and massive patchset. (Something like a MAC randomization patch is, OTOH, about 20 lines of code), and it also slightly complicates the LTS story as I indicated in that other thread. The support we have now would need extensive polishing in any case.
|
Anyone have thoughts on my ongoing audit point above? Being secure is fine and dandy, but it's also nice to have reasonable confidence that your security measures are actually in effect 😄 |
@thoughtpolice 👍 We all know that buffer overflows (not only) are common in the C and C++ languages and every qualified security expert knows that each and every protection mechanism -- matters. Having just one single weak point can lead to a system compromise. |
Regarding AppArmor profiles: it's probably worth looking into re-using profiles provded by the apparmor-profiles package. In a lot of cases, it's simply a matter of patching the binary name. |
RBAC can be implemented in AppArmor via apparmor-pam, which would benefit everybody not just grsec users. AppArmor also has what looks to be a fairly comprehensive learning mode (all of that is currently broken on NixOS, though). All-in-all, it seems most cost-effective to focus on AppArmor for RBAC and automatic learning and rely on grsec mostly for PaX et al. |
As an update, I have a branch (not yet published) which:
Here's an example of an "Improved" grsecurity description using NixOps (NB: not representative of final overhaul):
Some points:
and still get proper service, udev rules, etc. I still need to do some extra stuff. In particular this current interface doesn't offer any help in configuring kernel parameters or setting configurations, so it's definitely not merge ready. Also, I have only just begun adding AppArmor and systemd policies, so I'll probably bundle/test more of them, too. Please let me know what you think. I'll open a PR later this week. |
I should mention I would like to get all purely-NixOS level work out of the way first, because while it may imply some changes to a few things, it does not imply an ungodly rebuild of all of |
I was also thinking that we could probably also pull off W^X at the filesystem level. |
Speaking on security "out-of-a-box", right now I am building a standard (non-hardened) LFS where I notice that systemd 219 by default enables all common protections like Canary (SSP with the buffer-size=4), PIE, Full RELRO and FORTIFY_SOURCE:
[ http://www.linuxfromscratch.org/lfs/view/stable-systemd/chapter06/systemd.html ] |
@vcunat yeah, I have a commit locally that updates it. I'll push it up at some point. |
grsecurity still holds a reference to it, but I prefer it to fail than to use a version that is most likely not secure anymore.
This is mostly @thoughtpolice's work, but I cleaned it up a bit.
Also, use systemd timers. Most of the work is by @thoughtpolice but I changed enough of it to warrant changing commit author.
For anyone following this, see #12895 |
Bumping milestone, but really looking forward to this work being merged. |
Just noticed that in packages like nginx, hardening and compiler being GCC is checked. Is there another flag to check for GNU LD? |
Not sure if this issue is still alive. Still, this is probably relevant: #14645 (The service interface should abide by the principle of least privilege). |
Just looking at this the first time. But if AppArmor is used by default, does this make it more difficult to use SELinux with NixOS in the future? Also has anybody considered Capsicum capabilities? See: |
I'm removing the milestone here as the description of the PR is fairly outdated. Is there anyone willing to take this meta-issue forward? It's fairly done and spread out in issues like #14645 and #12895. It's important to have an overview, but it doesn't help if it's outdated information since a year, that will mislead bycomers. If there's noone willing to clean this up, I'll just close it in a few weeks. |
Let's close. Better re-formulate in a new ticket, if you (still) prefer to have this kind of general one. |
Please link the new issue(s) here... |
See #14392 for another useful kernel hardening feature, the Yama LSM, which I'd like to get in before 16.09. |
Hello. I have updated to NixOS 16.09 and I have difficulty using GCC in it. GCC inserts stack protector checks into my program. I can not disable it (there is no option -fno-stack-protector-strong provided by GCC). I don't mind if packages will be built with hardening by default. But is it a bit inconvenient that the default gcc command enables hardening with no clear way to disable it. |
@joachifm Thank you. I'll try to work around it this way. |
This should be re-opened! |
(Spurred by some conversation in #7212, so I decided to write down my thoughts.)
Currently, NixOS and Nixpkgs have a lot of areas in which the security story can improve. While I think we're well positioned to take care of many of these, it's going to take some work, and others we'll need to reach consensus on. (The Good News is that many other people have gone down this road, so we have some good things to learn from)
As a rough starting point, I have identified roughly 5 major areas to look for future enhancements, with my personal suggestions on what we might want to do.
Binary hardening
As inspired by #7212, currently, NixOS doesn't default to enabling any hardening capabilities in GCC. There are an array of things we can do here to mitigate exploitability, that we can either enforce as a GCC spec file, or as part of
NIX_CFLAGS
/gcc-wrapper
. Here are a few of them:-D_FORTIFY_SOURCE=2
to improve protections for various built in string/buffer functions.-pie -fPIC
(PIEs are essentially just shared-objects-in-disguise much like transformers are robots-in-disguise).-fno-strict-overflow
protects against the compiler optimizing away arithmetic overflow tests, which can happen when it gets particularly aggressive about undefined behavior.-fstack-protector*
.-fstack-protector
protects functions with calls to things likealloca
or ones that allocate more than 8 bytes of stack data. This buffer size can be controlled via--ssp-buffer-size
, for example--ssp-buffer-size=1
triggers on any stack allocations..-fstack-protector-strong
, which implies more coverage than-fstack-protector
by default.-fstack-protector-all
which does this for all functions. I am pretty sure this implies--ssp-buffer-size=1
, but I'm not 100%.-z relro
to mark dynamic relocations by the dynamic linker as read-only afterld.so
resolves them. However, we also need-z now
in order to force non-lazy resolution of relocations so the linker resolves them all at startup: otherwise they are performed on demand, and not marked as read-only until resolved (by which point an attacker could have used an arbitrary write to overwrite the relocation).Suggestion: default all expressions to
-fstack-protector-all --ssp-buffer-size=1 -D_FORTIFY_SOURCE=2 -pie -fPIC -z relro -z now -fno-strict-overflow
as part ofcc/ld
. In other words, the whole gamut. For example,mkDerivation
could supportdisableHardeningOptions = true;
to disable options in specific expressions.Implications: actually, fairly large in theory. These all in conjunction can have a significant penalty to things like startup time (due to resolving dynamic symbols up front), or execution speed. In particular,
-fstack-protect-all
is going to be costly (I wouldn't say beyond utility, but probably a good 20% speed loss at least). Also, PIE is going to really, really hurt performance oni686-linux
, because PIE steals a register from the pathetic 32bit register set. Just PIE alone will have a significant impact on 32bit users, and-fstack-protect-all
will add insult to injury.Really, only benchmarking these things will tell us. I'd estimate the hit for these doesn't matter for a significant amount of software (common things like
systemd
,coreutils
, anything that issetuid
, etc). But a lot of things that are sort of nebulous middle grounds can be discussed on a maintainer-by-maintainer basis, I suppose (e.g. networking services are both performance and security critical, so we may want something better here).Binary determinism
This is issue #2281. I've had this on my queue to finish integrating for a while now; with my new machine this can hopefully be a reality soon enough... Unfortunately as we didn't make the cut for GSoC 2015, there won't be any sponsored work on this. There is still the issue of GCC PGO determinism too, which does not have a clear consensus, but the large majority of the work is elsewhere.
Suggestion: I get off my ass and merge this.
Service hardening
Currently, very few of our NixOS services try to take advantage of any security or isolation features that can be offered e.g. by
systemd
. Basic examples that are probably worthy of spreading around the tree:PrivateTmp
, since a lot of services are probably fine with their own/tmp
mount. (unless they do something weird like use it for cross service IPC, which is now handled by/run
).DevicePolicy
, which can restrict access to/dev
(lots of them could get by with e.g."closed"
)InaccessibleDirectories
,ReadOnlyDirectories
, andReadWriteDirectories
(for example, some would never need access to/home
, while others liketarsnap
could always restrict themselves toReadOnlyDirectories=/
with a specificReadWriteDirectories
for the cache).NoNewPrivileges
, which automatically setsprctl(PR_SET_NO_NEW_PRIVS)
for daemons. Again, useful for services liketarsnap
or logging services (but care is needed if e.g. things invokesetuid
programs likeping
).Suggestion: We begin enhancing services with these and encouraging maintainers of modules to do the same. Honestly this can probably be done pretty easily and fairly incrementally by maintainers or any interested newcomers.
We should pay attention to upstream systemd units or units from other distributions here too, since they'll have figured out some of this, too.
Kernel enhancements
Kernel security enhancements mostly come from one thing and one thing only (IMO): grsecurity. The good news is that grsecurity support mostly works with my module, and of course it's possible to go out of band with your own custom
linuxPackages
.One thing is that we don't currently offer prebuilt grsecurity packages. Hydra actually builds them, but there's no way for the module to automagically select the right one. This should be fixed. Futhermore, the binary builds and module need some clean up (e.g.
RANDSTACK
is actually completely useless for pre-built binaries since the random offset can be known by any attacker, soRANDSTACK
should always imply a local kernel build.)Also, my grsecurity module could use a bit of work. Unfortunately it's split up in several places due to needing build/module support, but I do think this could be cleaned up/refactored a bit.
There are other things to consider, too. For example, there are kernel patches floating around to do various other things; it would be really nice, for example, if we could have a patch to randomize the MAC address assigned by the kernel - this would be good for my laptop and could be controlled by a boot switch for the kernel.
Suggestion: Well, it mostly works I guess.
MAC/RBAC
NixOS currently doesn't offer any form of policy enforcement in the place of MAC systems. There are a lot to choose from, but it basically comes down to AppArmor vs SELinux. Right now there's nobody really supporting either, so in lieu of this, the support that does exist is geared towards AppArmor. AppArmor isn't as expressive as SELinux, but it's a hell of a lot simpler and the policies are far easier to maintain. I think this is pretty important to get people to write policies.
The good news is the actual infrastructure is there: apparmor is packaged and works. There are even NixOS modules for it, but nothing really uses it. This is pretty easy though: we can begin clipping apparmor policies from upstream packages and places like Ubuntu.
Suggestion: We should just start writing policies, and enforce AppArmor by default on NixOS. This is the best way to ensure people keep using it, IMO.
Sidenote:
gradm
The story is better for grsecurity users: you get
gradm
which is a totally badass RBAC system. The unfortunate news is that last time I tried it,gradm
was buggy on my 3.4 kernel, and I didn't take the time to find a way to reconcile things like wanting NixOS modules to declare RBAC policies with the self-learning mode. In practice it may just be best to instead have services that activate/deactivate the learning mode viasystemd
, or just do nothing at all and expect users to enforce it themselves.Small-form categorized TODO list
gcc
to support transparent hardening.NIX_CFLAGS
, might work OK.-D_FORTIFY_SOURCE=2
as far as I can tell, but that one can be added toNIX_CFLAGS
most likely orgcc-wrapper
.-D_FORTIFY_SOURCE=2
is, IIRC, only available at-O2
or above, so that can be annoying if we don't deal with it ingcc-wrapper
. Or we could just patch the stupid warnings it emits out of GCC, too.-z relro
&-z now
), as well as-fno-strict-overflow
.mkDerivation
to allow opt-out.system_tarball_pc
remains deterministic.grsecurity
module serve prebuilt images.CC: @wizeman @domenkozar @peti @edolstra
The text was updated successfully, but these errors were encountered: