Thoughts on how to improve Linux kernel maintenance a bit #140281

Ma27 · 2021-10-02T14:20:18Z

Motivation

I'm one of the folks who are currently testing/merging/backporting Linux kernel updates. All in all, this is quite a smooth process thanks to @TredwellGit who tirelessly files update PRs in the first place and an existing updater-script.

However there are a few minor issues I have that I'd like to discuss here before submitting any code.

Topics

Hardened Kernel

We have all active Kernel releases packaged and - if available - also linux-hardened which provides a few additional security properties. However these releases lag behind the upstream Kernel usually, on a few occasions we had to wait for about a week[1][2].

However, Kernel updates are quite relevant usually, either because of important bugfixes or because of CVEs to be fixed. Because of that, we decided to ignore outdated hardened patches at some point[3] and right now the hardened kernel is broken rather regularly[4]. Another issue is what's supposed to happen if a kernel minor-release gets EOL and no hardened patches are available for the next higher minor release (that's also _latest)[5].

IMHO neither the "old" approach (awaiting hardened patches and thus delaying kernel updates for everyone) nor the current one (ignoring build failures) is optimal. Current ideas I have are:

Drop all hardened kernels. With Nix it isn't very hard to re-add packages via an overlay.
Use older versions of a Kernel for hardened variants (i.e. let's say we are at Linux 5.14.5 and want to upgrade to 5.14.6, but no hardened patch is available yet. Now we upgrade to 5.14.6, but use continue to use 5.14.5 for the hardened kernel).

Maintainer team

This is a rather trivial one 😅

I noticed that I get pinged rather inconsistently on Kernel PRs which is totally understandable, I also wouldn't want to keep track of everyone I have to ping for each PR I open, especially because this is automatable.

Currently we seem to have a single person listed as maintainer for Kernels:

$ nix-instantiate -A linuxPackages.kernel.meta.maintainers --eval --strict
[ { email = "aseipp@pobox.com"; github = "thoughtpolice"; githubId = 3416; name = "Austin Seipp"; } ]

My suggestion is rather simple - we should for a maintainer team with both a team in the NixOS GH org and an entry in maintainers/team-list.nix. I'd nominate

@NeQuissimus
@TredwellGit
@mweinelt
@alyssais
myself - if everyone's OK with that

and make sure the members are requested for a review on each Kernel bump.

[1] #119543
[2] #125469
[3] #134692
[4] https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.linuxPackages_hardened.kernel.x86_64-linux/all
[5] #133587 (review)

The text was updated successfully, but these errors were encountered:

dasJ · 2021-10-03T13:01:57Z

Since we are using the hardened kernels on our infrastructure, I'm a big fan of changing the way it currently works. I don't know how many people other than us use the hardened kernels, but I don't think it's fesible for everybody to overlay in the hardened kernel and keep it up to date, especially for new users who are switching from other Distros and want to keep their kernels. For this reason, I tend to prefer your second idea.

I did this for OpenSSH in #115012 as well and I think most of the user base who are using the stock ssh benefit from this change with the others having to accept the less maintained packages.

happysalada · 2021-10-04T03:32:40Z

Just one more detail. For zfs, there is the latestCompatibleLinuxPackages = linuxPackages_5_14; to track the latest kernel version compatible with zfs. It would be really nice to have something like latestCompatibleHardened.

I don't use hardened yet, but I am interested in it, the second choice would be amazing.

Ma27 · 2021-10-04T07:29:54Z

Just one more detail. For zfs, there is the latestCompatibleLinuxPackages = linuxPackages_5_14; to track the latest kernel version compatible with zfs. It would be really nice to have something like latestCompatibleHardened.

Yeah, I'm aware of that, but I don't think this is useful here. linuxPackages_latest_hardened isn't a good idea because this can lead to "silent" downgrades as mentioned in the OP.

Also, this doesn't help when trying to update the "latest compatible version" in question, if no hardened patch is available for a new patch-release we can either delay the upgrade or ignore the hardened kernel.

NeQuissimus · 2021-10-04T12:27:22Z

I have not had as much time to do this maintenance as I used to. I would go with the "keep a specific version for the hardened kernels"-approach.

wamserma · 2021-10-05T07:02:47Z

To chime in, I support decoupling hardened kernel bumps from vanilla kernel bumps. Adopting the the hardening patches takes time and testing, so hardened kernels will often be delayed by a few days.
For minor version bumps the upgrade script should attempt to apply the patchset from the previous version, if no matching patchset is available. If it applies cleanly and compiles, fine. If not: simply delay the Kernel update.

Another thing that we should take care of before 21.11 are the other Kernel flavours, e.g. Xanmod. While the version in unstable is pretty up-to-date, 21.05 is currently stuck with 5.12.13-xanmod.
Maybe the update-script can be extended to ensure that backports are taken care of and we should provide linuxPackages_xanmod_latest and linuxPackages_xanmod_lts.
ping @fortuneteller2k @lovesegfault for their view on this

lovesegfault · 2021-10-05T07:05:29Z

If someone who uses -stable wants to take up the maintenance of xanmod backports I'll happily help merge PRs. I live on unstable-small in all my systems, so I don't have time or a host to test the backports properly.

wamserma · 2021-10-05T07:15:17Z

I've been switching around Kernel versions on -stable a lot lately (because of issues with nouveau, not NixOS specific) and it always worked like a charm.
Given that Xanmod is targeted towards desktop use, where a rollback is just a reboot away I think we are fine with a "no guarantees" approach. The user base on stable seems to be rather small, given that no one noticed the outdated kernel for quite some time.

moni-dz · 2021-10-05T07:21:53Z

Maybe the update-script can be extended to ensure that backports are taken care of and we should provide linuxPackages_xanmod_latest and linuxPackages_xanmod_lts.

ping @fortuneteller2k @lovesegfault for their view on this

I'm fine with a stable version of xanmod if there's enough demand of it, personally I use the latest available, as I don't use ZFS and the like.

Artturin · 2021-10-05T07:26:27Z

I use xanmod latest and zfs

alyssais · 2021-10-05T09:06:29Z

Hardened plan sounds fine to me. I'm happy to be in the maintainer team so I get pinged if I'm around, as long as everybody has the expectation going in that I don't often find the time to do kernel patch releases.

Ma27 · 2021-10-05T09:48:43Z

Splendid! I'll file PRs for both the hardening topic and the maintainer team during one fo the next days :)

TredwellGit · 2021-10-05T21:31:34Z

Can XanMod be added to https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/linux/kernel/update.sh?

lunik1 · 2021-10-14T09:09:42Z

I (evidently 😉) use the hardened kernel so would prefer the second approach. Do you believe this will be a significant maintenance burden?

Ma27 · 2021-10-14T09:21:36Z

there's already #141100 doing exactly that :)

Now there are a few more folks who should get pinged on kernel changes: $ nix-instantiate -E 'with import ./. {}; (map (x: x.github) linux.meta.maintainers)' --eval --strict [ "TredwellGit" "mweinelt" "ma27" "nequissimus" "alyssais" "thoughtpolice" ] Refs NixOS#140281

Until now we merged kernel updates even if no hardened versions were available yet. On one hand we don't want to delay patch-level updates, on the other hand users of hardened kernels have frequent breakage now[1]. This change aims to provide a solution this issue: * The hardened patchset now references the kernel version it's released for (including a sha256 hash for the fixed-output path of the source tarball). * The `hardenedKernelFor`-function doesn't just append hardened patches now, but also overrides version & src to match the kernel version the patch was built & tested for. Refs NixOS#140281 [1] https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.linuxPackages_hardened.kernel.x86_64-linux/all

linux: changes for #140281

Now there are a few more folks who should get pinged on kernel changes: $ nix-instantiate -E 'with import ./. {}; (map (x: x.github) linux.meta.maintainers)' --eval --strict [ "TredwellGit" "mweinelt" "ma27" "nequissimus" "alyssais" "thoughtpolice" ] Refs NixOS#140281 (cherry picked from commit 65930ca)

Until now we merged kernel updates even if no hardened versions were available yet. On one hand we don't want to delay patch-level updates, on the other hand users of hardened kernels have frequent breakage now[1]. This change aims to provide a solution this issue: * The hardened patchset now references the kernel version it's released for (including a sha256 hash for the fixed-output path of the source tarball). * The `hardenedKernelFor`-function doesn't just append hardened patches now, but also overrides version & src to match the kernel version the patch was built & tested for. Refs NixOS#140281 [1] https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.linuxPackages_hardened.kernel.x86_64-linux/all (cherry picked from commit bb5aa01)

Now there are a few more folks who should get pinged on kernel changes: $ nix-instantiate -E 'with import ./. {}; (map (x: x.github) linux.meta.maintainers)' --eval --strict [ "TredwellGit" "mweinelt" "ma27" "nequissimus" "alyssais" "thoughtpolice" ] Refs NixOS#140281 (cherry picked from commit 65930ca)

Until now we merged kernel updates even if no hardened versions were available yet. On one hand we don't want to delay patch-level updates, on the other hand users of hardened kernels have frequent breakage now[1]. This change aims to provide a solution this issue: * The hardened patchset now references the kernel version it's released for (including a sha256 hash for the fixed-output path of the source tarball). * The `hardenedKernelFor`-function doesn't just append hardened patches now, but also overrides version & src to match the kernel version the patch was built & tested for. Refs NixOS#140281 [1] https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.linuxPackages_hardened.kernel.x86_64-linux/all (cherry picked from commit bb5aa01)

[21.05] linux: changes for #140281

Ma27 added 6.topic: kernel The Linux kernel 6.topic: policy discussion labels Oct 2, 2021

Ma27 mentioned this issue Oct 3, 2021

21.11 Feature Freeze #140168

Closed

This was referenced Oct 4, 2021

Kernels 2021-09-30 #140110

Merged

Unable to build 5.10.64-hardened kernel on 21.05 #138203

Closed

8573 mentioned this issue Oct 9, 2021

linux_hardened: make hardened patchset less picky #139038

Closed

12 tasks

Ma27 mentioned this issue Oct 9, 2021

linux: changes for #140281 #141100

Merged

12 tasks

lunik1 mentioned this issue Oct 14, 2021

linuxPackages_hardened.kernel fails to build #141097

Closed

Ma27 closed this as completed in #141100 Oct 20, 2021

Ma27 added a commit that referenced this issue Oct 20, 2021

Merge pull request #141100 from Ma27/kernel-maintenance

dd7d2ee

linux: changes for #140281

Ma27 added a commit that referenced this issue Oct 29, 2021

Merge pull request #143466 from Ma27/backport-linux-changes

e5faf9d

[21.05] linux: changes for #140281

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Thoughts on how to improve Linux kernel maintenance a bit #140281

Thoughts on how to improve Linux kernel maintenance a bit #140281

Ma27 commented Oct 2, 2021

dasJ commented Oct 3, 2021 •

edited by ajs124

Loading

happysalada commented Oct 4, 2021

Ma27 commented Oct 4, 2021

NeQuissimus commented Oct 4, 2021

wamserma commented Oct 5, 2021

lovesegfault commented Oct 5, 2021

wamserma commented Oct 5, 2021

moni-dz commented Oct 5, 2021

Artturin commented Oct 5, 2021

alyssais commented Oct 5, 2021 via email

Ma27 commented Oct 5, 2021

TredwellGit commented Oct 5, 2021

lunik1 commented Oct 14, 2021

Ma27 commented Oct 14, 2021

Thoughts on how to improve Linux kernel maintenance a bit #140281

Thoughts on how to improve Linux kernel maintenance a bit #140281

Comments

Ma27 commented Oct 2, 2021

Motivation

Topics

Hardened Kernel

Maintainer team

dasJ commented Oct 3, 2021 • edited by ajs124 Loading

happysalada commented Oct 4, 2021

Ma27 commented Oct 4, 2021

NeQuissimus commented Oct 4, 2021

wamserma commented Oct 5, 2021

lovesegfault commented Oct 5, 2021

wamserma commented Oct 5, 2021

moni-dz commented Oct 5, 2021

Artturin commented Oct 5, 2021

alyssais commented Oct 5, 2021 via email

Ma27 commented Oct 5, 2021

TredwellGit commented Oct 5, 2021

lunik1 commented Oct 14, 2021

Ma27 commented Oct 14, 2021

dasJ commented Oct 3, 2021 •

edited by ajs124

Loading