systemd: add systemd-importd

[wucke13]

Motivation for this change

Adding systemd-importd to the build, so that machinectls import-.*
may actually do anything. Currently they fail with

Failed to transfer image: The name org.freedesktop.import1 was not provided by any .service files

as systemd-importd is not built.
This potentially will fix #70348

Testing

1. spawn a nixos vm based on this branch with gnupg and curl in the system environment and working sudo
2. run this (as normal user with working sudo) :

sudo gpg --no-default-keyring --keyring=/etc/systemd/import-pubring.gpg --fingerprint
tfile=$(mktemp -u /tmp/masterkey.nspawn.org.XXXXXXXXXXX)
curl https://nspawn.org/storage/masterkey.pgp -o "$tfile"
sudo gpg --no-default-keyring --keyring=/etc/systemd/import-pubring.gpg --import "$tfile"
machinectl pull-tar https://nspawn.org/storage/ubuntu/bionic/tar/image.tar.xz
machinectl start image
machinectl shell image

You will get a lot of selinux warning (which is perfectly fine if your vm has no se linux). In the end, the image should be verified by checksum and the signature proven by the imported key.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

Notify maintainers

cc @

[andir]

I'll try to have a look at this in the next days. I vaguely remember reasons for not doing what you did there with dbus. I can't remember right now. Hopefully that gets back to me within the next days.

[wucke13]

ping @andir

[edolstra]

What's the effect on closure size? It doesn't seem desirable to me to make systemd dependent on X.org (via libxkbdcommon).

[wucke13]

What's the effect on closure size?

How would I find out?

Regarding libxkbdcommon: I have no clue why I added it, but it seems to be not necessary. I removed the unneeded deps and rebased on master. I will test 7d20b67 it in a vm and report back in a few hours.

[edolstra]

How would I find out?

nix path-info -S (or -rS to see the paths in the closure).

[wucke13]

Ah, with the path to the nix store. I always tried nix path-info -S systemd. The closure size becomes 147.0M, so round about +35M.
While testing, I found out that there is an issue with direct calls to gpg and tar in the import code, though that should be fixed in the next push.

[edolstra]

Hm, I'd really prefer to avoid a hard dependency on X11 or gpg in the base system.

[wucke13]

That seems reasonable. Do we have any other systemd-* component which is seperated from the systemd base expression?
X11 is not in question, that was plain wrong by me (though gpg is, for the sake of signed images).

Would you prefer a split derivation (with a var which optionally enables systemd-importd) or a more or less completely separated expression? What is it that makes being dependent on gpg bad?
Btw: b7b78a0 is fully functional:

• machinectl import-tar works
• machinectl pull-tar works (Inlcuding check of pgp signature and checksum
• `machinectl start`` and so on works too.

So this branch may be considered a full fix to #70348 .

Btw 2:
Debian (via systemd ) and Archlinux (via pacman) for example depend on gpg for their base system. Interestingly, debian has systemd and systemd-container (so they split apart the nspawn stuff into a separate package), but gpg is a dependency of the systemd package. Obviously this might be related to their use of gpg in their chain of trust, but nonetheless having gpg in the base system doesn't seem to be terrible unpopular.

[andir]

Is it feasible to have those additional binaries in another output without increasing the closure of the out output?

That obviously wouldn't help against the build time dependencies. You could probably add the suggested switch and then we have another systemd package that carries all the extra bloat that most/many users do not care about.

[wucke13]

Is it feasible to have those additional binaries in another output without increasing the closure of the out output?

I'm afraid that it might become difficult marrying both parts together. I will look into it though.

another systemd package that carries all the extra bloat that most/many users do not care about.

You do realize that the current state is shipping not working bloat (at least machinectl) in the normal package? Of course having finer control over what to build and whatnot is good, but the user gets bloat+regressions right now (of which at least the regressions shall go IMO). Maybe a general decision should be made on how to deal with systemd, IMHO the current package ships a lot of things which may be considered bloat since they are not at all needed for an init.

TL;DR
systemd is bloated in its nature. That may be handled by splitting packages, ... . This PR addresses a regression in nixos' systemd package but does not address systemds bloat.

Edit: I started trying to split a systemd-container package from systemd one. The package compiles but doesn't work. The old monolithic approach is is available here: https://github.com/wucke13/nixpkgs/tree/systemd-monolithic, just in case that you change your mind regarding gnupg.

• What is a nice way of inheriting all the meson flags regarding folders and destinations ("-Drootlibdir=${placeholder "lib"}/lib" and alike) but not all the ones regarding features (is there something like if override )? Like, how can both be split into to distinct variables of which I can then only append to one in the systemd-container package?
• When building systemd monolithic (all in one package), I had to add "systemd-importd.service" and "dbus-org.freedesktop.import1.service" to upstreamSystemUnits (in nixos/modules/system/boot/systemd.nix). How can I achieve the same for something which is not part of the systemd package?
• What does meson do if I enable a feature and disable it afterwards (using the mesonFlags array)? My systemd-container derivation ends up including everything the normal systemd package provides (plus more) despite me deactivating all features but importd and machined?

Edit 2:
I have not found a way to reliably build only a a feature of systemd without the base install. It looks like most other distros (fedora) build systemd with all features and split away the results in multiple packages. Would that be a sufficient solution?

Conclusion

• Building only parts of systemd (like systemd-importd without all of the base systemd-* stuff) is beyond my capabilities.
  => The way to go is either a switch in the expression or split out paths
• Having eventually many switches in the systemd compile process will create a lot permutations for all the different combinations of all the different systemd packages available.
• Compiling systemd with all features and stripping away the different components in many outs is beyond my capabilities too

My wish is to have machinectl working. I fixed that. Bloated systemd is another issue, which has to be addressed, but not by me. I have neither the time nor the skill to come up with a decent solution for a decomposed systemd package (and it looks like some strategic decision should be made before somebody tries to actually implement anything).

[flokli]

Not squashing is fine, the first is good on its own too. "systemd: etc" is what's not really saying. It could be something like: ``` systemd: add importd This will fix the `machinectl import-.*` commands which currently fail with ... ```

[wucke13]

Ah I see! Done.

[flokli]

This doesn't seem to evaluate correctly.

[wucke13]

@GrahamcOfBorg build systemd

[flokli]

Can you remove the gnupg-minimal = callPackage ../tools/security/gnupg/minimal.nix { }; addition / removal in between the two commits?

[flokli]

I added a systemd-nspawn test which downloads an nspawn container via http from a local server, and ensures sha256sum verification and gpg signature verification work. PTAL.

[flokli]

@GrahamcOfBorg test systemd-nspawn

[flokli]

Ran tests manually on both aarch64-linux and x86_64-linux.

[flokli]

Added this test to the list in #72828 (it still needs to trickle from staging to master until it can be fixed there).

[kirelagin]

Hey, sorry for a stupid question, but how does this work? If I am reading meson.build correctly, want_importd requires zlib, but it is not in buildInputs.

[wucke13]

Hey, sorry for a stupid question, but how does this work? If I am reading meson.build correctly, want_importd requires zlib, but it is not in buildInputs.

I guess things which are not explicitly deactivated are enabled on demand!

[kirelagin]

The thing is in this PR you set -Dimportd=true, which activates this if and then, if zlib is not found, this if should trigger and configure should fail with an error. And HAVE_ZLIB is only ever set to 1 if zlib was actually found, of course.

[kirelagin]

Ok, it ends up being in the dependencies closure somehow, probably as a propagated dep of some other package, huh.

[adrianparvino]

Hi, this caused a regression on our machines.

58fb23f

exchange.> reloading the following units: dbus.service, firewall.service

6317f5b

exchange.> stopping the following units: dbus.service, dbus.socket, nscd.service
exchange.> Failed to create bus connection: Connection refused

Additional information: I'm using #46013 to remove polkit from our build which I believe is what resulted in the breaking combination.

[flokli]

I think this is the same issue as #74626 and not immediately caused by this PR.