RFE: per-service credentials system

[flokli]

Moved over from @poettering's post in NixOS/rfcs#59 (comment):

BTW, I thought about a creds concept in systemd the past weeks. It's a big topic, but this is what I'd probably propose in systemd:

1. Introduce a new setting per service setting LoadCredential=<name>:<path> for loading a credential off disk. The name is some user chosen ID, the path is some file system path to load the credential from.
2. Similar, add PassCredential=<newname>:<oldname> which takes a credential systemd itself received (think: pid1 in a container gets some creds from the container manager, or systemd --user from its parent).
3. Similar, add AskCredential=<name> for querying a credential from the user via the systemd-ask-password framework.
4. Then, add CredentialMode= for configuring how to pass credentials to the service. Can be a combination of options: file (pass as file in the fs), keyring (for kernel keyring), fd (for passing as fd via socket activation). We'd default to file and keyring. We should be graceful here, i.e. if keyring cannot work (because we are in a container where kernel keyring is blocked), use only the fs. We'd always attempt to pass all configured creds through all methods in parallel.
5. Then, let's add CredentialDirectory=, similar to RuntimeDirectory= and its friends, for defining a subdir of /run/credentials/ to pass the creds in in case file mode is used. There'd be $CREDENTIAL_DIRECTORY (again, similar to $RUNTIME_DIRECTORY and friends) we'd set for services to the resulting dir)
6. When passing via file system, we'd mount a ramfs (and not a tmpfs!) to /run/credentials// and place the creds there. ramfs has the benefit of never being swapped out. needs graceful fallback logic to using the dir without mount however, to cover for containers. we'd populate the fs atomically, and keep it read-only afterwards.
7. We'd never pass creds in env vars, they suck since they propagate down the process tree, even to suid programs.

everything passed will be immutable and only accessible to the user of the service, nothing else. there'd be no API for dynamically requesting more creds.

[flokli]

Questions:

1. If passing secrets via file, why /run/credentials/…/, and not just /run/credentials? I'd assume this will always be in a private mount namespace anyways, so we don't really need another directory level here (similar to how PrivateTmp also is just mounted at /tmp)
2. How would integrations with other credential providers look like? I assume they'd need to go via the systemd-ask-password framework? Is that right? Has anyone seen a vault provider? How does it work with TPMs, decrypting various secrets, or cloud providers with metadata servers providing means of accessing secrets?
3. Not sure if I'm alone here, but IMHO, theres two different concerns here when it comes to describing secrets:

• Distros/Services: might ship a unit file describing they need a certain secret (myservice-token), possibly configure ExecStart to pass /run/credentials/myservice-token via some CLI parameter, and set CredentialMode=file
• Admins: might want to describe how myservice-token is actually populated on that system (persisted somewhere on disk, retrieved from a secret store, …)

Regarding 3., I'd personally stronger separate the second concern from the unit config, maybe even for the simple LoadCredential part. WDYT?

[poettering]

1. If passing secrets via file, why /run/credentials/…/, and not just /run/credentials? I'd assume this will always be in a private mount namespace anyways, so we don't really need another directory level here (similar to how PrivateTmp also is just mounted at /tmp)

I don't think we should rely on mount namespaces for this. Most of our code is will gracefully degrade if they aren't available (because compiled out in kernel or turned off in containers via seccomp), and I think we should do the same here too. Also, it's kinda nice when exploring this as an admin from the host side, since then you have a different dir there for each service and can easily see who gets what.

We have to allocate a per-service ramfs anyway I think because otherwise we cannot populate it "atomically" (meaning: make it show up in the host tree fully populated at once without a phase where it is mounted on the host but not populated), and release it atomically. We need to mount that somewhere hence we need a per-service mount point anyway.

I think we also should make sure that what the service sees where things are and what the admin sees from host side where things are should not deviate needlessly...

2. How would integrations with other credential providers look like? I assume they'd need to go via the systemd-ask-password framework? Is that right? Has anyone seen a vault provider? How does it work with TPMs, decrypting various secrets, or cloud providers with metadata servers providing means of accessing secrets?

I explicitly don't want to provide any fancy API for credential plugins. That said, I see two simple ways to hook this up:

a. Have a preparatory service that puts the keys somewhere, and then pick them up in the main service from there via LoadCredential=

b. If you specify an absolute path to an AF_UNIX file system socket via LoadCredential= we should connect() to it, and read from it until EOF and use that as credential. It's similar how StandardInput= and friends similarly support connect()ing to AF_UNIX sockets.

systemd-ask-password really should be about interactive, string-based, NUL-terminated creds only. i.e. passwords, passphrases, recovery keys, i.e. stuff humans type in. For anything else, i.e binary keys, and so on hook it up via a file you put somewhere or the AF_UNIX stuff, systemd-ask-password is a bad fit.

Not sure if I'm alone here, but IMHO, theres two different concerns here when it comes to describing secrets.

I see what you mean. Not sure it's a big problem though. We could for example say that if you do:

PassCredential=foo
LoadCredential=foo:/etc/credentials/foo
AskCredential=foo

i.e. declare that the same cred called "foo" is to be acquired by three different methods, that this means we'd first try the PassCredential= one, then the LoadCredential= one and finally the AskCredential= one (always in the same order, preferring static over dynamic data). That means vendors can define some basic behaviour, but the admin can override it (via drop-ins) or extend it, and rely on the order how things re tried.

[flokli]

I explicitly don't want to provide any fancy API for credential plugins. That said, I see two simple ways to hook this up:

a. Have a preparatory service that puts the keys somewhere, and then pick them up in the main service from there via LoadCredential=

b. If you specify an absolute path to an AF_UNIX file system socket via LoadCredential= we should connect() to it, and read from it until EOF and use that as credential. It's similar how StandardInput= and friends similarly support connect()ing to AF_UNIX sockets.

I agree systemd-ask-password might be the wrong framework for this. "Secrets" could be more complicated than a single ascii string - (looking at you, services only allowing secrets to be inline inside a config file, and not read from a file)

However, I think credential providers like vault, KMS etc. shouldn't be ignored entirely while designing such an architecture. In cloud environments, they can very likely be the only thing providing secrets.

Let's say systemd could also support reading files from a "global" secrets path, where credential providers could mount a fuse filesystem, or create socket file(s). [1]

That could be as simple as a "if the path in LoadCredential is relative, look into /run/system-credentials/… first).

Secret providers might then just mount a fuse filesystem (accessible to root only), and query their backend on access, or administrators mount their encrypted volume containing secrets to that location during bootup, etc. Distributions/Packagers could even provide some sane defaults in unit files for simple services only requiring a single token to work.

---

What worries me about the current approach is secret rotation and service reloads - currently, there's no nice way to describe a secret being expired, or a way of modelling the dependency between secret (files) and services.

By now designing such a credentials system, such a thing could finally be provided.

This could be as "simple" as doing some inotify on the secret files, and triggering a reload on change. Other options could be making the socket protocol a bit more complex, to allow specifying expiry dates. What about a single endpoint, and just using HTTP (HEAD, and the expiry headers)?

Currently users use yet another external tool like confd watching files and restarting services, together with ugly bash scripts waiting for secrets to be provided, delaying the startup of the existing service.

[flokli]

cc-ing @cedws due to https://cedwards.xyz/protecting-systemd-service-secrets/

[poettering]

My initial implementation of this you find in #16568. Please have a look. The PR is not ready to merge (has no tests), but it does include documentation that should explain what this does.

Let's continue discussion there.