nixos/espanso: fix wayland problems due to missing capabilities

[pitkling]

Description of changes

This PR fixes #249364 (which I assume is actually not only Wayland-specific). When enabled via programs.espanso.capdacoverride.enable = true, it injects an overlay that overrides pkgs.espanso-wayland with a package that has the DAC_OVERRIDE capability.

With this in place, espanso can be used under Wayland (both in NixOS and Home Manager) via the standard way:

services.espanso = {
  enable = true;
  package = pkgs.espanso-wayland;
};

There is also a programs.espanso.capdacoverride.package option that allows to set which espanso-wayland is used as a base for the DAC-OVERRIDE-enabled package version. For example, if you pulled in espanso-wayland from nixpkgs-unstable under pkgs.unstable.espanso-wayland), you can set:

programs.espanso.capdacoverride.package = pkgs.unstable.espanso-wayland

If this module is added, one should probably add a note to the services.espanso module mentioning that this must be enabled for Wayland (or it won't work). Alternatively, one could detect automatically whether the user wants to use an espanso-wayland package and activate this module automatically.

Detailed Description

Under Wayland, Espanso requires the DAC_OVERRIDE capability (see the documentation). NixOS does not support capabilities in the Nix store but instead provides a framework to create wrapper binaries with suitable capabilities. While the wrapper does some work to maintain those capabilities across forks, this is not enough in the case of Espanso, which drops the extra capabilities early on and relies on file capabilities to regain those capabilities when forking a worker process (via /proc/self/exe). Unfortunately, the symlink under /proc/self/exe does not point to the capability-enabled wrapper but to the original binary in the nix store.

Thus, this PR wraps the original espanso-wayland package definition and injects a wrapper library that intercepts the readlink-call and – if it looks up /proc/self/exe – returns the capability-enabled wrapper. Since espanso-wayland is broken without those capabilities, the wrapped espanso-wayland package is then overlayed (unless explicitly disabled via programs.espanso.capdacoverride.enable = false) over pkgs.espanso-wayland (making the original pkgs.espanso_wayland available under pkgs._espanso-wayland-orig).

Note that the updated package definition must be created on-the-fly during evaluation of the system configuration, since the path to the capability-enabled wrapper depends on the security.wrapperDir.

This has been working pretty well flawlessly on my system for several weeks months. It seems currently the only workaround for #249364 that keeps all of Espanso's features. For example, there is also #339594, but with it we lose Espanso's self-monitoring (which is used, e.g., for automatically re-loading the configuration when it changes). Happy about any feedback (especially from the corresponding maintainers @kimat @pyrox0 @n8henrie @numkem) to improve this if you think this is a viable way forward.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[n8henrie]

Thanks for your work on this!

This adds a considerable amount of complexity, and I'm a little concerned about how it will hold up over time (as context, espanso is having some growing pains and may be having some significant refactoring in the near future).

As far as I can tell, CAP_DAC_OVERRIDE allows Espanso arbitrary read / write access to the entire filesystem; while there are some mitigations (https://espanso.org/docs/install/linux#adding-the-required-capabilities), the espanso team is in charge of wielding these responsibly.

As an alternative, it seems that we could just move the service to run as root (and likely implement some of the numerous systemd-level protections to keep services well confined) and have a considerably simpler service module that -- AFAICT -- has effectively the same security concerns, but would have the benefit of keeping its systemd "jail" even as the package is updated over time.

Assuming that espanso could (in theory) read / write / change any file on the entire system before lowering its effective privileges, what do you see as the advantage of keeping this a user service instead of system-level / root?

[pitkling]

Thanks for the reply! :) I totally see the concerns regarding the complexity and that running this as a system-level service might be the more maintainable solution (I am not too familiar with the additional systemd level protections).

My main motivation for this route (in contrast to using it as a privileged system service) was to stay as close to the intended installation of espanso as possible. And, from my understanding of the difference between system/user level services (which might be incomplete/wrong), I tend to avoid running system level services for tasks used by probably only a few users. In particular, a text-expander seems like something that is very user-specific and into which a user should have to explicitly opt-in. If this is run as a system service, users might be unaware of the fact that there is a service running that could, in principle, capture their keystrokes.

[n8henrie]

I think your points are fair but also might miss my point:

If this is run as a system service, users might be unaware of the fact that there is a service running that could, in principle, capture their keystrokes.

From what I understand, setting these capabilities in effect gives this service the ability to capture and modify all content on the device -- including those same keystrokes, so I don't think it makes a difference. With these capabilities you are basically running espanso as setuid root, even though the espanso team promises to be well behaved and drop privileges as soon and as low as it can.

One thing I'm unsure of is WRT multiple user-level installs; the espanso config usually goes somewhere in ~/.config, so I'm not sure how well that would work if multiple users wanted to run their own snippets (perhaps one of the %i-style replacements for the user?).

Other ideas perhaps worth looking into (you may have already done this): the CapabilityBoundingSet systemd option seems to be how a few other packages have set CAP_DAC_OVERRIDE:

• https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#Capabilities

• nixpkgs/nixos/modules/services/networking/tetrd.nix

  Line 84 in 3f8971c

  "CAP_DAC_OVERRIDE"

• nixpkgs/nixos/modules/services/monitoring/netdata.nix

  Line 294 in 3f8971c

  "CAP_DAC_OVERRIDE" # is required for freeipmi and slabinfo plugins

[pitkling]

I think concerning the possibility to abuse CAP_DAC_OVERRIDE, we're on the same page. I absolutely agree that giving the user service those capabilities means it could exploit the unrestricted access to do more or less what it wants. The benefit of CAP_DAC_OVERRIDE is over setuid root is, as far as I understand, that (as long as espanso does as promised and surrenders the capabilities when no longer required) dependencies pulled in by espanso (which might get compromised at some point) that are used later have only restricted access.

[…] From what I understand, setting these capabilities in effect gives this service the ability to capture and modify all content on the device -- including those same keystrokes, so I don't think it makes a difference. […]

The difference that I see is that on a multi-user system, the system service would still be running in the background even for users that do not use espanso. As a user service, it is a true opt-in and a user not knowing espanso would not have to wonder whether espanso (or any of its dependencies) is trustworthy enough to have it listing on keystrokes. Even if a user knows and wants to use espanso, they might not want to have it running as a service but decide to run it when needed via espanso service start --unmanaged.

Other ideas perhaps worth looking into (you may have already done this): the CapabilityBoundingSet systemd option seems to be how a few other packages have set CAP_DAC_OVERRIDE: […]

I indeed stumbled over these but decided against them since I still think running this as a user service is more desirable. But I agree that if one wants to run espanso as a system service, this is the way to go.

About your concerns wrt multiple user-level installs: If espanso is run as a user service by systemd, that should be fine (espanso service register also relies on user-level systemd service). But I'm also not sure how that can be handled if it is run as a true system service, I'm not really familiar with the %i-style replacements.

---

I guess one could also completely decouple the services.espanso module from the programs.espanso module of my PR. So the services.espanso module would still use the original package. And users could manually enable programs.enableCapDacOverride to make available a version of the package with the capabilities enabled. This still leaves the question whether the maintenance of the extra code in the package is worth it to allow the option to build capability-enabled variant. However, note that the definition and injection of the wrapper lib is comparatively independent of the rest of the derivation.

Anyway, I totally get it if you think the maintenance burden of this might be too large to justify the possibility of running espanso manually or as a pure user-level service. I'd still try to polish this a little bit (feedback is welcome!) in case someone else wants to use this in their own nixpkgs fork.

[pitkling]

@n8henrie FYI: I did refactor this quite a bit. It is now simply a single, new module espanso-capdacoverride that, if enabled, provides a new espanso package (in the config attribute programs.espanso-capdacoverride.packageOverriden) which can be used purely on a user-level (so, e.g., manually as a non-root user or as a user-level service via home-manager). It requires no changes to the original espanso package or the system service module at all, so the maintenance burden should be minimal.

At the least, this should make it much easier if anyone wants to use this, even if this is not added to nixpkgs.

[n8henrie]

I've been thinking about this a fair amount but haven't come up with any better solutions. And to be frank, you seem much more knowledgeable than me about much of the details -- part of my hesitation is my poor familiarity with C and the C wrapper (simple as it may be).

Part of my wonders -- wouldn't it make sense for the wrapped package to be the default espanso on Linux? It seems like the setcap process is part of the recommended installation. I also use darwin quite a bit, and espanso works great with just a cargo build.

If espanso without the wrapper doesn't work, wouldn't it be best to make the wrapped espanso the only option?

[n8henrie]

Actually, the programs and wrappers stuff is all NixOS and so wouldn't apply to darwin anyway.

[pitkling]

I'm not too sure myself whether this is the right solution, but so far it is the best I could come up with to get it to work in user-space. Concerning the C wrapper: I stumbled upon this only recently, but it has been used in nixpkgs at other places, especially for binary drivers that sometimes contain hardcoded firmware file locations (see e.g., PR #260715). I'm using a similar wrapping library in another PR (#326272), also for a binary driver.

But given that this PR involves granting extra capabilities, it would clearly be good if someone more experienced with such things could take a look at the wrapper library and maybe comment whether this is a reasonable approach.

I wonder whether maybe @robryk might want to take a look at this. Quite some commits on security.wrappers originate from him and at least some of them seemingly involved readlink and /proc/self/exe (see, e.g., PR #251770). I was actually wondering whether it would not be more suitable to have security.wrappers make sure that process forks initiated by a wrapped, capability-enabled binary reference the wrapped binary instead of the original binary without the capabilities. But then again, not sure whether there are many other programs that have similar problems like espanso…

---

Concerning making the wrapped, capability-enabled espanso the default on Linux: In principle I would agree. One possible problem: I cannot see an implementation as a package only. The wrapped binary must know the config.security.wrapperDir path, so the package for the capability-enabled espanso must depend on the system config. As far as I understand (and I might be wrong, I'm relatively new to NixOS), this means we must use a module. But one could go one step further than my current draft PR and have programs.espanso-capdacoverride.enabled = true (or maybe even programs.espanso.enabled = true) automatically add an overlay on Linux that ensures that pkgs.espanso and pkgs.espanso-wayland refer to the capability-enabled package.

Then one should include a comment in the description of pkgs.espanso and pkgs.espanso-wayland that, on Linux, espanso must be enabled via programs.espanso.enabled = true to work correctly.

[robryk]

Initial comment after a quick skim: why doesn't the capability get inherited by the child process? (I don't remember the inheritance rules by heart, and they are somewhat weird given that they're trying to handle a very weird model, but the wrappers execs the actual process and that execs ends up preserving capabilities, so I would expect the next one to also do so.)

Concrete questions: do you know that this doesn't "just work" with wrappers?

[pitkling]

@robryk Thanks for taking a look and the quick response!

I did indeed try to simply use the wrapper, without any success. That said, I'm using Linux only occasionally and am not too familiar with the capability framework, so it is very well possible that I simply did overlook something.

Specifically, I tried the following wrapper for espanso under wayland:

({ config, lib, ... }: {
  security.wrappers.espanso = {
    source = "${lib.getExe pkgs.espanso-wayland}";
    capabilities = "cap_dac_override+eip";
    owner = "root";
    group = "root";
  }
})

If I start espanso with this via /run/wrappers/bin/espanso service start --unmanaged, it fails and the log (/run/wrappers/bin/espanso log) indicates that the process is missing the CAP_DAC_OVERRIDE capability. Above I used cap_dac_override+eip to be on the safe side. Espanso originally only requires cap_dac_override+p (see "Adding the required Capabilities" in the documentation).

Before I did this implementation, I had a quick look at the capability inheritance rules in the man page and also at how these are transformed during a fork (under "Transformation of capabilities during execve()" in the man page). I admit that I might very well be misunderstanding something, but here is what I think might be the problem:

• The wrapper has the correct file capabilities set via the security.wrappers framework.
• If I understand the security.wrappers implementation correctly, it raises those file capabilities on the wrapper into the ambient set, so that those capabilities are inherited when the wrapper forks the actual binary.
• Now, as explained in the espanso documentation (under "Security Considerations"), espanso is rather careful to drop any unnecessary capabilities whenever they are not required. In particular, the espanso main process (as started by the user) does not require those capabilities and drops them before forking other processes (some of which do require the capabilities).
• Since espanso drops the capabilities even from the permitted set, this also clears the capabilities from the ambient set (see "Thread capability sets" in the capabilities man page).
• The subsequent fork of the worker process uses /proc/self/exe to determine the forked binary. That link points not to the wrapper (with the file capabilities) but to the actual binary (without the file capabilities).
• So when the forking happens, the running process has no ambient capabilities set and does not use a binary with file capabilities. Thus, the worker process does not inherit the required capabilities.

Again, take this with a grain of salt, since I neither have much experience with the linux capabilities nor with Rust (so, I might be misreading what espanso is doing).

[robryk]

Above I used cap_dac_override+eip to be on the safe side. Espanso originally only requires cap_dac_override+p (see "Adding the required Capabilities" in the documentation).

The capsets in the wrappers are a bit of a lie anyway (due to the weirdnesses I mentioned earlier it's impossible to simulate filecaps on the target file with a wrapper), so you're getting more than you asked for usually anyway.

I'll check whether the story you're painting is correct later, but it both sounds plausible and would explain the issue. Give me some time to think about the least weird way to deal with this.

[pitkling]

Just updated the PR draft, doing some cleanup and incorporating points from several discussions with @n8henrie (in particular this comment in the related PR #339594 that uses an alternative approach to fix the same issue).

The wrapper is now only used for pkgs.espanso-wayland. Moreover, by default it automatically overlays pkgs.espanso-wayland with a capability-enabled version. As described in the updated PR explanation, this must be done via an overlay this the capability-enabled espanso-wayland package depends on the system configuration (security.wrapperDir).

[pitkling]

BTW: I'm not sure whether a module-generated overlay to override a package is a good idea, especially if enabled by default. It was just an idea to have this automatically pick up by anything using pkgs.espanso-wayland, which is basically broken without the capabilities.

Instead of enabling this by default, it seems to make most sense to have it disabled by default but enable it in the NixOS Espanso module whenever that is enabled. Unfortunately, we cannot enable this from the HM Espanso module (I think), so HM users would have to enable it manually in their NixOS system configuration.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/espanso-daemon-problem/35309/27

[pitkling]

I removed the draft status of this PR and did some minor cleanups in the description and code. In particular, the module is now not enabled by default.

@n8henrie Did you have time to think about this and/or your approach in #339594? I thought again a bit about the general issue during the last couple of days but couldn't come up with another, more elegant way that keeps Espanso's monitoring capabilities. But still, it is probably completely fine to use #339594 instead if we don't want the extra monitoring capabilities.

Would be nice if we could get a fix for espanso-wayland (either this or #339594) into the 24.11 NixOS release 🙂 (if possible, not sure about the time frame…).

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/home-managers-espanso-module-does-not-create-config-directory/51170/14

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/using-espanso-on-wayland-nixos/55268/2

[n8henrie]

I think this is looking better and more palatable over time. I still wonder if a more general fix could be implemented at the security.wrappers level, but I imagine that would require significant ceremony and/or investment in the effort by one of the nix VIPs. This seems like a pretty reasonable workaround in the meantime, if nothing else ("nothing is as permanent as a temporary fix" haha).

@pitkling thanks for all the effort you've put into this and for your excellent communication.

[pitkling]

@n8henrie: It's still on my todo-list to investigate security.wrappers a bit more and open an issue/discussion to see whether that is possible, just didn't get around to it yet… but in any case I also think that such a more general fix will take time, so for now I think it would be good to implement a workaround like this one.

Some thoughts/questions we should probably still discuss (even though they are not necessarily part of this PR):

• Do you think the current interface (automatic override of pkgs.espanso-wayland if the user sets programs.espanso.capdacoverride.enable = true) is fine? Let me know if I should change anything.

• pkgs.espanso-wayland is basically broken without this. So I guess we should at least reference the requirement to set programs.espanso.capdacoverride.enable = true in the documentation for pkgs.espanso-wayland and possibly in the documentation for the NixOS module services.espanso as well as in the documentation for the Home Manager (HM) module services.espanso.

  One might also think about whether to do one of the following:

  • Automatically set programs.espanso.capdacoverride.enable = true if the user tries to set services.espanso.package to a Wayland package. This won't be possible from the HM module.
  • Assert that programs.espanso.capdacoverride.enable = true if the user tries to set services.espanso.package to a Wayland package. This should probably be also possible from the HM Module, but only if it is not a standalone HM installation (for which pkgs.espanso-wayland wouldn't work in any case).

• Since this change defines the new espanso-wayland package at evaluation time (because it depends on the option security.wrapperDir), installing pkgs.espanso-wayland with this will always result in a local rebuild (not using the binary cache). In almost all cases, security.wrapperDir will be "/run/wrappers/bin" (in particular, since it is an internal variable which is only there for NixOS developers). It might make sense to add a "hidden" derivation like pkgs._espanso-wayland with the default value of security.wrapperDir hardcoded. That way, it should end up in the binary cache (I think I have seen packages doing something like this for the same reasons, but not sure; pkgs.singularity seems to define pkgs.singularity-overriden-nixos solely because of the binary cache, see here, here, and here).

[n8henrie]

Thanks for all your efforts maintaining this @pitkling

I hope we can get some attention from folks a little more senior (and with the merge permissions). Have you posted to the "PRs ready for review" thread by chance?

Any opinions from other espanso maintainers -- @numkem is listed for the module.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/4910

[pitkling]

[…] I hope we can get some attention from folks a little more senior (and with the merge permissions). Have you posted to the "PRs ready for review" thread by chance?

Now I have, totally forgot about that thread before 😅.

@robryk: Just pinging you since you took already a quick glance at this some time back. If you have time to review this and give your opinion would be great, but no worries if not.