doas ignores environment variables set up in PAM, leading to many issues on NixOS

[PaulGrandperrin]

The initial weird bug

I wanted to set my time locale to ISO-8601, so I did:

i18n.extraLocaleSettings.LC_TIME = "en_DK.UTF-8"; # yes, that means ISO-8601 ;-)

but then, using doas to get root printed some warnings

$ doas -s                                                                                                                                                   
/nix/store/4qnbxbg1v7b51vgfpmbxkv35mh1vn7bh-set-environment: line 14: warning: setlocale: LC_TIME: cannot change locale (en_DK.UTF-8): No such file or directory
#

but date in that root shell was using the correct locale

$ date
2022-02-10T15:27:12 CET

I still investigated a bit and found that doas date did not use the correct locale !???

$ doas date
Thu Feb 10 15:30:00 CET 2022

After spending way too many hours troubleshooting this mystery, I finally found the root cause: Duncaen/OpenDoas#2. It can also lead to many others difficult to understand issues..

Explanations

NixOS relies on some environment variables to be set to work properly.
For example, PATH must include /run/wrappers/bin for SUID bins, /run/current-system/sw/bin for system bins and /etc/profiles/per-user/$USER/bin for user bins.
We'll come back to PATH later.

Another important one is LOCALE_ARCHIVE, which is used by Nixpkgs' patched glibc to find which locale archive to load.
Basically, the patch first tries to lookup LOCALE_ARCHIVE and if unsuccessful, loads a basic archive with only the "C/POSIX" locale available.
(more info can be found here also: #85823)

To ensure that they are set everywhere, NixOS configures PAM to include them in all new sessions:
https://github.com/NixOS/nixpkgs/blob/nixos-21.11/nixos/modules/config/system-environment.nix#L68
https://github.com/NixOS/nixpkgs/blob/nixos-21.11/nixos/modules/security/pam.nix#L573

Unfortunately doas ignore those: Duncaen/OpenDoas#2.

This makes binaries started with doas behave incorrectly on NixOS:

• If it's a shell, the locale-archive will fail to load and fall back to a basic one. However, login shells also source /etc/set-environment which also load those variables. This doesn't solve the locale issue for the shell because the locale-archive is loaded only at startup time but it makes programs started from the shell behave correctly.
• If it's a command, many variables will be missing, including LOCALE_ARCHIVE, the user env in PATH (/etc/profiles/per-user/$USER/bin), NIX_PATH, the XDG vars, INFOPATH, LD_LIBRARY_PATH and many others. Right now the PATH has some nix-specific paths thanks to this patch: https://github.com/NixOS/nixpkgs/blob/nixos-21.11/pkgs/tools/security/doas/0001-add-NixOS-specific-dirs-to-safe-PATH.patch

For example, on my machine, git is installed through home-manager and so is in /etc/profiles/per-user/$USER/bin.
Scripts launched with doas will not find git.

$ doas ruby -e 'puts `git`' 
Traceback (most recent call last):
	1: from -e:1:in `<main>'
-e:1:in ``': No such file or directory - git (Errno::ENOENT)

$ sudo ruby -e 'puts `git`' 
usage: git [--version] [--help] [-C <path>] [-c <name>=<value>]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p | --paginate | -P | --no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           [--super-prefix=<path>] [--config-env=<name>=<envvar>]
           <command> [<args>]
...

They'll also fail to load any locale.

Comparing with sudo

$ doas ruby -e 'ENV.each {|k,v| puts "#{k}=#{v}"};'                                                                                                         
DISPLAY=:0
DOAS_USER=paulg
HOME=/root
LOGNAME=root
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/run/wrappers/bin:/run/current-system/sw/bin:/run/current-system/sw/sbin:/usr/local/bin:/usr/local/sbin
SHELL=/run/current-system/sw/bin/fish
SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
TERM=xterm-256color
TERMINFO_DIRS=/home/paulg/.nix-profile/share/terminfo:/etc/profiles/per-user/paulg/share/terminfo:/nix/var/nix/profiles/default/share/terminfo:/run/current-system/sw/share/terminfo
USER=root

$ sudo ruby -e 'ENV.each {|k,v| puts "#{k}=#{v}"};'                                                                                                         
LANG=en_US.UTF-8
XAUTHORITY=/run/user/1000/.mutter-Xwaylandauth.LWUIH1
PATH=/nix/store/2c9w4p2x6x0l64fdvcmc11app7x4xran-python3-3.9.6/bin:/nix/store/jqnrfnvn02lhh5qgpn6s6mlr4972fw5s-terminator-2.1.1/bin:/nix/store/7d5h782gxrvcwryzybx4jhh2fb4jpqsh-cairo-1.16.0-dev/bin:/nix/store/7rhxra449n5k56yfz25dvksj09rhmir8-freetype-2.11.0-dev/bin:/nix/store/fzl3is02f0lhqzcdx7v22wyi6d0maj4a-bzip2-1.0.6.0.2-bin/bin:/nix/store/7zsf9zcywvxvq6krgljrr7km3hsbls6s-libpng-apng-1.6.37-dev/bin:/nix/store/brl5wr9r4p1k5f57ysrivdilc4iqkf82-fontconfig-2.13.94-bin/bin:/nix/store/aa46grxlz348yf1lcqjmlxf1cz1fvnsp-expat-2.4.3-dev/bin:/nix/store/8v697gb948563iznn9g70ijbifaza34y-glib-2.70.1-dev/bin:/nix/store/7rhf667rkqhcljcyz9b9mrzjw1sh3qji-gettext-0.21/bin:/nix/store/wys2ank0vwrjf3zi751m3lcp9c6g0hxc-glib-2.70.1-bin/bin:/run/wrappers/bin:/etc/profiles/per-user/paulg/bin:/run/current-system/sw/bin:/home/paulg/.nix-profile/bin:/home/paulg/.nix-profile/bin:/nix/var/nix/profiles/default/bin
TERMINFO_DIRS=/home/paulg/.nix-profile/share/terminfo:/etc/profiles/per-user/paulg/share/terminfo:/nix/var/nix/profiles/default/share/terminfo:/run/current-system/sw/share/terminfo
LC_MONETARY=fr_FR.UTF-8
LC_MEASUREMENT=en_DK.UTF-8
LC_TIME=en_DK.UTF-8
DISPLAY=:0
SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
COLORTERM=truecolor
TERM=xterm-256color
MAIL=/var/mail/root
LOGNAME=root
USER=root
HOME=/root
SHELL=/run/current-system/sw/bin/fish
SUDO_COMMAND=/etc/profiles/per-user/paulg/bin/ruby -e ENV.each {|k,v| puts "#{k}=#{v}"};
SUDO_USER=paulg
SUDO_UID=1000
SUDO_GID=100
GTK_PATH=/root/.nix-profile/lib/gtk-2.0:/root/.nix-profile/lib/gtk-3.0:/etc/profiles/per-user/root/lib/gtk-2.0:/etc/profiles/per-user/root/lib/gtk-3.0:/nix/var/nix/profiles/default/lib/gtk-2.0:/nix/var/nix/profiles/default/lib/gtk-3.0:/run/current-system/sw/lib/gtk-2.0:/run/current-system/sw/lib/gtk-3.0
INFOPATH=/root/.nix-profile/info:/root/.nix-profile/share/info:/etc/profiles/per-user/root/info:/etc/profiles/per-user/root/share/info:/nix/var/nix/profiles/default/info:/nix/var/nix/profiles/default/share/info:/run/current-system/sw/info:/run/current-system/sw/share/info
KDEDIRS=/root/.nix-profile:/etc/profiles/per-user/root:/nix/var/nix/profiles/default:/run/current-system/sw
LD_LIBRARY_PATH=/nix/store/cr3mzb6gi2h94ah58yr64s0w78zsazp5-pipewire-0.3.40-jack/lib
LIBEXEC_PATH=/root/.nix-profile/lib/libexec:/etc/profiles/per-user/root/lib/libexec:/nix/var/nix/profiles/default/lib/libexec:/run/current-system/sw/lib/libexec
LIBVA_DRIVER_NAME=iHD
LOCALE_ARCHIVE=/run/current-system/sw/lib/locale/locale-archive
MOZ_PLUGIN_PATH=/root/.nix-profile/lib/mozilla/plugins:/etc/profiles/per-user/root/lib/mozilla/plugins:/nix/var/nix/profiles/default/lib/mozilla/plugins:/run/current-system/sw/lib/mozilla/plugins
NAUTILUS_EXTENSION_DIR=/nix/store/nma7q20asrr2rmydxi8v3fcx077p7zwm-system-path/lib/nautilus/extensions-3.0
NIX_GSETTINGS_OVERRIDES_DIR=/nix/store/56dmq50if1d3cgmrzjkqvk30967jbnf5-nixos-gsettings-desktop-schemas/share/gsettings-schemas/nixos-gsettings-overrides/glib-2.0/schemas
NIX_PATH=nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixos:nixos-config=/etc/nixos/configuration.nix:/nix/var/nix/profiles/per-user/root/channels
QTWEBKIT_PLUGIN_PATH=/root/.nix-profile/lib/mozilla/plugins/:/etc/profiles/per-user/root/lib/mozilla/plugins/:/nix/var/nix/profiles/default/lib/mozilla/plugins/:/run/current-system/sw/lib/mozilla/plugins/
QT_PLUGIN_PATH=/root/.nix-profile/lib/qt4/plugins:/root/.nix-profile/lib/kde4/plugins:/etc/profiles/per-user/root/lib/qt4/plugins:/etc/profiles/per-user/root/lib/kde4/plugins:/nix/var/nix/profiles/default/lib/qt4/plugins:/nix/var/nix/profiles/default/lib/kde4/plugins:/run/current-system/sw/lib/qt4/plugins:/run/current-system/sw/lib/kde4/plugins
TZDIR=/etc/zoneinfo
VDPAU_DRIVER=va_gl
XCURSOR_PATH=/root/.icons:/root/.local/share/icons:/root/.nix-profile/share/icons:/root/.nix-profile/share/pixmaps:/etc/profiles/per-user/root/share/icons:/etc/profiles/per-user/root/share/pixmaps:/nix/var/nix/profiles/default/share/icons:/nix/var/nix/profiles/default/share/pixmaps:/run/current-system/sw/share/icons:/run/current-system/sw/share/pixmaps
XDG_CONFIG_DIRS=/root/.nix-profile/etc/xdg:/etc/profiles/per-user/root/etc/xdg:/nix/var/nix/profiles/default/etc/xdg:/run/current-system/sw/etc/xdg
XDG_DATA_DIRS=/nix/store/mp04xqq9nbn9qkvbv2sc7qz4v7g28y89-gnome-mimeapps/share:/nix/store/8b34cja46caj1wmagvfgki9qvxsh13ql-desktops/share:/root/.nix-profile/share:/etc/profiles/per-user/root/share:/nix/var/nix/profiles/default/share:/run/current-system/sw/share
XDG_DESKTOP_PORTAL_DIR=/nix/store/4av5fcs8wh753wdrcc6l1f6v26vba31l-xdg-portals/share/xdg-desktop-portal/portals

Solutions

Fix doas

The obvious solution is to make doas set the vars given by PAM, just like sudo. I'll speak with the maintainer referring to this issue.

Work around

Since we control what goes in /etc/pam/environment , we can also set them in /etc/doas.conf.
It's an eyesore and can cannot handle variable interpolation like PAM, but here's my workaround:

        setEnv = with lib; let # because of https://github.com/Duncaen/OpenDoas/issues/2 we need to add here all variables that should have been read from PAM_env
          # code inspired from https://github.com/NixOS/nixpkgs/blob/nixos-21.11/nixos/modules/config/system-environment.nix#L69
          suffixedVariables = 
            flip mapAttrs config.environment.profileRelativeSessionVariables (envVar: suffixes:
              flip concatMap config.environment.profiles (profile:
                map (suffix: "${profile}${suffix}") suffixes
              )
            );
          suffixedVariablesWithWrappers = (zipAttrsWith (n: concatLists)
            [
              # Make sure security wrappers are prioritized without polluting
              # shell environments with an extra entry. Sessions which depend on
              # pam for its environment will otherwise have eg. broken sudo. In
              # particular Gnome Shell sometimes fails to source a proper
              # environment from a shell.
              { PATH = [ config.security.wrapperDir ]; }

              (mapAttrs (n: toList) config.environment.sessionVariables)
              suffixedVariables
            ]
            );
          replaceEnvVars = replaceStrings ["$HOME" "$USER"] ["/root" "root"];
          doasVariable = k: v: ''${k}=${concatStringsSep ":" (map replaceEnvVars (toList v))}'';

        in mapAttrsToList doasVariable suffixedVariablesWithWrappers;

Inform the users?

In the meantime, I think we should update the docs to warn NixOS users of those shortcomings and workarounds.
This is especially important because those bugs are difficult to analyze, and it is very unexpected that doas ignores PAM env vars.

[MattSturgeon]

I think I ran into this when running doas nixos-rebuild switch --flake .... Flakes require git, which I have installed via home-manager, but when running under doas, git cannot be found.

At first I assumed keepEnv wasn't being applied correctly, however the generated /etc/doas.conf has it present. Investigating further it looks like PATH isn't being "kept" by keepEnv, but other variables are:

$ which git
/home/matt/.nix-profile/bin/git

$ doas which git
which: no git in (/bin:/sbin:/usr/bin:/usr/sbin:/run/wrappers/bin:/run/current-system/sw/bin:/run/current-system/sw/sbin:/usr/local/bin:/usr/local/sbin)

$ echo $PATH
/run/wrappers/bin:/home/matt/.nix-profile/bin:/etc/profiles/per-user/matt/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin

$ doas bash -c 'echo $PATH'
/bin:/sbin:/usr/bin:/usr/sbin:/run/wrappers/bin:/run/current-system/sw/bin:/run/current-system/sw/sbin:/usr/local/bin:/usr/local/sbin

$ export FOO=bar

$ doas bash -c 'echo $FOO'
bar

Assuming this is because PATH is set by pam as per this issue.

Can the workaround be included in my nix system configuration or does it need to be added to nixpkgs instead?

[pyrotelekinetic]

Since we control what goes in /etc/pam/environment , we can also set them in /etc/doas.conf.
It's an eyesore and can cannot handle variable interpolation like PAM, but here's my workaround:

Is there a reason not to upstream this workaround into the doas module?

[PaulGrandperrin]

Because it's really an eyesore 😬

And anyway, the upstream project seems kinda dead.

I removed it from my conf and installed please instead.

[PaulGrandperrin]

Oh actually, sudo-rs seems to be the project to follow!