Sonoma 14.6 breaks ssh-askpass

[lukaskuzmiak]

Just after update to Sonoma 14.6 today I get:

sign_and_send_pubkey: signing failed for ED25519 "/Users/lukash/.ssh/<redacted>" from agent: agent refused operation

Same for my co-worker, it definitely worked ok on 14.5 just before this update (hours ago).

[dataviruset]

Seems the service can't start:

√ ~$ brew services restart theseal/ssh-askpass/ssh-askpass
Stopping `ssh-askpass`... (might take a while)
==> Successfully stopped `ssh-askpass` (label: homebrew.mxcl.ssh-askpass)
==> Successfully started `ssh-askpass` (label: homebrew.mxcl.ssh-askpass)

But it's still not running:

√ ~$ brew services info ssh-askpass
ssh-askpass (homebrew.mxcl.ssh-askpass)
Running: ✘
Loaded: ✔
Schedulable: ✘

When trying to start it:

√ ~$ brew services start ssh-askpass
Bootstrap failed: 5: Input/output error
Try re-running the command as root for richer errors.
Error: Failure while executing; `/bin/launchctl bootstrap gui/501 /Users/redacted/Library/LaunchAgents/homebrew.mxcl.ssh-askpass.plist` exited with 5.
?1 ~$ sudo brew services start ssh-askpass
Password:
Warning: Taking root:admin ownership of some ssh-askpass paths:
  /bin
  /bin/sh
  /opt/homebrew/opt/ssh-askpass
  /opt/homebrew/opt/ssh-askpass/bin
  /opt/homebrew/var/homebrew/linked/ssh-askpass
This will require manual removal of these paths using `sudo rm` on
brew upgrade/reinstall/uninstall.
Error: Operation not permitted @ apply2files - /bin

I tried deleting the /opt/homebrew folders mentioned and reinstalling ssh-askpass but it didn't help. I don't wanna try what happens if I delete /bin or /bin/sh... 😅
The owner of /bin is root:wheel these days and not root:admin.

[zeyugao]

@dataviruset You can directly set SSH_ASKPASS and SUDO_ASKPASS to test it without the services. It is due to the system I think, not the services

export SSH_ASKPASS=/opt/homebrew/opt/touch-auth/bin/touch-auth  # Change it
unset SSH_AUTH_SOCK
echo $SSH_AUTH_SOCK
eval $(ssh-agent -s)
echo $SSH_AUTH_SOCK

[dataviruset]

@dataviruset You can directly set SSH_ASKPASS and SUDO_ASKPASS to test it without the services. It is due to the system I think, not the services

export SSH_ASKPASS=/opt/homebrew/opt/touch-auth/bin/touch-auth  # Change it
unset SSH_AUTH_SOCK
echo $SSH_AUTH_SOCK
eval $(ssh-agent -s)
echo $SSH_AUTH_SOCK

Using my original SSH_ASKPASS value (it's set to /opt/homebrew/opt/ssh-askpass/bin/ssh-askpass) and then running these commands worked for me:

unset SSH_AUTH_SOCK
eval $(ssh-agent -s)
ssh-add -c ~/.ssh/my_key
ssh test-machine.example.com

Interestingly the SSH_ASKPASS environment variable is already set. Probably the brew package set that up for me. But the built-in Mac OS SSH agent seems to not work anymore as of Mac OS 14.6...

[EXHades]

I added keys to the ssh-agent using keepassxc.
just deselect Require user confirmation when this key is used option, locked keepassxc and then unlocked it again, Now ssh -T example.com works fine, But SSH_ASKPASS still doesn't work.

keepassxreboot/keepassxc#9955 (comment)

---

macOS 14.6 upgrade break ssh-agent SSH_AGENT_CONSTRAIN_CONFIRM support?

[micolous]

If your organisation depends on this working and has a support agreement with Apple, please point them at this bug. 😄

I dug in to this a bit today, it looks like Sonoma 14.6 has changed the environment variable inheritance rules for system-provided user daemons (/System/Library/LaunchAgents); so that environment variables set with launchctl setenv no longer apply.

This means that when macOS starts ssh-agent through launchctl (to handle the default SSH_AUTH_SOCK on demand), it no longer picks up SSH_ASKPASS and DISPLAY from this ssh-agent's launchd plist.

The work-around @zeyugao proposed side-steps all of this by not using macOS' ssh-agent config – but this only works in a per-shell context. Similarly, if you replaced ssh with the one from Homebrew and put it first in your path; or used a different environment variable to SSH_AUTH_SOCK to point at a custom ssh-agent, you'd also work around this... but that also kinda smells.

On a system with Sonoma 14.5 with xquartz installed, I see:

% launchctl print gui/501/com.openssh.ssh-agent
gui/501/com.openssh.ssh-agent = {
	active count = 1
	path = /System/Library/LaunchAgents/com.openssh.ssh-agent.plist
	type = LaunchAgent
	state = running

	program = /usr/bin/ssh-agent
	arguments = {
		/usr/bin/ssh-agent
		-l
	}

	inherited environment = {
		DISPLAY => /private/tmp/com.apple.launchd.XXXX/org.macosforge.xquartz:0
		SUDO_ASKPASS => /usr/local/opt/ssh-askpass/bin/ssh-askpass
		SSH_ASKPASS => /usr/local/opt/ssh-askpass/bin/ssh-askpass
		SSH_AUTH_SOCK => /private/tmp/com.apple.launchd.XXXX/Listeners
	}

	default environment = {
		PATH => /usr/bin:/bin:/usr/sbin:/sbin
	}

	environment = {
		MallocSpaceEfficient => 1
		XPC_SERVICE_NAME => com.openssh.ssh-agent
	}

On a system with Sonoma 14.6 without xquartz, I see:

% launchctl print gui/501/com.openssh.ssh-agent
gui/501/com.openssh.ssh-agent = {
	active count = 1
	path = /System/Library/LaunchAgents/com.openssh.ssh-agent.plist
	type = LaunchAgent
	state = running

	program = /usr/bin/ssh-agent
	arguments = {
		/usr/bin/ssh-agent
		-l
	}

	inherited environment = {
		SSH_AUTH_SOCK => /private/tmp/com.apple.launchd.XXXX/Listeners
	}

	default environment = {
		PATH => /usr/bin:/bin:/usr/sbin:/sbin
	}

	environment = {
		MallocSpaceEfficient => 1
		XPC_SERVICE_NAME => com.openssh.ssh-agent
	}

With Sonoma 14.6 with xquartz installed, I see:

% launchctl print gui/501/com.openssh.ssh-agent
gui/501/com.openssh.ssh-agent = {
	active count = 1
	path = /System/Library/LaunchAgents/com.openssh.ssh-agent.plist
	type = LaunchAgent
	state = running

	program = /usr/bin/ssh-agent
	arguments = {
		/usr/bin/ssh-agent
		-l
	}

	inherited environment = {
		DISPLAY => /private/tmp/com.apple.launchd.XXXX/org.xquartz:0
		SSH_AUTH_SOCK => /private/tmp/com.apple.launchd.XXXX/Listeners
	}

	default environment = {
		PATH => /usr/bin:/bin:/usr/sbin:/sbin
	}

	environment = {
		MallocSpaceEfficient => 1
		XPC_SERVICE_NAME => com.openssh.ssh-agent
	}

Other macOS system daemons (eg: com.apple.weatherd) are impacted in the same way – so this is not specific to ssh-agent or the way ssh-askpass works.

User-installed software in /Library/LaunchAgents and ~/Library/LaunchAgents don't seem to be affected by this change.

The fact that xquartz (a non-system application) can still set the DISPLAY environment variable for ssh-agent suggests that it should be possible for this ssh-askpass to set environment variables in some way, but from what I'm reading of the launchd.plist man page, it won't be elegant.

I suspect that this was part of some other security / platform hardening change, but I'm unable to find any specific release notes about this.

It's something that Apple will need to fix.

[zeyugao]

@micolous Have you succeeded in setting SSH_ASKPASS to workaround this manually? I just propose a method to bypass the usage of services as @dataviruset says the services failed to start.

I manually set the SSH_ASKPASS and started a ssh-agent, but I still got agent refused operation

[micolous]

@micolous Have you succeeded in setting SSH_ASKPASS to workaround this manually?

No.

What matters is the environment that ssh-agent runs in, which when it's spawned by launchd, ignores launchctl setenv on macOS Sonoma 14.6.

The error we're seeing is because ssh-agent has no way to ask us for confirmation:

• if DISPLAY is set, SSH_ASKPASS is still unset, so it doesn't know what to do and reject it.
• if DISPLAY is unset, then it'll ask on stdin... but that's not actually connected to anything so it'll reject it.

The work-around I've got for now is to disable confirmation, but I've set up my SSH keys using /usr/lib/ssh-keychain.dylib to access my keys over PKCS#11 / smartcard auth, and the smart card (Yubikey with PIV applet) is configured to require physical confirmation for signing operations.

It just means now I don't get any prompt on screen, just a flashing light on the side of my computer.

If you're using ssh-askpass to confirm access to a key stored on disk, then disabling confirmation has a pretty significant security drawback 😓

[lukaskuzmiak]

I worked around it by running a user-land instance of ssh-agent with proper environment variables set, then overwriting SSH_AUTH_SOCK in .zprofile and on top of that setting IdentityAgent in ~/.ssh/config to the path of the user-land ssh-agent.

Some GUI apps might not pick that up, but so far without issues. Other than it being annoying to setup.

For anyone interested here are some resources that helped me:

• https://gist.github.com/vancluever/de1c3985c8f9e2a3c4bdc42a057e075e
• 1Password uses a similar approach as described above and has a nice client compatibility table here: https://developer.1password.com/docs/ssh/agent

Given the fact Apple likely broke this for good I think that will be the way forward.

[micolous]

Yeah, your setup and 1Password's replace the default Apple-provided SSH agent setup (partially or entirely). There are usability edge cases to that, as you've already noted. If you go further (as 1Password do, and has been suggested elsewhere) and replace ssh-agent entirely, you also miss out on security hardening features only available in Apple's SSH binaries.

I acknowledge that there are tradeoffs, like that those hardening features can also prevent you from using some more "obscure" functionality (like FIDO2 resident keys and PIV with ECDSA keys).

For what it's worth, there are large corporates and government agencies with macOS workstations and a custom ssh-askpass implementation (to support PKCS#11 auth with hardware keystores), but their corporate IT teams have probably noticed this in testing and are holding back the update from their fleets.

At this point, I'm quietly optimistic that it won't remain broken forever... it just needs to get in front of the right set of eyes. 😄

[sprig]

I think the issue has been (at lease partially) misdiagnosed by @micolous;

(FWIW I use a different askpass program but I think the issue is similar)
First, I've noticed that if I don't initially load keys into ssh-agent then I do get asked for a password by ssh-askpass. However if I do load keys ssh complains that agent refused operation.

As an experiment, I tried unsetting the SSH_ASKPASS variable in the terminal but setting SSH_ASKPASS_REQUIRE=force, and running ssh -vv host:
I notice lines like the following:

ssh_askpass: exec(/usr/X11R6/bin/ssh-askpass): No such file or directory

So I add a tiny script to the above location:

#!/bin/sh
echo "$@" > ~/askpass
date >> ~/askpass
env >> ~/askpass

I notice that when keys are not loaded, I get output as follows:

$ cat ~/askpass
Enter passphrase for key '/Users/user/.ssh/id_rsa': 
Sun Aug  4 11:07:55 PDT 2024
TERM_PROGRAM=Apple_Terminal
LC_MONETARY=en_US.UTF-8
QT_STYLE_OVERRIDE=Fusion
LUA_BINDIR=<redacted>
TERM=xterm-256color
SHELL=/bin/zsh
TMPDIR=/var/folders/vx/wq1n25gs5t54v4sp3_6yj3c00000gn/T/
TERM_PROGRAM_VERSION=453
LC_NUMERIC=en_US.UTF-8
CURL_SSL_BACKEND=secure-transport
TERM_SESSION_ID=516E43F7-2240-481C-A308-A13AD18FE7B4
LC_ALL=en_US.UTF-8
CDPATH=.:<redacted>
USER=user
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.LVjkwSsCS6/Listeners
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
PAGER=less
PATH=<long list of paths>
LC_MESSAGES=en_US.UTF-8
LUA_DIR=<redacted>
LaunchInstanceID=9A548B16-A27C-4770-B569-8A81CC12830F
QT_QPA_PLATFORMTHEME=gtk2
WORDCHARS=
__CFBundleIdentifier=com.apple.Terminal
LC_COLLATE=en_US.UTF-8
PROFILE_RAN=Sun Aug  4 09:49:08 PDT 2024
PWD=/Users/user
EDITOR=vim
LANG=en_US.UTF-8
LUA_PATH=<redacted>
XPC_FLAGS=0x0
SSH_ASKPASS_REQUIRE=force
XPC_SERVICE_NAME=0
SHLVL=2
HOME=/Users/user
LOGNAME=user
PYTHONPATH=<redacted>
LESS=-F -g -i -M -R -S -w -z-4
LC_CTYPE=en_US.UTF-8
BROWSER=open
DISPLAY=/private/tmp/com.apple.launchd.NVQ3vVyfbz/org.xquartz:0
SECURITYSESSIONID=186ab
LC_TIME=en_US.UTF-8
_=/usr/bin/env

In particular, the PROFILE_RAN variable above is set to date whenever my ~/.profile is sourced, one can notice that it is significantly different from when the askpass script was run. In fact, it is identical to what I have in that particular terminal session:

$ echo $PROFILE_RAN
Sun Aug  4 09:49:08 PDT 2024

i.e. the script runs in the same context as the ssh command, likely invoked directly by ssh.

Whereas if keys are loaded I get similar to the following:

$ cat ~/askpass 
Allow use of key user@host?
Key fingerprint SHA256:<redacted>.
SHELL=/bin/zsh
SSH_ASKPASS_PROMPT=confirm
TMPDIR=/var/folders/vx/wq1n25gs5t54v4sp3_6yj3c00000gn/T/
MallocSpaceEfficient=1
USER=user
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.LVjkwSsCS6/Listeners
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
PATH=/usr/bin:/bin:/usr/sbin:/sbin
PWD=/
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
SHLVL=1
HOME=/Users/user
LOGNAME=user
DISPLAY=/private/tmp/com.apple.launchd.NVQ3vVyfbz/org.xquartz:0
_=/usr/bin/env

which is a fairly vanilla environment, likely invoked by ssh-agent. Furthermore, since the script above returns 0 (success), the authentication is successful.

The conclusion is that rather than not recognizing SSH_ASKPASS environment variable which is read by ssh and not by the agent, ssh-agent fails to create a window when invoked by ssh-agent. Likely similar issues to what prompted reattach-to-user-namespace in https://github.com/ChrisJohnsen/tmux-MacOSX-pasteboard. I don't know if that tool still works but if it doesn't this suggests that another viable workaround; Separate this into a client/server setup where the server would run in the user's session and be able to open windows whereas the client would simply forward authentication requests from the agent.

[sprig]

update: I can confirm that reattach-to-user-namespace still works and using a wrapper script in /usr/X11R6/bin/ssh-askpass such as

#!/bin/sh
exec ~/local/bin/reattach-to-user-namespace ~/local/ssh-askpass/ssh-askpass "$@"

where programs are at the respective locations above opens the confirmation dialog and proceeds to successfully authenticate. Presumably pointing SSH_ASKPASS at the wrapper would work as well.

EDIT:
Upon further testing it seems that the issue @micolous identified is at play here as well - the environment does not get set. However, the path above is the default location where ssh-askpass is searched for and so putting the wrapper there still works as a workaround.

[micolous]

I can confirm that reattach-to-user-namespace still works and using a wrapper script in /usr/X11R6/bin/ssh-askpass

I tried your work-around on Sonoma 14.5 and 14.6 with XQuartz installed (so DISPLAY is set by SecureSocketWithKey, rather than launchctl setenv).

I removed the ssh-askpass.plist LaunchAgent and cleared out any traces from it from my shell and reset ssh-agent back to an initial state with no loaded keys:

brew services stop ssh-askpass
launchctl unsetenv SSH_ASKPASS
launchctl unsetenv SUDO_ASKPASS
unset SSH_ASKPASS
unset SUDO_ASKPASS
killall ssh-agent

I then added the keys again with confirmation enabled:

# For SSH key in the Keychain:
ssh-add -cs /usr/lib/ssh-keychain.dylib
# If you were using a private key stored on disk, you would do:
ssh-add -c

I tested putting both your tiny script and this repository's version of ssh-askpass at /usr/X11R6/bin/ssh-askpass (ie: without reattach-to-user-namespace), and they both ran without issues on both Sonoma 14.5 and 14.6.

Even adding a symlink to a Homebrew-installed version of ssh-askpass worked fine.

Looking at the source of Apple's version of SSH, it will try to use /usr/X11R6/bin/ssh-askpass if the SSH_ASKPASS environment variable is unset:

https://github.com/apple-oss-distributions/OpenSSH/blob/9b6202341ee10b42e7391229ad5c0f2eb8aea8af/openssh/readpass.c#L174-L178

This would suggest that the only trigger for the issue is that environment variables from launchctl setenv are no longer propagated to system daemons, which means ssh-askpass must be at /usr/X11R6/bin/ssh-askpass.

While I don't doubt that reattach-to-user-namespace is useful in some contexts, it seems that at least for GUI sessions, ssh-agent runs in a namespace where child processes can create working dialogs. With this version of ssh-askpass, you can check the parent process for osascript in Activity Monitor and confirm where it's spawned from.

(This post was edited multiple times to clarify things.)

[micolous]

A step-by-step, minimum-viable workaround for Sonoma 14.6 with macOS' default, out-of-the-box ssh-agent setup, based on @sprig's comment and my previous comment:

1. Install ssh-askpass, but don't install ssh-askpass.plist to ~/Library/LaunchAgents/ or register it with launchctl (unless you want SUDO_ASKPASS).

   If the LaunchAgent is already installed, it technically doesn't matter – it just has no effect on ssh-agent on Sonoma 14.6.

2. Install XQuartz. This can be downloaded from the project's website or can be installed with brew install xquartz.

   You don't have to actually use or run XQuartz – this is just so that the DISPLAY environment variable is set for system processes like ssh-agent.

3. Check that the DISPLAY environment variable would be set for ssh-agent (under "inherited environment"):

   launchctl print gui/$UID/com.openssh.ssh-agent

   If DISPLAY isn't set, then XQuartz is not installed correctly.

4. Terminate any running ssh-agent processes, which may have old environment variables:

   killall -v ssh-agent

5. Create a symlink to where you installed ssh-askpass in /private/var/select/X11/bin (which is in turn symlinked from /usr/X11R6/bin):

   # Create /private/var/select/X11/bin if it doesn't already exist
   sudo mkdir -p /private/var/select/X11/bin
   
   # Link to where you installed ssh-askpass
   sudo ln -s /usr/local/bin/ssh-askpass /private/var/select/X11/bin/ssh-askpass

Then you can load your keys into your SSH agent as normal (ssh-add -c...)

This should also work on Sonoma 14.5 and earlier, if you want to prepare a system before upgrading to 14.6.

A work-around without XQuartz would be to have ssh-askpass.plist pretend to be an X server with using SecureSocketWithKey. But this would break any application that actually tried to use that X server (probably in surprising or difficult to debug ways), and conflict with XQuartz.

Edited (2024-08-06): fix typo: replace /var/private with /private/var

[D54]

An alternative workaround is to clone and shadow the original system launch-agent, which cannot be disabled or modified (due to SIP), and does not inherit global environment.

Make a copy:

cp /System/Library/LaunchAgents/com.openssh.ssh-agent.plist ~/Library/LaunchAgents/com.openssh.ssh-agent-my.plist

Change:

• the Label to e.g. com.openssh.ssh-agent-my
• and the SecureSocketWithKey to e.g. SSH_AUTH_SOCK_MY

Load the launch-agent

launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.openssh.ssh-agent-my.plist

Even if it's the same executable, it does inherit the global environment as it is not a system launch-agent. So it gets the $SSH_ASKPASS variable.

Restart your terminal app to acquire the fresh global env.

Now the $SSH_AUTH_SOCK_MY environment variable should contain a new launchctl listener socket-file.

To make it work in your terminal, you can override the original env variable in your shell config:

export SSH_AUTH_SOCK=$SSH_AUTH_SOCK_MY

To make it work with all apps, we have to override the socket-file itself:

ln -sf $SSH_AUTH_SOCK_MY $SSH_AUTH_SOCK

(Idea source: maxgoedjen/secretive

Finally, it's working just as before the OS upgrade.

[sprig]

@micolous - In my case ssh-askpass stopped working without using the reattach-to-user-namespace wrapper. Just putting the ssh-askpass script in the above location makes ssh fail without any dialog opening. EDIT: Actually you are correct - reattach-to-user-namespace is not required. ssh-askpass.plist is still useful if you ever try to ssh with a key that isn't loaded into ssh-agent (or using a password). I don't know regarding SUDO_ASKPASS - I never had any dialog popup when I use sudo, when this is set.

However, there's no mention of /var/private anywhere on my system so I'm not sure where you got the information regarding it being symlinked from /usr/X11R6/bin. In my case /usr/X11R6 is a symlink to /opt/x11, whereas /usr/X11 is linked to /private/var/select/X11, (/private/var, not /var/private!) which itself in turn is again linked to /opt/x11. I'm guessing that was a typo. Regardless, I'm not sure why you would need to create the directory if it already is a symlink.

[micolous]

I'm guessing that was a typo.

@sprig Yes, that was a typo, I've now fixed that, thanks. 😄

Regardless, I'm not sure why you would need to create the directory if it already is a symlink.

Note: I've set TZ=UTC there so everything has consistent timestamps, and that you can reproduce it yourself. 😄

On a system with Sonoma 14.5 on x86_64 with XQuartz installed via Homebrew, I see:

% TZ=UTC ls -l /usr
total 0
lrwxr-xr-x    1 root  wheel     25  7 May 07:01 X11 -> ../private/var/select/X11
lrwxr-xr-x    1 root  wheel     25  7 May 07:01 X11R6 -> ../private/var/select/X11
drwxr-xr-x  985 root  wheel  31520  7 May 07:01 bin
drwxr-xr-x   32 root  wheel   1024  7 May 07:01 lib
[snip]

On a system on Sonoma 14.6 on aarch64 which has never had XQuartz (or any other X) installed, I see:

% TZ=UTC ls -l /usr
total 0
lrwxr-xr-x    1 root  wheel     25 19 Jul 03:08 X11 -> ../private/var/select/X11
lrwxr-xr-x    1 root  wheel     25 19 Jul 03:08 X11R6 -> ../private/var/select/X11
drwxr-xr-x  985 root  wheel  31520 19 Jul 03:08 bin
drwxr-xr-x   32 root  wheel   1024 19 Jul 03:08 lib
[snip]

Out of the box, this is a broken symlink. This is from a machine with Sonoma 14.6 on aarch64 that doesn't have XQuartz or ssh-askpass installed, so I've never done my workaround:

% TZ=UTC ls -l /private/var/select
total 0
lrwxr-xr-x  1 root  wheel  9 19 Jul 03:08 sh -> /bin/bash
%

I also jumped into macOS recovery on a machine running Sonoma 14.6 on aarch64. I was able to check that it was actually running that version of Sonoma with uname -a, which reported a build timestamp of Fri Jul 5 18:01:46 PDT 2024 and running xnu-10063.141.1~2/RELEASE_ARM64_T8112.

There's no /usr/X11 or /usr/X11R6 symlinks in the recovery environment's root filesystem.

By default, the user data partition isn't mounted, and the (sealed) OS partition is mounted as read-only at /Volumes/Macintosh HD. The symlinks at /Volumes/Macintosh HD/usr/X11 and /Volumes/Macintosh HD/usr/X11R6 were present as before, and /Volumes/Macintosh HD/private was empty. /Volumes/Macintosh HD/Users was also empty.

All of the macOS systems I used have always had system integrity protection enabled.

I'll leave reproducing this on a fresh install of macOS as an exercise for the reader. 😄

To make it work with all apps, we have to override the socket-file itself:

ln -sf $SSH_AUTH_SOCK_MY $SSH_AUTH_SOCK

@D54 You'll have run this every time you log in.

GUI app compatibility is one of the draw-backs of not using Apple's provided ssh-agent config, which has been previously noted in a similar work-around proposed earlier this thread.

By comparison, the work-around I proposed works with everything that Apple's default ssh-agent config works with, and with no changes to your shell profile. 😄

[simmel]

Wow! What an active and amazing community we have! ❤️

I'll try to answer the unanswered questions and correct, as best as I can, a few things.
And sorry for the spam here because I'm gonna answer them in multiple comments as just one would just get too much information.

[simmel]

But it's still not running:

√ ~$ brew services info ssh-askpass
ssh-askpass (homebrew.mxcl.ssh-askpass)
Running: ✘
Loaded: ✔
Schedulable: ✘

I'm guessing this is because of #53

We don't actually have a process running we just use the service to set the SSH_ASKPASS environment variable.

When trying to start it:

√ ~$ brew services start ssh-askpass
Bootstrap failed: 5: Input/output error
Try re-running the command as root for richer errors.
Error: Failure while executing; `/bin/launchctl bootstrap gui/501 /Users/redacted/Library/LaunchAgents/homebrew.mxcl.ssh-askpass.plist` exited with 5.
?1 ~$ sudo brew services start ssh-askpass
Password:
Warning: Taking root:admin ownership of some ssh-askpass paths:
  /bin
  /bin/sh
  /opt/homebrew/opt/ssh-askpass
  /opt/homebrew/opt/ssh-askpass/bin
  /opt/homebrew/var/homebrew/linked/ssh-askpass
This will require manual removal of these paths using `sudo rm` on
brew upgrade/reinstall/uninstall.
Error: Operation not permitted @ apply2files - /bin

I've never seen this before and the output is not something we do. Uninstall and reinstall to get the correct version.

[simmel]

If your organisation depends on this working and has a support agreement with Apple, please point them at this bug. 😄

This is basically it. Apple released a new version and it had an unintended bug in it by removing environment variables from system agents that is set by launchctl setenv.

Why do we classify this as a bug?

$ man launchctl
[...]
     setenv key value
              Specify an environment variable to be set on all future processes launched by launchd in the caller's context.
[...]

AFAICT gui/$(id -u) is the callers context.

[simmel]

@micolous proposed work-around works.

Homebrew doesn't want us to write to /usr and I'm surprised that Apple lets us (but I guess it's because of XQuartz and things being put in /usr/local/) so there's not much this project can do right now.

[simmel]

The error we're seeing is because ssh-agent has no way to ask us for confirmation:

• if DISPLAY is set, SSH_ASKPASS is still unset, so it doesn't know what to do and reject it.

If I understand the logic correctly if DISPLAY is set and SSH_ASKPASS is unset and stdin is not a tty (i.e. we're running in the background, like ssh-agent is) then we use the default /usr/X11R6/bin/ssh-askpass.

This is also why you don't get a password prompt with ssh-add -c when you are running it in a shell because then stdin is a tty but if you run echo | ssh-add -c ssh-askpass will popup and let you enter the password.

[simmel]

@micolous and @D54 got me thinking of a very ugly idea... 😆

We could, and I just tried it - it works, in our launchd plist:

• Create a SecureSocketWithKey called SSH_ASKPASS which would inject it into every process even ssh-agent
• In our ProgramArguments script:
  • Overwrite the SSH_ASKPASS socket with a symlink to our ssh-askpass.

It's NOT pretty but it works. But since there are workarounds we'll wait a bit for Apple to fix it. We are not the only ones using launchctl setenv.

But if you have a support contract please remember to contact Apple about it!

[achal1012]

Hi are you folks also seeing ssh-agent being unable to respond to connect requests after the update until ssh-add is called?

[simmel]

Hi are you folks also seeing ssh-agent being unable to respond to connect requests after the update until ssh-add is called?

ssh-agent isn't started until something tries to use it via the socket AFAIK.

[zeyugao]

14.6.1 is released, is it fixed or it is intended by apple?

[micolous]

14.6.1 is released, is it fixed or it is intended by apple?

@zeyugao The issue is not fixed in Sonoma 14.6.1.

If you want to test if a version of macOS is affected (without installing ssh-askpass), you can try manually setting the SSH_ASKPASS environment variable (which is what ssh-askpass.plist does) and, then reading it back in the context of ssh-agent:

# Set SSH_ASKPASS to reject all requests, but only if it is unset.
[ -z "$(launchctl getenv SSH_ASKPASS)" ] && launchctl setenv SSH_ASKPASS /bin/false

# Check for SSH_ASKPASS from ssh-agent's environment
launchctl print gui/$UID/com.openssh.ssh-agent | grep SSH_ASKPASS

If the second command returns nothing, then that version of macOS is affected by the issue.

If the second command returns SSH_ASKPASS => and a path, then that version is not affected by the issue.

To explicitly unset SSH_ASKPASS in launchd, run:

launchctl unsetenv SSH_ASKPASS

[micolous]

I don't think launchd has changed between Sonoma 14.6.0 and 14.6.1.

On a Sonoma 14.5 x86_64 machine, I see:

% launchctl version
Darwin Bootstrapper Version 7.0.0: Thu Apr 25 21:30:47 PDT 2024; root:libxpc_executables-2748.121.1~1/launchd/RELEASE_X86_64

On a Sonoma 14.6.0 and 14.6.1 aarch64 machine, I see:

% launchctl version
Darwin Bootstrapper Version 7.0.0: Sat Jun 29 04:48:38 PDT 2024; root:libxpc_executables-2748.140.10~372/launchd/RELEASE_ARM64E

Interestingly, the security release notes for Sonoma 14.6 says:

libxpc

Available for: macOS Sonoma

Impact: An app may be able to bypass Privacy preferences

Description: A permissions issue was addressed with additional restrictions.

CVE-2024-40805

There are similar patches for CVE-2024-40805 in iOS, iPadOS, tvOS and watchOS; and there's no other fixes for libxpc in those versions.

There aren't any publicly-available details around what that bug actually is, which is normal for an issue that's either privately disclosed or discovered internally at Apple. My outsider, armchair theory is:

• I see how the words "additional restrictions" could describe the inability to set environment variables on macOS-provided LaunchAgents.

• I have no idea whether the bug is in ssh-agent itself or some other macOS daemon... but locking it down in this way feels like a scorched-earth approach.

• That suggests to me there's some class of issue which affects multiple system LaunchAgents.

There have been narrower environment variable restrictions in the past: the previous (open source) version of launchd prevented setting DYLD_* environment variables, which could be otherwise used to inject code into other processes.

I still feel that this issue may be an unintended side-effect: Apple's default ssh-agent service will still run your own ssh-askpass, as long as it's at /usr/X11R6/bin/ssh-askpass. This raises the bar for misusing that feature to "you need root". However, the harshest restrictions on macOS come from System Integrity Protection, which they didn't use here, even though they could have.

[MichaelRoosz]

here is the workaround I am using:
https://github.com/MichaelRoosz/homebrew-ssh/blob/main/etc/install-libsk-libfido2-v1.1.5.zsh#L24

this stops any other ssh-agent, then uses its own socket env var to start the agent with custom config, then symlinks the original ssh socket to the custom one

[simmel]

@MichaelRoosz Won't launchd use socket activation and start a new ssh-agent and inherit SSH_AUTH_SOCK with a new path to all GUI applications?

[MichaelRoosz]

@simmel due to the symlink, socket activation will start the custom ssh-agent

[micolous]

The workaround is actually pretty similar to this workaround posted earlier, but automates the socket symlinking hack at the end.

All of the launchctl setenv steps of that script shouldn't be needed:

• They don't actually apply to the ssh-agent launched at the end of the script - the EnvironmentVariables key does.
• ssh-agent is responsible for calling ssh-askpass and deciding when to call it. Your normal ssh client should just connect to SSH_AUTH_SOCK.

In any case, if a future macOS version starts protecting launchd's sockets (because this seems like a loophole that which impacts the integrity of the system), then this and @simmel's tongue-in-cheek workaround would break.

Launching ssh-agent yourself is only really needed for U2F/FIDO2-based keys over SSH, because that needs extra libraries and environment variables to make it work. If you're using PKCS#11/PIV or even a key file on disk, you don't need that. 😄

[MichaelRoosz]

@micolous the setenv (at least SSH_SK_PROVIDER) are needed to make the other ssh tools pickup the FIDO2 provider lib. they are there for a reason. SSH_ASKPASS was needed to make ssh work when used by ansible, if I remember correctly, but it surely does not hurt to set it.

to be honest, it is a shame that Apple does not support setting these configs in an easy way without needing to use those workarounds. I really hope they properly implement a way to configure the ssh-agent before they mess it up even more.

[lepus2589]

From the comments and suggestions here, I tried to use the workaround masking the system ssh_agent (#54 (comment)). After some tinkering, I found a solution with automatic linking of the system ssh-agent socket to the user ssh-agent socket, which works well for me and is non-invasive and easily reversible.

I compiled my solution into a gist (https://gist.github.com/lepus2589/5235d78172e529b2d076d2c0e9670b27). Feel free to test it and to give feedback!

[schiffy91]

I made a self-contained installer / uninstaller here. The script is aware of the 1Password location – because that's what I use – but if there's a similarly pre-defined socket path for AskPass, just add it here before running the script.

# List of known alternative socket paths
KNOWN_SOCKETS=(
    "$HOME/Library/Group Containers/2BUA8C4S2C.com.1password/t/agent.sock"  # 1Password
    "$HOME/.gnupg/S.gpg-agent.ssh"  # GPG Agent
)