Sonoma 14.6 breaks ssh-askpass

[lukaskuzmiak]

Just after update to Sonoma 14.6 today I get:

sign_and_send_pubkey: signing failed for ED25519 "/Users/lukash/.ssh/<redacted>" from agent: agent refused operation

Same for my co-worker, it definitely worked ok on 14.5 just before this update (hours ago).

[dataviruset]

Seems the service can't start:

√ ~$ brew services restart theseal/ssh-askpass/ssh-askpass
Stopping `ssh-askpass`... (might take a while)
==> Successfully stopped `ssh-askpass` (label: homebrew.mxcl.ssh-askpass)
==> Successfully started `ssh-askpass` (label: homebrew.mxcl.ssh-askpass)

But it's still not running:

√ ~$ brew services info ssh-askpass
ssh-askpass (homebrew.mxcl.ssh-askpass)
Running: ✘
Loaded: ✔
Schedulable: ✘

When trying to start it:

√ ~$ brew services start ssh-askpass
Bootstrap failed: 5: Input/output error
Try re-running the command as root for richer errors.
Error: Failure while executing; `/bin/launchctl bootstrap gui/501 /Users/redacted/Library/LaunchAgents/homebrew.mxcl.ssh-askpass.plist` exited with 5.
?1 ~$ sudo brew services start ssh-askpass
Password:
Warning: Taking root:admin ownership of some ssh-askpass paths:
  /bin
  /bin/sh
  /opt/homebrew/opt/ssh-askpass
  /opt/homebrew/opt/ssh-askpass/bin
  /opt/homebrew/var/homebrew/linked/ssh-askpass
This will require manual removal of these paths using `sudo rm` on
brew upgrade/reinstall/uninstall.
Error: Operation not permitted @ apply2files - /bin

I tried deleting the /opt/homebrew folders mentioned and reinstalling ssh-askpass but it didn't help. I don't wanna try what happens if I delete /bin or /bin/sh... 😅
The owner of /bin is root:wheel these days and not root:admin.

[zeyugao]

@dataviruset You can directly set SSH_ASKPASS and SUDO_ASKPASS to test it without the services. It is due to the system I think, not the services

export SSH_ASKPASS=/opt/homebrew/opt/touch-auth/bin/touch-auth  # Change it
unset SSH_AUTH_SOCK
echo $SSH_AUTH_SOCK
eval $(ssh-agent -s)
echo $SSH_AUTH_SOCK

[dataviruset]

@dataviruset You can directly set SSH_ASKPASS and SUDO_ASKPASS to test it without the services. It is due to the system I think, not the services

export SSH_ASKPASS=/opt/homebrew/opt/touch-auth/bin/touch-auth  # Change it
unset SSH_AUTH_SOCK
echo $SSH_AUTH_SOCK
eval $(ssh-agent -s)
echo $SSH_AUTH_SOCK

Using my original SSH_ASKPASS value (it's set to /opt/homebrew/opt/ssh-askpass/bin/ssh-askpass) and then running these commands worked for me:

unset SSH_AUTH_SOCK
eval $(ssh-agent -s)
ssh-add -c ~/.ssh/my_key
ssh test-machine.example.com

Interestingly the SSH_ASKPASS environment variable is already set. Probably the brew package set that up for me. But the built-in Mac OS SSH agent seems to not work anymore as of Mac OS 14.6...

[EXHades]

I added keys to the ssh-agent using keepassxc.
just deselect Require user confirmation when this key is used option, locked keepassxc and then unlocked it again, Now ssh -T example.com works fine, But SSH_ASKPASS still doesn't work.

keepassxreboot/keepassxc#9955 (comment)

---

macOS 14.6 upgrade break ssh-agent SSH_AGENT_CONSTRAIN_CONFIRM support?

[micolous]

If your organisation depends on this working and has a support agreement with Apple, please point them at this bug. 😄

I dug in to this a bit today, it looks like Sonoma 14.6 has changed the environment variable inheritance rules for system-provided user daemons (/System/Library/LaunchAgents); so that environment variables set with launchctl setenv no longer apply.

This means that when macOS starts ssh-agent through launchctl (to handle the default SSH_AUTH_SOCK on demand), it no longer picks up SSH_ASKPASS and DISPLAY from this ssh-agent's launchd plist.

The work-around @zeyugao proposed side-steps all of this by not using macOS' ssh-agent config – but this only works in a per-shell context. Similarly, if you replaced ssh with the one from Homebrew and put it first in your path; or used a different environment variable to SSH_AUTH_SOCK to point at a custom ssh-agent, you'd also work around this... but that also kinda smells.

On a system with Sonoma 14.5 with xquartz installed, I see:

% launchctl print gui/501/com.openssh.ssh-agent
gui/501/com.openssh.ssh-agent = {
	active count = 1
	path = /System/Library/LaunchAgents/com.openssh.ssh-agent.plist
	type = LaunchAgent
	state = running

	program = /usr/bin/ssh-agent
	arguments = {
		/usr/bin/ssh-agent
		-l
	}

	inherited environment = {
		DISPLAY => /private/tmp/com.apple.launchd.XXXX/org.macosforge.xquartz:0
		SUDO_ASKPASS => /usr/local/opt/ssh-askpass/bin/ssh-askpass
		SSH_ASKPASS => /usr/local/opt/ssh-askpass/bin/ssh-askpass
		SSH_AUTH_SOCK => /private/tmp/com.apple.launchd.XXXX/Listeners
	}

	default environment = {
		PATH => /usr/bin:/bin:/usr/sbin:/sbin
	}

	environment = {
		MallocSpaceEfficient => 1
		XPC_SERVICE_NAME => com.openssh.ssh-agent
	}

On a system with Sonoma 14.6 without xquartz, I see:

% launchctl print gui/501/com.openssh.ssh-agent
gui/501/com.openssh.ssh-agent = {
	active count = 1
	path = /System/Library/LaunchAgents/com.openssh.ssh-agent.plist
	type = LaunchAgent
	state = running

	program = /usr/bin/ssh-agent
	arguments = {
		/usr/bin/ssh-agent
		-l
	}

	inherited environment = {
		SSH_AUTH_SOCK => /private/tmp/com.apple.launchd.XXXX/Listeners
	}

	default environment = {
		PATH => /usr/bin:/bin:/usr/sbin:/sbin
	}

	environment = {
		MallocSpaceEfficient => 1
		XPC_SERVICE_NAME => com.openssh.ssh-agent
	}

With Sonoma 14.6 with xquartz installed, I see:

% launchctl print gui/501/com.openssh.ssh-agent
gui/501/com.openssh.ssh-agent = {
	active count = 1
	path = /System/Library/LaunchAgents/com.openssh.ssh-agent.plist
	type = LaunchAgent
	state = running

	program = /usr/bin/ssh-agent
	arguments = {
		/usr/bin/ssh-agent
		-l
	}

	inherited environment = {
		DISPLAY => /private/tmp/com.apple.launchd.XXXX/org.xquartz:0
		SSH_AUTH_SOCK => /private/tmp/com.apple.launchd.XXXX/Listeners
	}

	default environment = {
		PATH => /usr/bin:/bin:/usr/sbin:/sbin
	}

	environment = {
		MallocSpaceEfficient => 1
		XPC_SERVICE_NAME => com.openssh.ssh-agent
	}

Other macOS system daemons (eg: com.apple.weatherd) are impacted in the same way – so this is not specific to ssh-agent or the way ssh-askpass works.

User-installed software in /Library/LaunchAgents and ~/Library/LaunchAgents don't seem to be affected by this change.

The fact that xquartz (a non-system application) can still set the DISPLAY environment variable for ssh-agent suggests that it should be possible for this ssh-askpass to set environment variables in some way, but from what I'm reading of the launchd.plist man page, it won't be elegant.

I suspect that this was part of some other security / platform hardening change, but I'm unable to find any specific release notes about this.

It's something that Apple will need to fix.

[zeyugao]

@micolous Have you succeeded in setting SSH_ASKPASS to workaround this manually? I just propose a method to bypass the usage of services as @dataviruset says the services failed to start.

I manually set the SSH_ASKPASS and started a ssh-agent, but I still got agent refused operation

[micolous]

@micolous Have you succeeded in setting SSH_ASKPASS to workaround this manually?

No.

What matters is the environment that ssh-agent runs in, which when it's spawned by launchd, ignores launchctl setenv on macOS Sonoma 14.6.

The error we're seeing is because ssh-agent has no way to ask us for confirmation:

• if DISPLAY is set, SSH_ASKPASS is still unset, so it doesn't know what to do and reject it.
• if DISPLAY is unset, then it'll ask on stdin... but that's not actually connected to anything so it'll reject it.

The work-around I've got for now is to disable confirmation, but I've set up my SSH keys using /usr/lib/ssh-keychain.dylib to access my keys over PKCS#11 / smartcard auth, and the smart card (Yubikey with PIV applet) is configured to require physical confirmation for signing operations.

It just means now I don't get any prompt on screen, just a flashing light on the side of my computer.

If you're using ssh-askpass to confirm access to a key stored on disk, then disabling confirmation has a pretty significant security drawback 😓

[lukaskuzmiak]

I worked around it by running a user-land instance of ssh-agent with proper environment variables set, then overwriting SSH_AUTH_SOCK in .zprofile and on top of that setting IdentityAgent in ~/.ssh/config to the path of the user-land ssh-agent.

Some GUI apps might not pick that up, but so far without issues. Other than it being annoying to setup.

For anyone interested here are some resources that helped me:

• https://gist.github.com/vancluever/de1c3985c8f9e2a3c4bdc42a057e075e
• 1Password uses a similar approach as described above and has a nice client compatibility table here: https://developer.1password.com/docs/ssh/agent

Given the fact Apple likely broke this for good I think that will be the way forward.

[micolous]

Yeah, your setup and 1Password's replace the default Apple-provided SSH agent setup (partially or entirely). There are usability edge cases to that, as you've already noted. If you go further (as 1Password do, and has been suggested elsewhere) and replace ssh-agent entirely, you also miss out on security hardening features only available in Apple's SSH binaries.

I acknowledge that there are tradeoffs, like that those hardening features can also prevent you from using some more "obscure" functionality (like FIDO2 resident keys and PIV with ECDSA keys).

For what it's worth, there are large corporates and government agencies with macOS workstations and a custom ssh-askpass implementation (to support PKCS#11 auth with hardware keystores), but their corporate IT teams have probably noticed this in testing and are holding back the update from their fleets.

At this point, I'm quietly optimistic that it won't remain broken forever... it just needs to get in front of the right set of eyes. 😄

[sprig]

I think the issue has been (at lease partially) misdiagnosed by @micolous;

(FWIW I use a different askpass program but I think the issue is similar)
First, I've noticed that if I don't initially load keys into ssh-agent then I do get asked for a password by ssh-askpass. However if I do load keys ssh complains that agent refused operation.

As an experiment, I tried unsetting the SSH_ASKPASS variable in the terminal but setting SSH_ASKPASS_REQUIRE=force, and running ssh -vv host:
I notice lines like the following:

ssh_askpass: exec(/usr/X11R6/bin/ssh-askpass): No such file or directory

So I add a tiny script to the above location:

#!/bin/sh
echo "$@" > ~/askpass
date >> ~/askpass
env >> ~/askpass

I notice that when keys are not loaded, I get output as follows:

$ cat ~/askpass
Enter passphrase for key '/Users/user/.ssh/id_rsa': 
Sun Aug  4 11:07:55 PDT 2024
TERM_PROGRAM=Apple_Terminal
LC_MONETARY=en_US.UTF-8
QT_STYLE_OVERRIDE=Fusion
LUA_BINDIR=<redacted>
TERM=xterm-256color
SHELL=/bin/zsh
TMPDIR=/var/folders/vx/wq1n25gs5t54v4sp3_6yj3c00000gn/T/
TERM_PROGRAM_VERSION=453
LC_NUMERIC=en_US.UTF-8
CURL_SSL_BACKEND=secure-transport
TERM_SESSION_ID=516E43F7-2240-481C-A308-A13AD18FE7B4
LC_ALL=en_US.UTF-8
CDPATH=.:<redacted>
USER=user
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.LVjkwSsCS6/Listeners
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
PAGER=less
PATH=<long list of paths>
LC_MESSAGES=en_US.UTF-8
LUA_DIR=<redacted>
LaunchInstanceID=9A548B16-A27C-4770-B569-8A81CC12830F
QT_QPA_PLATFORMTHEME=gtk2
WORDCHARS=
__CFBundleIdentifier=com.apple.Terminal
LC_COLLATE=en_US.UTF-8
PROFILE_RAN=Sun Aug  4 09:49:08 PDT 2024
PWD=/Users/user
EDITOR=vim
LANG=en_US.UTF-8
LUA_PATH=<redacted>
XPC_FLAGS=0x0
SSH_ASKPASS_REQUIRE=force
XPC_SERVICE_NAME=0
SHLVL=2
HOME=/Users/user
LOGNAME=user
PYTHONPATH=<redacted>
LESS=-F -g -i -M -R -S -w -z-4
LC_CTYPE=en_US.UTF-8
BROWSER=open
DISPLAY=/private/tmp/com.apple.launchd.NVQ3vVyfbz/org.xquartz:0
SECURITYSESSIONID=186ab
LC_TIME=en_US.UTF-8
_=/usr/bin/env

In particular, the PROFILE_RAN variable above is set to date whenever my ~/.profile is sourced, one can notice that it is significantly different from when the askpass script was run. In fact, it is identical to what I have in that particular terminal session:

$ echo $PROFILE_RAN
Sun Aug  4 09:49:08 PDT 2024

i.e. the script runs in the same context as the ssh command, likely invoked directly by ssh.

Whereas if keys are loaded I get similar to the following:

$ cat ~/askpass 
Allow use of key user@host?
Key fingerprint SHA256:<redacted>.
SHELL=/bin/zsh
SSH_ASKPASS_PROMPT=confirm
TMPDIR=/var/folders/vx/wq1n25gs5t54v4sp3_6yj3c00000gn/T/
MallocSpaceEfficient=1
USER=user
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.LVjkwSsCS6/Listeners
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
PATH=/usr/bin:/bin:/usr/sbin:/sbin
PWD=/
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
SHLVL=1
HOME=/Users/user
LOGNAME=user
DISPLAY=/private/tmp/com.apple.launchd.NVQ3vVyfbz/org.xquartz:0
_=/usr/bin/env

which is a fairly vanilla environment, likely invoked by ssh-agent. Furthermore, since the script above returns 0 (success), the authentication is successful.

The conclusion is that rather than not recognizing SSH_ASKPASS environment variable which is read by ssh and not by the agent, ssh-agent fails to create a window when invoked by ssh-agent. Likely similar issues to what prompted reattach-to-user-namespace in https://github.com/ChrisJohnsen/tmux-MacOSX-pasteboard. I don't know if that tool still works but if it doesn't this suggests that another viable workaround; Separate this into a client/server setup where the server would run in the user's session and be able to open windows whereas the client would simply forward authentication requests from the agent.

[sprig]

update: I can confirm that reattach-to-user-namespace still works and using a wrapper script in /usr/X11R6/bin/ssh-askpass such as

#!/bin/sh
exec ~/local/bin/reattach-to-user-namespace ~/local/ssh-askpass/ssh-askpass "$@"

where programs are at the respective locations above opens the confirmation dialog and proceeds to successfully authenticate. Presumably pointing SSH_ASKPASS at the wrapper would work as well.

EDIT:
Upon further testing it seems that the issue @micolous identified is at play here as well - the environment does not get set. However, the path above is the default location where ssh-askpass is searched for and so putting the wrapper there still works as a workaround.

[micolous]

I can confirm that reattach-to-user-namespace still works and using a wrapper script in /usr/X11R6/bin/ssh-askpass

I tried your work-around on Sonoma 14.5 and 14.6 with XQuartz installed (so DISPLAY is set by SecureSocketWithKey, rather than launchctl setenv).

I removed the ssh-askpass.plist LaunchAgent and cleared out any traces from it from my shell and reset ssh-agent back to an initial state with no loaded keys:

brew services stop ssh-askpass
launchctl unsetenv SSH_ASKPASS
launchctl unsetenv SUDO_ASKPASS
unset SSH_ASKPASS
unset SUDO_ASKPASS
killall ssh-agent

I then added the keys again with confirmation enabled:

# For SSH key in the Keychain:
ssh-add -cs /usr/lib/ssh-keychain.dylib
# If you were using a private key stored on disk, you would do:
ssh-add -c

I tested putting both your tiny script and this repository's version of ssh-askpass at /usr/X11R6/bin/ssh-askpass (ie: without reattach-to-user-namespace), and they both ran without issues on both Sonoma 14.5 and 14.6.

Even adding a symlink to a Homebrew-installed version of ssh-askpass worked fine.

Looking at the source of Apple's version of SSH, it will try to use /usr/X11R6/bin/ssh-askpass if the SSH_ASKPASS environment variable is unset:

https://github.com/apple-oss-distributions/OpenSSH/blob/9b6202341ee10b42e7391229ad5c0f2eb8aea8af/openssh/readpass.c#L174-L178

This would suggest that the only trigger for the issue is that environment variables from launchctl setenv are no longer propagated to system daemons, which means ssh-askpass must be at /usr/X11R6/bin/ssh-askpass.

While I don't doubt that reattach-to-user-namespace is useful in some contexts, it seems that at least for GUI sessions, ssh-agent runs in a namespace where child processes can create working dialogs. With this version of ssh-askpass, you can check the parent process for osascript in Activity Monitor and confirm where it's spawned from.

(This post was edited multiple times to clarify things.)

[micolous]

A step-by-step, minimum-viable workaround for Sonoma 14.6 and later with macOS' default, out-of-the-box ssh-agent setup, based on @sprig's comment and my previous comment:

1. Remove any LaunchAgents which attempt to replace Apple's ssh-agent, manipulate SSH_AUTH_SOCK or the file it points to.

2. Install ssh-askpass, but don't install ssh-askpass.plist to ~/Library/LaunchAgents/ or register it with launchctl (unless you want SUDO_ASKPASS).

   If the LaunchAgent is already installed (eg: via Homebrew), it technically doesn't matter – it just has no effect on ssh-agent on Sonoma 14.6 and later.

3. Install XQuartz. This can be downloaded from the project's website or can be installed with brew install xquartz.

   Note: You don't have to actually use or run XQuartz – this is just so that the DISPLAY environment variable is set using SecureSocketWithKey, which works for system LaunchAgents like Apple's ssh-agent.

4. Log out of macOS and log in again. This makes Apple's ssh-agent pick up XQuartz' new environment variables from launchd.

5. Check that the DISPLAY environment variable would be set for Apple's ssh-agent (under "inherited environment"):

   launchctl print gui/$UID/com.openssh.ssh-agent

   If DISPLAY isn't set, then XQuartz is not installed correctly (check your start-up items, and log out and log in again).

6. Terminate any running ssh-agent processes, which may have old environment variables:

   killall -v ssh-agent

7. Create a symlink to where you installed ssh-askpass in /private/var/select/X11/bin (which is in turn symlinked from /usr/X11R6/bin):

   # Create /private/var/select/X11/bin if it doesn't already exist
   sudo mkdir -p /private/var/select/X11/bin
   
   # Link to where you installed ssh-askpass at /private/var/select/X11/bin/ssh-askpass, eg:
   # If you installed ssh-askpass with Homebrew on Apple Silicon:
   sudo ln -s /opt/homebrew/bin/ssh-askpass /private/var/select/X11/bin/ssh-askpass
   # If you installed ssh-askpass with Homebrew on Intel:
   sudo ln -s /usr/local/bin/ssh-askpass /private/var/select/X11/bin/ssh-askpass

Then you can load your keys into your SSH agent as normal (ssh-add -c...)

This should also work on Sonoma 14.5 and earlier, if you want to prepare a system before upgrading to 14.6.

A work-around without XQuartz would be to have ssh-askpass.plist pretend to be an X server with using SecureSocketWithKey (like XQuartz's LaunchAgent does). But this would break any application that actually tried to use that X server (probably in surprising or difficult to debug ways), and conflict with XQuartz.

Edited (2024-08-06): Fix typo: replace /var/private with /private/var

Edited (2025-02-18): Added that you may need to log out and log in again after installing XQuartz for the DISPLAY environment variable to show up properly (tested with Homebrew package on Sequoia 15.3.1). More explicitly noted that setting environment variables with SecureSocketWithKey is what makes this all work. Noted Apple Silicon Homebrew install location for ssh-askpass.

[D54]

An alternative workaround is to clone and shadow the original system launch-agent, which cannot be disabled or modified (due to SIP), and does not inherit global environment.

Make a copy:

cp /System/Library/LaunchAgents/com.openssh.ssh-agent.plist ~/Library/LaunchAgents/com.openssh.ssh-agent-my.plist

Change:

• the Label to e.g. com.openssh.ssh-agent-my
• and the SecureSocketWithKey to e.g. SSH_AUTH_SOCK_MY

Load the launch-agent

launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.openssh.ssh-agent-my.plist

Even if it's the same executable, it does inherit the global environment as it is not a system launch-agent. So it gets the $SSH_ASKPASS variable.

Restart your terminal app to acquire the fresh global env.

Now the $SSH_AUTH_SOCK_MY environment variable should contain a new launchctl listener socket-file.

To make it work in your terminal, you can override the original env variable in your shell config:

export SSH_AUTH_SOCK=$SSH_AUTH_SOCK_MY

To make it work with all apps, we have to override the socket-file itself:

ln -sf $SSH_AUTH_SOCK_MY $SSH_AUTH_SOCK

(Idea source: maxgoedjen/secretive

Finally, it's working just as before the OS upgrade.

[sprig]

@micolous - In my case ssh-askpass stopped working without using the reattach-to-user-namespace wrapper. Just putting the ssh-askpass script in the above location makes ssh fail without any dialog opening. EDIT: Actually you are correct - reattach-to-user-namespace is not required. ssh-askpass.plist is still useful if you ever try to ssh with a key that isn't loaded into ssh-agent (or using a password). I don't know regarding SUDO_ASKPASS - I never had any dialog popup when I use sudo, when this is set.

However, there's no mention of /var/private anywhere on my system so I'm not sure where you got the information regarding it being symlinked from /usr/X11R6/bin. In my case /usr/X11R6 is a symlink to /opt/x11, whereas /usr/X11 is linked to /private/var/select/X11, (/private/var, not /var/private!) which itself in turn is again linked to /opt/x11. I'm guessing that was a typo. Regardless, I'm not sure why you would need to create the directory if it already is a symlink.

[micolous]

I'm guessing that was a typo.

@sprig Yes, that was a typo, I've now fixed that, thanks. 😄

Regardless, I'm not sure why you would need to create the directory if it already is a symlink.

Note: I've set TZ=UTC there so everything has consistent timestamps, and that you can reproduce it yourself. 😄

On a system with Sonoma 14.5 on x86_64 with XQuartz installed via Homebrew, I see:

% TZ=UTC ls -l /usr
total 0
lrwxr-xr-x    1 root  wheel     25  7 May 07:01 X11 -> ../private/var/select/X11
lrwxr-xr-x    1 root  wheel     25  7 May 07:01 X11R6 -> ../private/var/select/X11
drwxr-xr-x  985 root  wheel  31520  7 May 07:01 bin
drwxr-xr-x   32 root  wheel   1024  7 May 07:01 lib
[snip]

On a system on Sonoma 14.6 on aarch64 which has never had XQuartz (or any other X) installed, I see:

% TZ=UTC ls -l /usr
total 0
lrwxr-xr-x    1 root  wheel     25 19 Jul 03:08 X11 -> ../private/var/select/X11
lrwxr-xr-x    1 root  wheel     25 19 Jul 03:08 X11R6 -> ../private/var/select/X11
drwxr-xr-x  985 root  wheel  31520 19 Jul 03:08 bin
drwxr-xr-x   32 root  wheel   1024 19 Jul 03:08 lib
[snip]

Out of the box, this is a broken symlink. This is from a machine with Sonoma 14.6 on aarch64 that doesn't have XQuartz or ssh-askpass installed, so I've never done my workaround:

% TZ=UTC ls -l /private/var/select
total 0
lrwxr-xr-x  1 root  wheel  9 19 Jul 03:08 sh -> /bin/bash
%

I also jumped into macOS recovery on a machine running Sonoma 14.6 on aarch64. I was able to check that it was actually running that version of Sonoma with uname -a, which reported a build timestamp of Fri Jul 5 18:01:46 PDT 2024 and running xnu-10063.141.1~2/RELEASE_ARM64_T8112.

There's no /usr/X11 or /usr/X11R6 symlinks in the recovery environment's root filesystem.

By default, the user data partition isn't mounted, and the (sealed) OS partition is mounted as read-only at /Volumes/Macintosh HD. The symlinks at /Volumes/Macintosh HD/usr/X11 and /Volumes/Macintosh HD/usr/X11R6 were present as before, and /Volumes/Macintosh HD/private was empty. /Volumes/Macintosh HD/Users was also empty.

All of the macOS systems I used have always had system integrity protection enabled.

I'll leave reproducing this on a fresh install of macOS as an exercise for the reader. 😄

To make it work with all apps, we have to override the socket-file itself:

ln -sf $SSH_AUTH_SOCK_MY $SSH_AUTH_SOCK

@D54 You'll have run this every time you log in.

GUI app compatibility is one of the draw-backs of not using Apple's provided ssh-agent config, which has been previously noted in a similar work-around proposed earlier this thread.

By comparison, the work-around I proposed works with everything that Apple's default ssh-agent config works with, and with no changes to your shell profile. 😄

[simmel]

Wow! What an active and amazing community we have! ❤️

I'll try to answer the unanswered questions and correct, as best as I can, a few things.
And sorry for the spam here because I'm gonna answer them in multiple comments as just one would just get too much information.

[simmel]

But it's still not running:

√ ~$ brew services info ssh-askpass
ssh-askpass (homebrew.mxcl.ssh-askpass)
Running: ✘
Loaded: ✔
Schedulable: ✘

I'm guessing this is because of #53

We don't actually have a process running we just use the service to set the SSH_ASKPASS environment variable.

When trying to start it:

√ ~$ brew services start ssh-askpass
Bootstrap failed: 5: Input/output error
Try re-running the command as root for richer errors.
Error: Failure while executing; `/bin/launchctl bootstrap gui/501 /Users/redacted/Library/LaunchAgents/homebrew.mxcl.ssh-askpass.plist` exited with 5.
?1 ~$ sudo brew services start ssh-askpass
Password:
Warning: Taking root:admin ownership of some ssh-askpass paths:
  /bin
  /bin/sh
  /opt/homebrew/opt/ssh-askpass
  /opt/homebrew/opt/ssh-askpass/bin
  /opt/homebrew/var/homebrew/linked/ssh-askpass
This will require manual removal of these paths using `sudo rm` on
brew upgrade/reinstall/uninstall.
Error: Operation not permitted @ apply2files - /bin

I've never seen this before and the output is not something we do. Uninstall and reinstall to get the correct version.

[simmel]

If your organisation depends on this working and has a support agreement with Apple, please point them at this bug. 😄

This is basically it. Apple released a new version and it had an unintended bug in it by removing environment variables from system agents that is set by launchctl setenv.

Why do we classify this as a bug?

$ man launchctl
[...]
     setenv key value
              Specify an environment variable to be set on all future processes launched by launchd in the caller's context.
[...]

AFAICT gui/$(id -u) is the callers context.

[simmel]

@micolous proposed work-around works.

Homebrew doesn't want us to write to /usr and I'm surprised that Apple lets us (but I guess it's because of XQuartz and things being put in /usr/local/) so there's not much this project can do right now.

[simmel]

The error we're seeing is because ssh-agent has no way to ask us for confirmation:

• if DISPLAY is set, SSH_ASKPASS is still unset, so it doesn't know what to do and reject it.

If I understand the logic correctly if DISPLAY is set and SSH_ASKPASS is unset and stdin is not a tty (i.e. we're running in the background, like ssh-agent is) then we use the default /usr/X11R6/bin/ssh-askpass.

This is also why you don't get a password prompt with ssh-add -c when you are running it in a shell because then stdin is a tty but if you run echo | ssh-add -c ssh-askpass will popup and let you enter the password.

[simmel]

@micolous and @D54 got me thinking of a very ugly idea... 😆

We could, and I just tried it - it works, in our launchd plist:

• Create a SecureSocketWithKey called SSH_ASKPASS which would inject it into every process even ssh-agent
• In our ProgramArguments script:
  • Overwrite the SSH_ASKPASS socket with a symlink to our ssh-askpass.

It's NOT pretty but it works. But since there are workarounds we'll wait a bit for Apple to fix it. We are not the only ones using launchctl setenv.

But if you have a support contract please remember to contact Apple about it!

[achal1012]

Hi are you folks also seeing ssh-agent being unable to respond to connect requests after the update until ssh-add is called?

[simmel]

Hi are you folks also seeing ssh-agent being unable to respond to connect requests after the update until ssh-add is called?

ssh-agent isn't started until something tries to use it via the socket AFAIK.

[zeyugao]

14.6.1 is released, is it fixed or it is intended by apple?

[micolous]

14.6.1 is released, is it fixed or it is intended by apple?

@zeyugao The issue is not fixed in Sonoma 14.6.1.

If you want to test if a version of macOS is affected (without installing ssh-askpass), you can try manually setting the SSH_ASKPASS environment variable (which is what ssh-askpass.plist does) and, then reading it back in the context of ssh-agent:

# Set SSH_ASKPASS to reject all requests, but only if it is unset.
[ -z "$(launchctl getenv SSH_ASKPASS)" ] && launchctl setenv SSH_ASKPASS /bin/false

# Check for SSH_ASKPASS from ssh-agent's environment
launchctl print gui/$UID/com.openssh.ssh-agent | grep SSH_ASKPASS

If the second command returns nothing, then that version of macOS is affected by the issue.

If the second command returns SSH_ASKPASS => and a path, then that version is not affected by the issue.

To explicitly unset SSH_ASKPASS in launchd, run:

launchctl unsetenv SSH_ASKPASS

[micolous]

I don't think launchd has changed between Sonoma 14.6.0 and 14.6.1.

On a Sonoma 14.5 x86_64 machine, I see:

% launchctl version
Darwin Bootstrapper Version 7.0.0: Thu Apr 25 21:30:47 PDT 2024; root:libxpc_executables-2748.121.1~1/launchd/RELEASE_X86_64

On a Sonoma 14.6.0 and 14.6.1 aarch64 machine, I see:

% launchctl version
Darwin Bootstrapper Version 7.0.0: Sat Jun 29 04:48:38 PDT 2024; root:libxpc_executables-2748.140.10~372/launchd/RELEASE_ARM64E

Interestingly, the security release notes for Sonoma 14.6 says:

libxpc

Available for: macOS Sonoma

Impact: An app may be able to bypass Privacy preferences

Description: A permissions issue was addressed with additional restrictions.

CVE-2024-40805

There are similar patches for CVE-2024-40805 in iOS, iPadOS, tvOS and watchOS; and there's no other fixes for libxpc in those versions.

There aren't any publicly-available details around what that bug actually is, which is normal for an issue that's either privately disclosed or discovered internally at Apple. My outsider, armchair theory is:

• I see how the words "additional restrictions" could describe the inability to set environment variables on macOS-provided LaunchAgents.

• I have no idea whether the bug is in ssh-agent itself or some other macOS daemon... but locking it down in this way feels like a scorched-earth approach.

• That suggests to me there's some class of issue which affects multiple system LaunchAgents.

There have been narrower environment variable restrictions in the past: the previous (open source) version of launchd prevented setting DYLD_* environment variables, which could be otherwise used to inject code into other processes.

I still feel that this issue may be an unintended side-effect: Apple's default ssh-agent service will still run your own ssh-askpass, as long as it's at /usr/X11R6/bin/ssh-askpass. This raises the bar for misusing that feature to "you need root". However, the harshest restrictions on macOS come from System Integrity Protection, which they didn't use here, even though they could have.

[MichaelRoosz]

here is the workaround I am using:
https://github.com/MichaelRoosz/homebrew-ssh/blob/main/etc/install-libsk-libfido2-v1.1.5.zsh#L24

this stops any other ssh-agent, then uses its own socket env var to start the agent with custom config, then symlinks the original ssh socket to the custom one

[simmel]

@MichaelRoosz Won't launchd use socket activation and start a new ssh-agent and inherit SSH_AUTH_SOCK with a new path to all GUI applications?

[MichaelRoosz]

@simmel due to the symlink, socket activation will start the custom ssh-agent

[micolous]

The workaround is actually pretty similar to this workaround posted earlier, but automates the socket symlinking hack at the end.

All of the launchctl setenv steps of that script shouldn't be needed:

• They don't actually apply to the ssh-agent launched at the end of the script - the EnvironmentVariables key does.
• ssh-agent is responsible for calling ssh-askpass and deciding when to call it. Your normal ssh client should just connect to SSH_AUTH_SOCK.

In any case, if a future macOS version starts protecting launchd's sockets (because this seems like a loophole that which impacts the integrity of the system), then this and @simmel's tongue-in-cheek workaround would break.

Launching ssh-agent yourself is only really needed for U2F/FIDO2-based keys over SSH, because that needs extra libraries and environment variables to make it work. If you're using PKCS#11/PIV or even a key file on disk, you don't need that. 😄

[MichaelRoosz]

@micolous the setenv (at least SSH_SK_PROVIDER) are needed to make the other ssh tools pickup the FIDO2 provider lib. they are there for a reason. SSH_ASKPASS was needed to make ssh work when used by ansible, if I remember correctly, but it surely does not hurt to set it.

to be honest, it is a shame that Apple does not support setting these configs in an easy way without needing to use those workarounds. I really hope they properly implement a way to configure the ssh-agent before they mess it up even more.

[lepus2589]

From the comments and suggestions here, I tried to use the workaround masking the system ssh_agent (#54 (comment)). After some tinkering, I found a solution with automatic linking of the system ssh-agent socket to the user ssh-agent socket, which works well for me and is non-invasive and easily reversible.

I compiled my solution into a gist (https://gist.github.com/lepus2589/5235d78172e529b2d076d2c0e9670b27). Feel free to test it and to give feedback!

[schiffy91]

I made a self-contained installer / uninstaller here. The script is aware of the 1Password location – because that's what I use – but if there's a similarly pre-defined socket path for AskPass, just add it here before running the script.

# List of known alternative socket paths
KNOWN_SOCKETS=(
    "$HOME/Library/Group Containers/2BUA8C4S2C.com.1password/t/agent.sock"  # 1Password
    "$HOME/.gnupg/S.gpg-agent.ssh"  # GPG Agent
)

[Boracem1]

I made a self-contained installer / uninstaller here. The script is aware of the 1Password location – because that's what I use – but if there's a similarly pre-defined socket path for AskPass, just add it here before running the script.

# List of known alternative socket paths

KNOWN_SOCKETS=(

    "$HOME/Library/Group Containers/2BUA8C4S2C.com.1password/t/agent.sock"  # 1Password

    "$HOME/.gnupg/S.gpg-agent.ssh"  # GPG Agent

)

@lepus2589 @schiffy91 Sorry maybe this is basic but I've been trying to figure this out for 3 days now. What would I have to add to known sockets here to make this script work? I'm not doing anything fancy I just have OpenSSH installed from homebrew and want to use my yubikey with Fido2 for ssh and have agent forwarding working. Any help is appreciated and thanks for all the work that everyone put into figuring out the issue.

[schiffy91]

@Boracem1

./ssh_agent_overlay.sh --install --path-to-socket=<path_to_socket> should have you covered. I don't know which socket you're trying to use, but that's probably the easiest way to specify a custom socket for your use case.

You could also add it to the KNOWN_SOCKETS array, as you have pointed out, which is only aware of 1Password and GPG, but I can always update the script with more paths.

[Boracem1]

@Boracem1

./ssh_agent_overlay.sh --install --path-to-socket=<path_to_socket> should have you covered. I don't know which socket you're trying to use, but that's probably the easiest way to specify a custom socket for your use case.

You could also add it to the KNOWN_SOCKETS array, as you have pointed out, which is only aware of 1Password and GPG, but I can always update the script with more paths.

@schiffy91 I took some time to read over your script and I believe you are trying to use a different agent than the simple ssh-agent. I just want to use the user started default ssh agent which means I shouldn't have to pass in anything as far as I understood as the solution by @lepus2589 already takes care of everything and you just add an extra step of replacing the agent to one the one you use instead of the one that gets started in user-space by https://gist.github.com/lepus2589/5235d78172e529b2d076d2c0e9670b27#file-com-openssh-ssh-agent-overlay-sh. Would be great if the script worked without having to specify alternate agent since it will then just use the above user-space ssh-agent. I would be happy to contribute but I don't know what I'm doing and spent a whole day just to decipher what the script was doing. Please correct me if I am wrong.

[Boracem1]

I've followed the instructions by @lepus2589 and checked that everything is working as he described and when I invoke ssh-add I do get a dialog but never when I try to ssh to a machine. It never invokes ssh-askpass and I cannot figure out why for some reason. I just get agent refused operation. Someone please help as I will go insane trying to debug this with my limited skills.

[micolous]

@Boracem1 My previous comment describes setting up with stock ssh-agent, which works with SSH keys in a file and SSH keys on a PIV smartcard (or token).

The stock Apple ssh-agent configuration (com.openssh.ssh-agent) does not work with FIDO2 (as you asked about FIDO2 here), because it does not allow loading the additional modules to let it support it. It is possible to use it with Apple's distribution of SSH, if you run your own ssh-agent (with a different SSH_AUTH_SOCK environment variable).

However, some security keys (like Yubikey) support PIV in addition to U2F and FIDO2 – check with your security key vendor if they support it, or how to enable it. The Apple ssh-agent supports using RSA keys on PIV tokens (not ECDSA), including with an ssh-askpass. I suspect that this is how Apple's corporate SSH access is configured for access to their production systems.

There's instructions for setting up PIV for SSH here, but that was written before Sonoma 14.6: so you'll need to use the ssh-keychain.dylib instructions (do not install openssh from Homebrew or use libykcs11.dylib) and the ssh-askpass workaround I mention here.

The important things that need to happen are:

• DISPLAY needs to be set by SecureSocketWithKey, which XQuartz does, and that needs to show up in whatever runs ssh and ssh-agent.
• ssh-askpass needs to be in a location ssh-agent will search by default, that is not protected by SIP.
• SSH_AUTH_SOCK points to a file owned by the default Apple com.openssh.ssh-agent daemon.
• Only use ssh-add with keys in a file, or with a provider signed by Apple (such as ssh-keychain.dylib).

Keep in mind that any environment variable that you set up in your shell, or your bashrc or similar, is not the environment variables that you'll see in applications or in Apple's com.openssh.ssh-agent daemon (which discards all user-set variables in macOS 14.6 and later except SecureSocketWithKey). The SSH_AUTH_SOCK environment variable is owned by com.openssh.ssh-agent, and that is enforced by launchd, so it is not possible to globally override it – only point applications at a different environment variable, or attempt to replace the file it points after launchctl has triggered starting ssh-agent.

For posterity: anything that needs to replace ssh-agent to work (like password managers and FIDO2) will have a harder time of all of this.

[micolous]

I compiled my solution into a gist (https://gist.github.com/lepus2589/5235d78172e529b2d076d2c0e9670b27). Feel free to test it and to give feedback!

@lepus2589 From your post:

Two workarounds are being offered in this issue. The first one (#54 (comment)) involves installing the XQuartz package, just to get the DISPLAY variable set and then relies implicitly on a default location, where the system ssh-agent looks for ssh-askpass in the absence of an SSH_ASKPASS variable. This, to me, seems unwieldy and kind of fragile.

Setting the DISPLAY environment variable and putting ssh-askpass in a certain path is perfectly normal, and is what OpenSSH looks for on other operating systems (like Linux and BSD). Because of the way XQuartz's launchd configuration works, if applications never connect to that DISPLAY socket (which ssh won't), then launchd will never start XQuartz.

Trying to race launchd and com.openssh.ssh-agent for SSH_AUTH_SOCK and replace the file underneath it is unwieldy and fragile, and a bad idea. It's also likely to break, because as I previously noted, this exploits something that could be considered a security weakness. 😄

[Boracem1]

@micolous Why is it that ssh-add does show a prompt for pin? Also what is your recommended solution for using fido since I have the Yubikey bio fido edition.

[micolous]

Why is it that ssh-add does show a prompt for pin?

@Boracem1 Because ssh-add itself is what is prompting you for the PIN, not ssh-agent.

Also what is your recommended solution for using fido

The short answer is that I wouldn't use FIDO2 for SSH authentication. It's an (admittedly novel) hack to get a security key to sign something that is vaguely SSH shaped, but requires server compatibility to make it work (some large cloud providers don't support), and throws out many WebAuthn security features in the process (though some of these don't have equivalents in SSH).

Longer term, I expect macOS to start locking down use of U2F/FIDO2 devices in a similar way to what Microsoft has already done on Windows 10 and 11, where all access to hardware authenticators must go through their platform WebAuthn API. macOS has a similar "Passkeys API for Web Browsers", but getting access to that is significantly more restricted than it is on Windows. This will break ssh-sk in the process.

There may be a way to have your device's secure element/enclave (aka: "Touch ID") generate a certificate for use with SSH, and expose it via ssh-keychain.dylib. I haven't gone down that path myself, but that would at least give you hardware-bound keys which can't be copied out of the SE (or at least, require iCloud Keychain sync), which is better than just storing it in a file (and maybe storing the private key's passphrase in your Keychain).

What limits you is that ssh-agent configuration is much more limited on Sonoma 14.6 and later. That's an Apple issue, not an ssh-askpass one (though it affects ssh-askpass too). If you don't like it, go talk to Apple. 😄

But as I previously mentioned, I suspect that this is to fix a security bug or weakness. The exact nature of the issue was not disclosed publicly, and that'll limit what Apple can / will "reasonably" do.

[MichaelRoosz]

if you need an easy solution on macOS with homebrew:

brew install michaelroosz/ssh/libsk-libfido2-install

this takes care of everything (installs the fido2 ssh module and configures ssh-agent)

[lepus2589]

From #54 (comment)

2. Install XQuartz. This can be [downloaded from the project's website](https://www.xquartz.org/) or can be installed with `brew install xquartz`.
   You don't have to actually _use or run_ XQuartz – this is just so that the `DISPLAY` environment variable is set for system processes like `ssh-agent`.

3. Check that the `DISPLAY` environment variable would be set for `ssh-agent` (under "inherited environment"):
   launchctl print gui/$UID/com.openssh.ssh-agent
     
   If `DISPLAY` _isn't_ set, then XQuartz is not installed correctly.

I tried to do this with MacPorts, which is also officially supported by the XQuartz project (https://www.xquartz.org/releases/index.html#macports):

$ sudo port -v install xorg-server

Both the privileged_startx launch daemon and the launchd_startx launch agent are installed properly and active. After relogging (and rebooting), there is still no DISPLAY variable set in launchctl print gui/$UID/com.openssh.ssh-agent.

Any ideas? Is this a bug in the MacPorts package?

BTW, @micolous, why would the injection into the system agent work for the DISPLAY variable, if the same doesn't work for SSH_ASKPASS?

[micolous]

Re @lepus2589:

Both the privileged_startx launch daemon and the launchd_startx launch agent are installed properly and active. After relogging (and rebooting), there is still no DISPLAY variable set in launchctl print gui/$UID/com.openssh.ssh-agent.

Any ideas? Is this a bug in the MacPorts package?

Maybe.

MacPorts xinit package still disables the user_startx LaunchAgent by default, possibly for coexistence with Apple's previously-supplied X11.app. If their instructions to enable it don't work, then it may be a MacPorts bug, or it's being interfered with by some other security software on your device.

I've installed XQuartz on a fresh Sequoua 15.3.1 install with Homebrew's xquartz cask (which uses the upstream package as-is), and the DISPLAY environment showed up after logging out and logging in again (I've added step that to my instructions above).

So my suggestion would be to just install the upstream package from the XQuartz website. You'll end up with two XQuartz installs if some other MacPorts package depends on it, but at least one of them should work. 🙃

BTW, @micolous, why would the injection into the system agent work for the DISPLAY variable, if the same doesn't work for SSH_ASKPASS?

Because it uses SecureSocketWithKey rather than launchctl setenv. Logs which demonstrate this are in #54 (comment)

The value of an environment variable set by SecureSocketWithKey is under launchd's control, and launchd creates the socket file at the location. It only launches your LaunchAgent when something attempts to open it. SecureSocketWithKey was not restricted in macOS 14.6 either because they didn't consider it, or that it's harder to abuse because it doesn't allow the environment variable to be set to arbitrary content (ie: DISABLE_ALL_THE_SECURITY=1).

Because it's not possible to have two LaunchAgents to declare a SecureSocketWithKey with the same environment variable (eg: SSH_AUTH_SOCK), all the ssh-agent replacement workarounds replace that socket file out from under launchd, with the idea that they run before any new process attempts to use SSH_AUTH_SOCK, and so they connect to the replacement ssh-agent instead of Apple's ssh-agent, and launchd is none the wiser.

Apple might decide prevent anything from replacing a system-provided SecureSocketWithKey file in a future version of macOS. They might also decide to ignore SecureSocketWithKey set by non-system daemons on system daemons. They could also replace the ssh-agent's transport layer entirely with something like XPC, or make ssh talk to coreauthd directly. 🤷

[lepus2589]

@micolous Thanks again for taking the time for this detailed explanation!

I tried deactivating the MacPorts package and installing the XQuartz.pkg package, and it worked as advertised. The DISPLAY variable was set.

After uninstalling the pkg package again and reactivating the MacPorts package, magically, the DISPLAY variable was also set. I have no idea, what went wrong before. I'm not aware, that I did anything to prevent the MacPorts package from working properly.

So, for anybody else, who has this problem with the MacPorts xinit package: Have you tried to deactivate and reactivate it again? 🙃

Edit: It turns out, I also had to disable the ssh-askpass plist. The stopping of the system ssh service com.openssh.ssh-agent, that is done there, apparently removed the DISPLAY variable again. This PR removes this behavior.