macOS SSH agent forwarding not working any longer

[outsinre]

Description

SSH agent forwarding described at https://docs.docker.com/desktop/networking/#ssh-agent-forwarding does not work any longer after recent Docker Desktop upgrade and macOS upgrade. It worked a few months ago.

Reproduce

1. Start Docker Desktop.

2. SSH agent on host.

   ~ $ ssh-add -l
   384 SHA256:a2TyZj/tyhwUrs6XYhg//+Zwnpvwp9yttted4OgmWtg ...
   4096 SHA256:x2gO9GTWyTkJLudIwcFwHXYez0BFktfSc9FZ/jGqPU ...

3. Launch a new container.

   ~ $ docker run --rm -it -u root \
   --mount "type=bind,src=/run/host-services/ssh-auth.sock,target=/run/host-services/ssh-auth.sock" \
   -e SSH_AUTH_SOCK="/run/host-services/ssh-auth.sock" \
   ubuntu:latest bash
   
   root@c7783e0b7cd5:/# apt-get update && apt-get install -y ssh

4. SSH agent in the container. No SSH keys found!

   root@c7783e0b7cd5:/# ssh-add -l
   The agent has no identities.

Expected behavior

SSH agent in the container should see the SSH keys in the forwarded SSH agent.

docker version

Client:
 Cloud integration: v1.0.35+desktop.10
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:13:26 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.27.2 (137060)
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:22 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    25.0.3
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     /Users/zachary/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.5-desktop.1
    Path:     /Users/zachary/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.24
    Path:     /Users/zachary/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/zachary/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     /Users/zachary/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /Users/zachary/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.0
    Path:     /Users/zachary/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/zachary/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.4.1
    Path:     /Users/zachary/.docker/cli-plugins/docker-scout
WARNING: Plugin "/Users/zachary/.docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /Users/zachary/.docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 58
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.12-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 7.657GiB
 Name: docker-desktop
 ID: c3700971-3280-4ddf-a6b2-ed3956797d34
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

FD6CD52B-6D96-4731-9597-7A67E1E7F302/20240228141331

Additional Info

No response

[outsinre]

After some search, found a similar issue: #6541 (Docker version 4.13.0) with a workaround to bypass this bug: start Docker Desktop (open -a Docker) from terminal instead of from launchpad. However, this seems to have performance issue. Docker engine does not respond to commands instantly.

Using this way, only root user can access to the forwarded agent. For non-root user, we have to update the permission as follows. See #4242 and #5303.

~ $ ll /run/host-services/ssh-auth.sock
srwxr-xr-x 1 root root 0 Feb 28 14:39 /run/host-services/ssh-auth.sock

~ $ sudo chmod a+w /run/host-services/ssh-auth.sock

Alternatively, setting the "file sharing" from "VirtioFS" to "gRPC FUSE". See https://stackoverflow.com/q/76343112.

[alx75]

I had the same issue and open -a Docker does the trick for root user. Would it make sense to start ssh-agent only if SSH_AUTH_SOCK is not already set to avoid this issue ?

Docker desktop version: Docker Desktop 4.35.1 (173168)