Simplify setup required for remote DOCKER_HOST over SSH

[leighmcculloch]

Description

When connecting to a remote host over ssh, by configuring the DOCKER_HOST environment variable, or by using the docker context functionality and specifying the host= parameter, some Linux/Unix environments require additional setup because the docker CLI assumes that the host it is SSHing into has docker located in the PATH.

Unfortunately while docker is almost always installed into the user PATH, it is not always installed into the system path. This causes the docker CLI to fail when attempting to use these systems as a remote host without additional setup.

One system that this occurs on is macOS. The default installation directory of docker on Mac's is /usr/local/bin but that path is not in the system PATH that sshd configures for the ssh session.

As an example, this is what the default PATH is on macOS:

% ssh <host> 'echo $PATH'
/usr/bin:/bin:/usr/sbin:/sbin

To work around this issue a user must run the following commands on macOS and other systems that have the same setup.

echo 'PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin' >> ~/.ssh/environment
sudo sh -c 'echo "PermitUserEnvironment PATH" >> /private/etc/ssh/sshd_config'

Steps to reproduce the issue:

1. DOCKER_HOST=ssh://<host> docker ps

Describe the results you received:

error during connect: Get http://docker/v1.24/containers/json: command [ssh -- Gigantor.local docker system dial-stdio] has exited with exit status 127, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=zsh:1: command not found: docker

Describe the results you expected:

I expected it to be relatively straight forward to provide an SSH host address for another machine running docker and that docker could find itself.

I think it would significantly simplify using docker remotely across machines if docker didn't only rely on PATH but searched some common paths that Docker typically installs itself, such as /usr/local/bin/.

Additional information you deem important (e.g. issue happens only occasionally):

Issue happens consistently.

Output of docker version:

Client: Docker Engine - Community
 Cloud integration: 1.0.9
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:13:00 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:15:47 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.5.0)

Server:
 Containers: 2
  Running: 1
  Paused: 0
  Stopped: 1
 Images: 3
 Server Version: 20.10.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.121-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 1.942GiB
 Name: docker-desktop
 ID: BTMT:FQ3F:RCNB:IDF5:W7AP:7NRK:XSTI:7ERE:62JZ:3ODT:IARG:OQWC
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: gateway.docker.internal:3128
 HTTPS Proxy: gateway.docker.internal:3129
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
This is all running on physical machines. I had the same problem with one flavor of Linux a little while ago on AWS, but I didn't note the details of which distro and version and don't remember.

[thaJeztah]

/cc @AkihiroSuda @tonistiigi any ideas for an elegant solution?

[damienmillet]

I choose to create a symblink to /bin and it's fine

[hairyhenderson]

I just ran into this myself setting up a new machine and it stumped me! SIP (system integrity protection) on macOS now makes it painful to create symlinks in /bin//sbin/etc, so an option to specify the remote path would be really useful!

The good news is the workaround suggested by @leighmcculloch works for me.

[thaJeztah]

@hairyhenderson 👋 (it's been a while, hope you're doing well!) I just realised we did merge a patch that (IIUC) may solve this;

• connhelper: Allow socket path when using SSH #4073

It's part of docker 24.0 and up, but let me know if I'm correct!

[thaJeztah]

OH, actually, reading it wrong; this is not about the socket path 🙈 (blame multi-tasking)

[hairyhenderson]

hi @thaJeztah 👋😁

Thanks for the link to the other PR - it's related, but different 😉...