support SSH connection #1014

AkihiroSuda · 2018-04-20T10:04:08Z

Signed-off-by: Akihiro Suda suda.akihiro@lab.ntt.co.jp

- What I did

Added support for SSH connection. e.g. docker -H ssh://me@server

- How I did it

Followed @tonistiigi 's suggestion: #889 (comment)

The cli should accept ssh://me@server for DOCKER_HOST and -H. Using that would execute ssh with the passed config.
The ssh command would call a hidden command on the docker CLI binary on the remote side. For example, docker dial-stdio.
This command will make a connection to the local DOCKER_HOST variable (almost always the default local socket) and forward that connection on the commands stdio.
Even though this command is supposed to run locally to the dockerd binary, we think that it is an invalid configuration for this feature to remove the local docker binary so we can rely on it always being present.

- How to verify it

docker -H ssh://me@server run -it --rm busybox

- Description for the changelog

support SSH connection

- A picture of a cute animal (not mandatory but encouraged)

https://commons.wikimedia.org/wiki/File:Manchot_Ad%C3%A9lie_juv%C3%A9nile.jpg

Vendor: moby/moby#36630 (merged)

Tests (currently tested manually):

echo hello | ./docker -H ssh://10.2.54.48 run -i --rm busybox cat; echo world
./docker -H ssh://10.2.54.48 run --rm busybox echo hello; echo world
./docker -H ssh://10.2.54.48 run -i --rm busybox echo hello; echo world

cpuguy83 · 2018-04-23T14:21:12Z

Design LGTM! cc @tonistiigi for design as well.
Tested this out and it works pretty well.

I did notice that this:

echo hello | ./docker-dev -H ssh://docker@$ip run -i --rm busybox cat

Doesn't echo anything to my term when using the ssh connector.
But a normal interactive container works fine. Image pull also works well.

AkihiroSuda · 2018-04-23T17:00:49Z

@cpuguy83
Thank you for testing.

The issue can be mitigated by increasing this timeout, which is the equivalent of socat -t and nc -q: https://github.com/docker/cli/pull/1014/files#diff-065a55b1c9d14a6817c97ca733652920R17

However, increasing this value introduces extra sleep.

I'll try to find a more robust way

vdemeester · 2018-04-24T16:18:50Z

Design LGTM too, this is cool 😻

tonistiigi

Overall design LGTM. I don't understand the timeout logic on handling EOF though.

tonistiigi · 2018-04-21T00:24:22Z

cli/command/system/dial_stdio.go

+	if !ok {
+		return errors.New("the raw stream connection does not implement halfCloser")
+	}
+	stdio := &cliHalfCloser{dockerCli}


It may be confusing to use dockerCli here. I'd just use os pkg.

tonistiigi · 2018-04-21T00:30:14Z

cli/command/system/dial_stdio.go

+	}()
+	select {
+	case err := <-stdin2conn:
+		logrus.Debugf("stdin2conn: %v", err)


is this wip?

tonistiigi · 2018-04-21T00:31:17Z

cli/connhelper/connhelper.go

+// ConnectionHelper allows to connect to a remote host with custom stream provider binary.
+type ConnectionHelper struct {
+	Dialer    func(ctx context.Context, network, addr string) (net.Conn, error)
+	DummyHost string // dummy URL used for HTTP requests. e.g. "http://docker"


Just Host. We can also leave a sane default here if WithHost is not called. So connection helper case doesn't need to call it at all (or even better if connectionhelper just returns the client options directly instead of a struct).

tonistiigi · 2018-04-25T01:05:22Z

cli/command/cli.go

+		clientOpts = append(clientOpts, client.WithHost(host))
+	} else {
+		clientOpts = append(clientOpts, func(c *client.Client) error {
+			httpClient := &http.Client{


I think we should just add WithDialer instead of overriding client and remove WithHijackDialer. Using it would create a new client with a transport pointing to the current dialer and set it to hijack as well.

tonistiigi · 2018-04-25T01:19:33Z

cli/command/system/dial_stdio.go

+func runDialStdio(dockerCli command.Cli, opts dialStdioOptions) error {
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	conn, err := dockerCli.Client().DialRaw(ctx)


I think this would be slightly better as a Dialer() func(ctx) (net.Conn, error) . Otherwise, it seems like a method that calls to the remote side. We could also solve this by breaking up NewClientWithOpts so that we always get access to dialer in cli before creating the client and store the dialer so we can reuse it.

tonistiigi · 2018-04-25T01:21:34Z

cli/command/system/dial_stdio.go

+	}
+
+	flags := cmd.Flags()
+	flags.DurationVarP(&opts.timeout, "timeout", "t", 500*time.Millisecond, "After EOF from one of inputs, wait for this duration before exit")


I'm not sure I understand this timeout. Why isn't the close of connection handled cleanly? This could be just a connection timeout but then it should be much bigger (20 sec) and increasing it shouldn't have any downsides when timeout isn't reached.

tonistiigi · 2018-04-25T01:32:26Z

cli/command/system/dial_stdio.go

+	stdin2conn := make(chan error)
+	conn2stdout := make(chan error)
+
+	go func() {


These might be better handled with a wrapper around conn. But I guess I don't understand the need for the current timeout atm.

cpuguy83 · 2018-07-02T18:53:15Z

Any update here?

AkihiroSuda · 2018-07-03T02:41:07Z

Sorry still stuck at the stream connection issue #1014 (comment)

Will try to look into this again ASAP

cpuguy83 · 2018-07-03T17:11:03Z

No worries I just thought this was already merged and realized it wasn't!

codecov-io · 2018-07-27T06:55:10Z

Codecov Report

Merging #1014 into master will decrease coverage by 0.28%.
The diff coverage is 34.46%.

@@            Coverage Diff            @@
##           master   #1014      +/-   ##
=========================================
- Coverage   54.29%     54%   -0.29%     
=========================================
  Files         268     272       +4     
  Lines       17843   18068     +225     
=========================================
+ Hits         9687    9758      +71     
- Misses       7546    7694     +148     
- Partials      610     616       +6

sandys · 2018-08-20T03:54:54Z

Thanks for your reply. Will it not accept parameters passed through command line ? This is very convenient for portability (when using through different laptops, etc). Secondly we have more -o options that we pass.

…

On Mon, 20 Aug, 2018, 09:15 Akihiro Suda, ***@***.***> wrote: @sandys <https://github.com/sandys> ~/.ssh/config should work Host foo Hostname gcp.red.com PKCS11Provider /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1014 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEsU0WZDcoIQ79_eUWMRCJMA12QJgdTks5uSjDWgaJpZM4TdNDQ> .

AkihiroSuda · 2018-08-20T04:00:33Z

Currently no, could you open a github issue?

Secondly we have more -o options that we pass.

Any option that cannot be configured via ~/.ssh/config?

BretFisher · 2018-10-17T21:59:24Z

From ops and sysadmins everywhere, we thank you for this fantastic and unexpected feature. I'm hoping this will seriously cut down the number of times I see people opening dockerd TCP w/o TLS and just opt for SSH endpoints for remote mgmt. 👍 😂

overmike · 2018-11-09T19:28:13Z

is this possible to use in docker golang client rather than cli? just like docker-py understand the ssh:// DOCKER_HOST

cpuguy83 · 2018-11-09T21:53:05Z

@overmike No, none of the client helpers will set this up for you. Nothing is stopping someone from doing this themselves with the client, though. client.NewClient accepts an http client, which you can setup on your own.

The CLI is using a helper subcommand (hidden from --help) to assist in wiring up std i/o with the remote.

overmike · 2018-11-12T10:33:17Z

@cpuguy83 , thanks for your feedback. It clarifies I need to add connhelper code in our golang client.

I think you meant "docker system dial-stdio" for the hidden subcommand. And seems like docker-py 3.6.0 is going to support ssh DOCKER_HOST.

bullno1 · 2018-11-14T06:43:06Z

How does it handle authentication? Does it support signed SSH certificate? What about revocation?

cpuguy83 · 2018-11-14T13:37:07Z

It just shells out to the ssh bin. You can add whatever options for the host to your ssh config. I think there is another pr to support passing ssh opts through an environment var as well.

zmilonas · 2018-11-20T15:13:35Z

cli/connhelper/connhelper.go

+// GetConnectionHelper returns nil without error when no helper is registered for the scheme.
+// URL is like "ssh://me@server01".
+func GetConnectionHelper(daemonURL string) (*ConnectionHelper, error) {
+	u, err := url.Parse(daemonURL)


You can actually get a unparsedHost here which is not parseable - has no schema.
Like 127.0.0.1:2364

GetConnectionHelper returns nil without error when no helper is registered for the scheme.

I think, this is actually unrelated to what I'm talking about. Just a line below you check for err from url.Parse() and since there is a possibility the passed daemonURL is in 127.0.0.1:2345 format url.Parse() will rightfully return an error per its specification - no protocol scheme. I am not sure where should the scheme be added - here or higher in the call stack but this line in particular introduced a regression in the newest stable release of docker engine - 18.09.0

@zmilonas looks like you mean the issue that was addressed by #1443 ?

Well it might have been addressed but is not fixed 😅 I will file a new issue shortly then.

It's fixed on master, which is what this repository is tracking.

A backport PR is open to include it in the 18.09 codebase

Can we expect it to end up in some kind of 18.09.01 release then?

Yes, that's what the backport is for. In the meantime you could either modify your configuration (use tcp://<IP|host>:<port>), or download the static binaries for Docker 18.06 (or "nightly", which has the latest changes from "master") https://download.docker.com/linux/static/

AkihiroSuda requested review from thaJeztah and vdemeester as code owners April 20, 2018 10:04

GordonTheTurtle added the status/0-triage label Apr 20, 2018

AkihiroSuda mentioned this pull request Apr 20, 2018

support connection helpers (e.g. docker -H ssh://me@server) #889

Closed

vdemeester added status/1-design-review and removed status/0-triage labels Apr 20, 2018

vdemeester requested review from tonistiigi, cpuguy83 and simonferquel April 20, 2018 10:05

AkihiroSuda mentioned this pull request Apr 20, 2018

client: add WithDialContext() and client.Dialer() moby/moby#36630

Merged

AkihiroSuda force-pushed the connhelper-sshonly branch 4 times, most recently from 9fbd086 to e1624b5 Compare April 20, 2018 10:35

thaJeztah added the status/needs-vendoring label Apr 23, 2018

AkihiroSuda closed this Apr 23, 2018

AkihiroSuda reopened this Apr 23, 2018

tonistiigi reviewed Apr 25, 2018

View reviewed changes

tonistiigi added status/2-code-review and removed status/1-design-review labels Apr 25, 2018

AkihiroSuda force-pushed the connhelper-sshonly branch 2 times, most recently from 5cd375a to e6fc199 Compare July 27, 2018 06:55

AkihiroSuda changed the title ~~support SSH connection~~ [WIP] support SSH connection Jul 27, 2018

yajo mentioned this pull request Aug 20, 2018

Allow to define SSH connection options #1301

Closed

This was referenced Oct 12, 2018

can't set default schema to tcp for docker host param #1442

Closed

fixes #1441 set default schema to tcp for docker host #1443

Merged

thaJeztah added docs/revisit impact/changelog labels Oct 23, 2018

minamijoyo mentioned this pull request Oct 26, 2018

Invalid bind address protocol: ssh:// in DOCKER_HOST docker/docker-py#2159

Closed

blakepettersson mentioned this pull request Oct 28, 2018

Native Docker build and publish support harness/harness#2462

Open

unshare mentioned this pull request Nov 2, 2018

RFC: enable AuthZ on the unix socket moby/moby#35711

Closed

albertvipe mentioned this pull request Nov 12, 2018

Add support for remote connections using SSH docker-java/docker-java#1130

Open

zmilonas reviewed Nov 20, 2018

View reviewed changes

tamsky mentioned this pull request Nov 21, 2018

Support for sharing unix sockets docker/for-mac#483

Closed

Elufimov mentioned this pull request Dec 12, 2018

Remote connections using SSH hashicorp/terraform-provider-docker#112

Closed

diablodale mentioned this pull request Jan 4, 2019

request: add the native docker SSH support now available in v18.09 apocas/docker-modem#99

Closed

douglasmiranda mentioned this pull request Apr 13, 2019

Deployment: Provisioning douglasmiranda/ddpt#2

Closed

2 tasks

rappdw mentioned this pull request Jun 19, 2019

Refactor Dockerutils to utilize ssh resero-labs/docker-utils#51

Closed

tianon mentioned this pull request Dec 10, 2019

Add "openssh-client" to the standard client images docker-library/docker#207

Merged

thaJeztah mentioned this pull request Jan 23, 2020

OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "process_linux.go:101: executing setns process caused \"exit status 1\"": unknown moby/moby#40399

Closed

bulletmark mentioned this pull request Apr 1, 2020

Dry can not use ssh with DOCKER_HOST moncho/dry#127

Closed

thaJeztah mentioned this pull request Apr 29, 2022

SSH scheme in DOCKER_HOST moby/moby#43549

Closed

kuboraam mentioned this pull request Jul 4, 2022

Why is the TLS docker daemon being removed? trajano/terraform-docker-swarm-aws#18

Closed

thaJeztah mentioned this pull request Jul 29, 2022

cli: set timeout connection ping on sockets as well #3722

Merged

catusax mentioned this pull request Jan 22, 2023

feat: add support for remote docker over ssh GoogleContainerTools/skaffold#8349

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support SSH connection #1014

support SSH connection #1014

AkihiroSuda commented Apr 20, 2018 •

edited

Loading

cpuguy83 commented Apr 23, 2018 •

edited

Loading

AkihiroSuda commented Apr 23, 2018

vdemeester commented Apr 24, 2018

tonistiigi left a comment

tonistiigi Apr 21, 2018

tonistiigi Apr 21, 2018

tonistiigi Apr 21, 2018

tonistiigi Apr 25, 2018

tonistiigi Apr 25, 2018

tonistiigi Apr 25, 2018

tonistiigi Apr 25, 2018

cpuguy83 commented Jul 2, 2018

AkihiroSuda commented Jul 3, 2018

cpuguy83 commented Jul 3, 2018

codecov-io commented Jul 27, 2018 •

edited

Loading

sandys commented Aug 20, 2018 via email

AkihiroSuda commented Aug 20, 2018

BretFisher commented Oct 17, 2018

overmike commented Nov 9, 2018

cpuguy83 commented Nov 9, 2018

overmike commented Nov 12, 2018

bullno1 commented Nov 14, 2018 •

edited

Loading

cpuguy83 commented Nov 14, 2018

zmilonas Nov 20, 2018

AkihiroSuda Nov 20, 2018

zmilonas Nov 21, 2018

thaJeztah Nov 21, 2018

zmilonas Nov 21, 2018

thaJeztah Nov 21, 2018

zmilonas Nov 21, 2018

thaJeztah Nov 21, 2018

support SSH connection #1014

support SSH connection #1014

Conversation

AkihiroSuda commented Apr 20, 2018 • edited Loading

cpuguy83 commented Apr 23, 2018 • edited Loading

AkihiroSuda commented Apr 23, 2018

vdemeester commented Apr 24, 2018

tonistiigi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpuguy83 commented Jul 2, 2018

AkihiroSuda commented Jul 3, 2018

cpuguy83 commented Jul 3, 2018

codecov-io commented Jul 27, 2018 • edited Loading

Codecov Report

sandys commented Aug 20, 2018 via email

AkihiroSuda commented Aug 20, 2018

BretFisher commented Oct 17, 2018

overmike commented Nov 9, 2018

cpuguy83 commented Nov 9, 2018

overmike commented Nov 12, 2018

bullno1 commented Nov 14, 2018 • edited Loading

cpuguy83 commented Nov 14, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AkihiroSuda commented Apr 20, 2018 •

edited

Loading

cpuguy83 commented Apr 23, 2018 •

edited

Loading

codecov-io commented Jul 27, 2018 •

edited

Loading

bullno1 commented Nov 14, 2018 •

edited

Loading