Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for SSH connection #1282

Merged
merged 1 commit into from
Oct 25, 2021
Merged

Conversation

matejvasek
Copy link
Contributor

@matejvasek matejvasek commented Sep 13, 2021

Summary

This allows for connection via SSH by setting DOCKER_HOST.

Documentation

  • Should this change be documented?
    • Yes, see #___
    • No

Related

Resolves #1259

@github-actions github-actions bot added this to the 0.22.0 milestone Sep 13, 2021
@github-actions github-actions bot added type/chore Issue that requests non-user facing changes. type/enhancement Issue that requests a new feature or improvement. labels Sep 13, 2021
@matejvasek
Copy link
Contributor Author

@dmikusa-pivotal please try this out.

You might need to turn off SELinux in the podman VM. Also you will need to set --docker-host flag (e.g. --docker-host=unix:///var/run/docker.sock).

@matejvasek matejvasek force-pushed the add-ssh-support branch 2 times, most recently from d1bc4f0 to aa529c0 Compare September 13, 2021 14:20
cmd/cmd.go Outdated Show resolved Hide resolved
@matejvasek
Copy link
Contributor Author

Work is still in progress. There are no prompts for password/passphrase.

@matejvasek
Copy link
Contributor Author

Why is CI using go 1.14?

@codecov
Copy link

codecov bot commented Sep 21, 2021

Codecov Report

Merging #1282 (224e0a4) into main (fc9cb50) will increase coverage by 0.35%.
The diff coverage is 91.43%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1282      +/-   ##
==========================================
+ Coverage   80.11%   80.46%   +0.35%     
==========================================
  Files         143      144       +1     
  Lines        8760     9039     +279     
==========================================
+ Hits         7017     7272     +255     
- Misses       1285     1301      +16     
- Partials      458      466       +8     
Flag Coverage Δ
os_linux 79.08% <91.43%> (+0.36%) ⬆️
os_macos 73.63% <0.36%> (-2.34%) ⬇️
os_windows 80.29% <89.29%> (+0.29%) ⬆️
unit 80.46% <91.43%> (+0.35%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

cmd/cmd.go Outdated
Comment on lines 175 to 168
Identity: os.Getenv("DOCKER_HOST_SSH_IDENTITY"),
PassPhrase: os.Getenv("DOCKER_HOST_SSH_IDENTITY_PASSPHRASE"),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How should we name these envvars?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these the same ones that Docker uses? If so, we should keep them

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No they are not used by docker. It's my addition. With docker you need to put identity at default location with default name or add it to ssh-agent, you cannot specify identity file AFAIK.

@matejvasek
Copy link
Contributor Author

@dmikusa-pivotal please try this out.

@matejvasek
Copy link
Contributor Author

Also it would be great if somebody tried this on Windows.

@matejvasek matejvasek marked this pull request as ready for review September 23, 2021 18:32
@matejvasek matejvasek requested a review from a team as a code owner September 23, 2021 18:32
cmd/cmd.go Outdated Show resolved Hide resolved
@matejvasek
Copy link
Contributor Author

@joshuawhite929 please try this out.

@dmikusa
Copy link
Contributor

dmikusa commented Sep 27, 2021

Sorry for the delay. It is working for me.

I checked out your code, make build and used ./out/pack to do a pack build. I had DOCKER_HOST set to ssh://user@my-vm:port. It didn't require setting --docker-host.

Looks good!

@jromero jromero self-assigned this Sep 27, 2021
@jromero jromero removed the type/chore Issue that requests non-user facing changes. label Sep 27, 2021
Copy link
Contributor

@aemengo aemengo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(fearfully) attempting to validate on Windows here. But not really sure how these VMs are set up to be able to communicate over SSH.

@dmikusa-pivotal, do you have some sort of ssh proxy forwarding traffic from 22 to a Docker daemon listening on a port? Maybe we can look at your setup together to speed this along.

cmd/cmd.go Outdated Show resolved Hide resolved
cmd/cmd.go Outdated Show resolved Hide resolved
cmd/cmd.go Outdated
Comment on lines 175 to 168
Identity: os.Getenv("DOCKER_HOST_SSH_IDENTITY"),
PassPhrase: os.Getenv("DOCKER_HOST_SSH_IDENTITY_PASSPHRASE"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these the same ones that Docker uses? If so, we should keep them

@dmikusa
Copy link
Contributor

dmikusa commented Sep 28, 2021

@aemengo

@dmikusa-pivotal, do you have some sort of ssh proxy forwarding traffic from 22 to a Docker daemon listening on a port? Maybe we can look at your setup together to speed this along.

You need a VM with two things to make this work:

  1. OpenSSH installed. Basically, you should be able to ssh into the VM. For a Linux VM, just install your distro's openssh server package.
  2. Docker Daemon running. You can use the Docker instructions for installing Docker community. If you can docker version on the remote VM as the user you're planning to connect with over ssh then you should be set (you may also need to add the user you're connecting with to the docker group so it can access the daemon without needing sudo).

There's nothing specific you have to do to Docker to make it support this. It's basically functionality in the docker client & now pack to run docker commands remotely.

Hope that helps!

@aemengo
Copy link
Contributor

aemengo commented Sep 30, 2021

@dmikusa-pivotal Thanks for the help. I followed your instructions but am still having trouble using an ssh connection on a windows machine. I am able to use the ssh client directly.

Please let me know if I'm doing something wrong. Happy to look at this with someone.

PS> $env:DOCKER_HOST="ssh://aemengo@127.0.0.1"
PS> $env:DOCKER_HOST_SSH_IDENTITY="C:\Users\aemengo\Desktop\rdp\id_ed25519"
PS> pack build aemengo/hello -p ..\hello -B cnbs/sample-builder:dotnet-framework-1809
ERROR: failed to build: failed to fetch builder image 'index.docker.io/cnbs/sample-builder:dotnet-framework-1809': error during connect: Post "http://example.com%2F/v1.38/images/create?fromImage=cnbs%2Fsample-builder&tag=dotnet-framework-1809": ssh: rejected: connect failed (open failed)

@dmikusa
Copy link
Contributor

dmikusa commented Sep 30, 2021

@aemengo - FYI, I only tested with password-less login. I'm not sure if it works if you have to login over ssh.

Are you able to ssh aemengo@127.0.0.1 without any additional prompts? Password most notably, but also make sure you've trusted the host key.

Also, can you docker images and see images from the remote docker? Docker should also work over ssh. If that works but pack doesn't, then it would be something specific with pack that's failing.

@matejvasek
Copy link
Contributor Author

matejvasek commented Sep 30, 2021

There should already be implemented prompt for password and for temporary trusting the server.

@matejvasek
Copy link
Contributor Author

matejvasek commented Sep 30, 2021

So ssh "ssh://aemengo@127.0.0.1" -i "C:\Users\aemengo\Desktop\rdp\id_ed25519" does work?
Is aemengo user in docker group?

@matejvasek
Copy link
Contributor Author

matejvasek commented Sep 30, 2021

@aemengo Does something like:
ssh -NL localhost:2374:/var/run/docker.sock "ssh://aemengo@127.0.0.1" -i "C:\Users\aemengo\Desktop\rdp\id_ed25519"
work?

@matejvasek
Copy link
Contributor Author

matejvasek commented Sep 30, 2021

also what Daniel said: try whether docker CLI works with DOCKER_HOST="ssh://aemengo@127.0.0.1"

@matejvasek
Copy link
Contributor Author

@aemengo could you please try Windows again, I did some changes.

@matejvasek
Copy link
Contributor Author

@dmikusa-pivotal I did some changes could you please re-test it?

@matejvasek
Copy link
Contributor Author

How can I rerun Codecov?

@matejvasek
Copy link
Contributor Author

@aemengo @jromero PTAL

@aemengo
Copy link
Contributor

aemengo commented Oct 11, 2021

@matejvasek Don't know your magic, but things seems to be working well on my windows machine 👍🏾
Please give me a few more to finish reviewing.

@matejvasek
Copy link
Contributor Author

matejvasek commented Oct 11, 2021

@matejvasek Don't know your magic, but things seems to be working well on my windows machine 👍🏾 Please give me a few more to finish reviewing.

I allowed use of standard docker system dial-stdio mechanism if the commands happens to be in the remote machine. If it's not it will use tunneling. I mean I call https://pkg.go.dev/github.com/docker/cli/cli/connhelper#GetConnectionHelper if it should work in given machine.

@matejvasek
Copy link
Contributor Author

I allowed use of standard docker system dial-stdio mechanism if the commands happens to be in the remote machine. If it's not it will use tunneling.

I could also do it other way around: use tunneling if possible (i.e. with everything but npipe) and use docker system dial-stdio as the fallback.

Copy link
Contributor

@aemengo aemengo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, I'm a little afraid of introducing something that a select few maintainers are likely to debug. But it did work stellarly during my testing.

Once again, we can't thank you enough for a PR of such depth.

cmd/cmd.go Outdated Show resolved Hide resolved
@n1hility
Copy link

Nice work on buildpack ssh support @matejvasek

cmd/cmd.go Outdated
if err != nil {
return pack.Client{}, err
}

return *client, nil
}

func tryInitSSHDockerClient() (dockerClient.CommonAPIClient, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we move these functions to a separate file in this directiory, maybe docker_init.go? I like keeping cmd/cmd.go simple and self-contained

cmd/cmd.go Outdated Show resolved Hide resolved
cmd/cmd.go Outdated Show resolved Hide resolved
internal/sshdialer/ssh_dialer.go Outdated Show resolved Hide resolved
internal/sshdialer/ssh_dialer.go Outdated Show resolved Hide resolved
internal/sshdialer/ssh_dialer.go Outdated Show resolved Hide resolved
@dfreilich
Copy link
Member

This is wildly impressive work, @matejvasek . I am a bit nervous about the future upkeep for this, so I would definitely rather the ssh_dialers functionality comes through an imported function from podman; if they'd consider exposing the required api and we could use that, I think it would definitely make it a lot safer for us to import it.

cmd/cmd.go Outdated
func newHostKeyCbk() sshdialer.HostKeyCallback {
var trust []byte
return func(hostPort string, pubKey ssh.PublicKey) error {
if bytes.Equal(trust, pubKey.Marshal()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see trust being persisted. Should it be? What's the experience connecting to the same connection a second time?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be written into ~/.ssh/known_hosts. But I am not too eager to write into system files. I could however print hint, something like: "add this line {key record here} into your ~/.ssh/known_hosts, or "to trust the key connect to the host via ssh first". BTW standard docker CLI is not solving this at all, you must add trust by yourself.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So right now you are asked each time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW standard docker CLI is not solving this at all, you must add trust by yourself.

to be more specific you will get:

~> docker image ls
error during connect: Get "http://docker.example.com/v1.24/images/json": command [ssh -l root -p 45383 -- localhost docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=ssh_askpass: exec(/usr/libexec/openssh/ssh-askpass): No such file or directory
Host key verification failed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also standard docker CLI does not support password authentication.
And you cannot specify identity file (and its passphrase) -- it either has to be at standard location with standard name,
or it has to be added to ssh-agent. Passphrase reading also has to be handled by ssh-agent.

cmd/cmd.go Outdated Show resolved Hide resolved
cmd/cmd.go Outdated Show resolved Hide resolved
cmd/cmd.go Outdated Show resolved Hide resolved
@matejvasek
Copy link
Contributor Author

@jromero wrt host key callbck

Where are these keys persisted?

We are not persisting it anywhere yet. I could print instruction what line should be appended to ~/.ssh/known_hosts to persist trust.

What/How do we handle non-interactive scenarios?

You can forward output of the yes utility to input of pack command: yes | pack build ....
Or you can add the host key into ~/.ssh/know_hosts somehow else.

@matejvasek
Copy link
Contributor Author

matejvasek commented Oct 22, 2021

This is wildly impressive work, @matejvasek . I am a bit nervous about the future upkeep for this, so I would definitely rather the ssh_dialers functionality comes through an imported function from podman; if they'd consider exposing the required api and we could use that, I think it would definitely make it a lot safer for us to import it.

@dfreilich I totally agree this is not best place to keep it. Initial I created separate repo: https://github.com/matejvasek/sshdialer.
Especially because this code might be useful in another projects. But then I would be responsible for maintaining it.

So latter I moved it here, one of the reasons was that I really liked your GithubActions. You even have custom runner for Windows with Linux containerization which I found really useful.

I could talk to people from https://github.com/containers about it. But I am afraid I won't be able to run WIndows tests in CI there.

cmd/cmd.go Outdated

if answer == "yes" || answer == "y" {
trust = pubKey.Marshal()
fmt.Fprintf(os.Stderr, "To avoid this in future add following line into your ~/.ssh/known_hosts:\n%s %s %s\n",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jromero here I added message for user with instruction how to make server trusted. Is it OK?

@matejvasek
Copy link
Contributor Author

@jromero @dfreilich PTAL

Copy link
Member

@dfreilich dfreilich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

I appreciate the rigorous tests, which definitely makes it a lot easier to support. At the same time, if the podman team is open to supporting this, that would be great, and I'd appreciate adding a link to an issue on their repo so we can track this integration.

@jromero , what do you think? Holding it off until your approval.

cmd/docker_init.go Outdated Show resolved Hide resolved
cmd/docker_init.go Outdated Show resolved Hide resolved
internal/sshdialer/ssh_dialer.go Show resolved Hide resolved
Copy link
Member

@jromero jromero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nits.

Overall good. Excited to get this in.

Could you resolve the merge conflict? I can then give it a quick test and merge this in.

EDIT: Looks like conflict is resolved.

internal/sshdialer/windows_test.go Show resolved Hide resolved
internal/sshdialer/posix_test.go Show resolved Hide resolved
internal/sshdialer/ssh_dialer.go Outdated Show resolved Hide resolved
@matejvasek
Copy link
Contributor Author

I am going to squash review requested fixups.

Signed-off-by: Matej Vasek <mvasek@redhat.com>
@jromero jromero removed the type/chore Issue that requests non-user facing changes. label Oct 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement Issue that requests a new feature or improvement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

pack doesn't honor DOCKER_HOST using ssh
6 participants