Connect to Windows Docker engine

[StefanScherer]

Has anybody started to add or think about a docker-machine driver and provisioner for Windows Docker engines?
So anybody can use docker-machine to secure and connect to a Windows Server 2016 TP4 as easy as we are used for a Linux Docker Engine?

Something like:

docker-machine create -d generic --generic-winrm-user "administrator" ... dev
docker-machine create -d azure --azure-username "administrator" --azure-winrm-port 5985 ... dev

Using WinRM (HTTP/HTTPS) instead of SSH to access the remote Windows Server and put the TLS certs into it and reconfigures the Windows Docker service and other things.
I know that packer has communicators for SSH and WinRM built in Go for provisioning.
Just connecting the dots...

ping @ahmetalpbalkan 😄

[nathanleclaire]

Due to the way the provisioner model currently works, it might not be too crazy to implement such functionality, although it would require re-thinking or papering over some pretty fundamental assumptions (e.g. the presence of SSH).

I have a feeling that the way it will need to be implemented is by having some flags which specifically indicate that it's a Windows machine. e.g. --generic-winrm-userpass.

[ahmetb]

I don't know much about Windows Server containers, however as Nathan said most of the work would be getting the SSH protocol on Windows either by waiting Microsoft to do it (it's happening) or providing a polyfill using WinRM maybe (I don't know how it works at all).

I remember some discussion about this was happening internally too. cc: @jstarks as he is on the windows containers team.

If you think you can prototype something, I would say go for it.

[enderjbr]

Hi @StefanScherer, I'm a PM in the Hyper-V team and I'm actually looking into how we could make this possible. One of the things we're thinking about is maybe we could use this as the OS that Docker Machine could pull. If we use Nano, then it would stay as a small download. The repo for the Open SSH in Windows project is here.

Any ideas are appreciated and if you're going to prototype something please let me know. I'll keep posting back here if I find out more about how to make this work.

cc: @dgageot

[StefanScherer]

@enderb-ms Oh, a Nano image, that would be great as well. And OpenSSH is also an option. In the meantime other projects like Packer and Vagrant improved Windows support to use WinRM and get rid of SSH to have a more native communication.

What I thought about as a first step is let docker-machine connect to a TP4 server, eg. with the generic driver.

Haven't tried the azure driver yet I think of selecting a TP4 VM + docker engine template at Azure and docker-machine then knows to provision a Windows VM instead of a Linux VM. It only has to do some minimal steps to connect the local machine where docker-machine is running with that remote Docker Engine.

These are the steps of a generic driver running against a Linux VM that already has Docker installed. I've added some ideas for Windows:

1. docker-machine connects to the remote IP address
2. check the version of the OS if it could be provisioned (cat /etc/os-release -> get some windows information instead)
3. update hostname?
4. update /etc/hosts
5. install curl (not needed to install it on Windows hosts)
6. check if docker is installed. If not install it. ( probably run the whole Install-ContainerHost.ps1 ? )
7. check docker version
8. create docker's config directory ( /etc/docker -> %ProgramData%/docker )
9. stop docker service
10. insert ca.pem ( Windows: %ProgramData%/docker/certs.d )
11. insert server.pem
12. insert server-key.pem
13. write docker's config file ( /etc/default/docker, probably an enhancement to %ProgramData%/docker/runDockerDaemon.cmd )
14. start the docker service
15. netstat -an

Phase 1:

For the Azure/local TP4 with preinstalled docker engine scenario:

Step 1.) it has to use WinRM (eg. have a look at packer's winrm communicator) instead of SSH.

Step 2.) could check if everything is up and running and if the VM is a TP4. Otherwise abort if it's not possible to install everything eg without a reboot.

It then has to do steps 9.) ... 14.) to stop docker, insert the TLS certs and restart docker. After that docker-machine has created the connection to an existing Azure VM / local or on-prem Windows VM and secured the docker port.

Phase 2:

Later on the step 2.) could be enhanced to allow "empty" 2016 servers to install Docker with step 6.)

Phase 3:

And then to have a "boot2docker.iso" with a Nano image in it to download locally.

Just some first thoughts ;-)

[dgageot]

For context: http://ashleypoole.co.uk/2016/docker-containers-windows-server-2016-guide/

[friism]

@dgageot are you planning to crank on this? If not, I think we should point the Microsoft folks in the right direction so that they can try and make progress

[dgageot]

@friism There's ongoing work with @enderb-ms already

[friism]

@dgageot great! Is there a place I can get more details?

[ppadala]

@dgageot I have a change that can spin up Windows server instances on Azure and setup Docker engine. Is there a repo, where @enderb-ms's work is available? I can submit mine here for review and possibly merge or unify with anything @enderb-ms is doing.

Quick description of what I have done

1. Add support for spinning up Windows image.
2. Use Windows server 2016 TP4 image on Azure
3. Spin it up. I am using the library https://github.com/Azure/azure-sdk-for-go instead of the vendored code in https://github.com/docker/machine/tree/master/vendor/github.com/MSOpenTech/azure-sdk-for-go, which seems like an older version.
4. Use WinRM to connect to the instance, install docker-engine
5. Perform any other steps. I have noticed that networking gets messed up sometimes and the Vswitch needs to be setup.

cc @ahmetalpbalkan

[StefanScherer]

@ppadala Amazing

[friism]

hawt

[ppadala]

Thanks @stefanfoulis @friism. The work is in https://github.com/containerx/machine, but that also includes some other changes unrelated to this PR.

I am creating a new branch, cleanup code and will submit a pull request soon.

[ppadala]

I made the pull request, any comments and reviews are appreciated.

There are still a couple of hard-coded items that need work, which I will be fixing in next few days.

[friism]

PR for reference: #3146

[ahmetb]

@ppadala really nice. I think @enderb-ms & docker-machine maintainers are currently trying to figure out if they should use WinRM or bootstrap OpenSSH on Windows. I'll let those folks to decide which work should go forward.

As far as azure driver concerned, I'm afraid we can't take changes to azure driver. I just rewrote the azure driver with Azure Resource Manager stack (it's a huge breaking change). Details are at: #2742 (comment) we're currently addressing flakiness issues in Azure SDK, once those are done I'll be sending a PR to release the new azure driver.

[ahmetb]

@ppadala it's also important that this probably should not be just an azure driver change as soon pretty much all cloud providers will have WS2016 images and (if required) docker-machine probably needs to add those windows-specific flags.

[ppadala]

@ppadala really nice. I think @enderb-ms & docker-machine maintainers are currently trying to figure out if they should use WinRM or bootstrap OpenSSH on Windows. I'll let those folks to decide which work should go forward.

I have actually considered the SSH option as well. I just didn't like the Windows OpenSSH server I tried, had trouble with connecting to the server from OS X. That said, I haven't tried all the possible SSH servers, you guys know better. Happy to jump in to the discussion, let me know where the discussion is happening.

As far as azure driver concerned, I'm afraid we can't take changes to azure driver. I just rewrote the azure driver with Azure Resource Manager stack (it's a huge breaking change). Details are at: #2742 (comment) we're currently addressing flakiness issues in Azure SDK, once those are done I'll be sending a PR to release the new azure driver.

No worries, the change I have done actually is not just to the azure driver, but also to rest of docker-machine to add an option for OS. I'll try out your code posted on #2742. Does the code already have support for spinning up Windows images on Azure?

[ppadala]

@ppadala it's also important that this probably should not be just an azure driver change as soon pretty much all cloud providers will have WS2016 images and (if required) docker-machine probably needs to add those windows-specific flags.

As I mentioned above, my change has both azure driver and rest of docker-machine changes. How shall we proceed from here? I can wait for your resource manager changes to be committed and then we can figure out adding windows-specific flags to docker-machine.

[ppadala]

@ahmetalpbalkan I checked the code you posted in #2742. As I understand, it's moving the driver to use ARM APIs and doesn't have Windows image support. Here's what I propose

1. Review and commit your code
2. Add spinning up Windows image support
3. Add the Windows provisioner using WinRM (or SSH if that's the choice we agree on), flags and other changes in docker-machine core

I have done 2 and 3 with current driver and I can make similar changes based on your ARM based driver. Let me know your thoughts.

[ahmetb]

@ppadala sounds good. if you try rebasing your code on top of my branch, you can get unblocked and get started doing some work. Functionality-wise the new driver is ready, but we're still addressing some issues in azure-sdk-for-go and these may still take some time (like a couple weeks) and I'll be sending the end result as a pull request.

[enderjbr]

@ppadala this is awesome! Thank you for doing this. Unfortunately, I have not been able to get any developer time, so we have not starting working on our ideas. But this might help me get people onboard.
One of the reasons we want to use SSH to be able to allow developers in other platforms (Linux and OS X) to be able to deploy Windows container hosts in VMs in their local machines or in the cloud. We have started talking about possibly including OpenSSH in some Azure images, that way no extra steps would be needed to add that capability. Could you tell me more about the issues you saw with OpenSSH?
Tagging @dgageot who also had some thoughts about WinRM.
Again, this is great, thanks for doing this @ppadala

[ppadala]

@ppadala this is awesome! Thank you for doing this. Unfortunately, I have not been able to get any developer time, so we have not starting working on our ideas. But this might help me get people onboard.
One of the reasons we want to use SSH to be able to allow developers in other platforms (Linux and OS X) to be able to deploy Windows container hosts in VMs in their local machines or in the cloud. We have started talking about possibly including OpenSSH in some Azure images, that way no extra steps would be needed to add that capability. Could you tell me more about the issues you saw with OpenSSH?

I was setting up https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH as the SSH server. When I run ssh <server> <cmd> from Mac, it doesn't seem to work, though plain interactive ssh to the server works.

If you include SSH server in Azure images, then that's definitely cleaner. docker-machine driver interface already seems to assume that SSH access is available.

[ahmetb]

@ppadala I'm puzzled, too. I bet the API is trying to say you can't specify extensions on the VM create call. For instance when you deploy an ARM template with extensions and watch the deployment happening, ARM first creates the VM and then creates the Extension resource separately once the VM is created...

There must be some methods to add extensions to the VMs in the arm/compute package. I suggest adding extensions once the VM is created, that should unblock you.

[ppadala]

Thanks @ahmetalpbalkan. I added VirtualMachineExtension after the VM creation and it worked. I had to wrestle a little bit to figure out the right way to setup WinRM HTTPS listeners, but I have the pull request #3329 ready. Let me know your comments.

I also uploaded the binaries at https://www.dropbox.com/sh/kpc46xtc8su2sj7/AABtxVIVbGo8KLy7tu6HnWwza?dl=0 for easy testing.

Usage: Three new parameters are added to Azure driver

--azure-os "linux"                  OS for the Azure VM (Windows|Linux)
--azure-winrm-user "docker-user"    Username for WinRM login [$AZURE_WINRM_USER]
--azure-winrm-password              Password for WinRM login [$AZURE_WINRM_PASSWORD]

Sample command

docker-machine -D create -d azure --azure-subscription-id $(cat ~/azure/subid) --azure-os "windows" --azure-image MicrosoftWindowsServer:WindowsServer:2016-Technical-Preview-with-Containers:2016.0.20151118 --azure-winrm-password "mysecretpassword" vhost-win-test

[friism]

@ppadala @ahmetalpbalkan I don't know if you already incorporated this, but this article on securing the windows daemon may also be good: https://github.com/Microsoft/Virtualization-Documentation/tree/master/windows-server-container-tools/DockerTLS

/cc @PatrickLang @swernli

[StefanScherer]

@ppadala Thanks, this is totally awesome. I've just tried the sample command to create a TP4 machine in Azure. And yeah, it worked with your prebuilt binaries right out of the box. (ok, you have to enter a more complex password to fit the windows password complexity requirements)

A secured Docker Engine with such a simple docker-machine command. That's a great breakthrough!

$ ./docker-machine-Darwin-x86_64 -D create -d azure --azure-subscription-id $(pass azure) --azure-os "windows" --azure-image MicrosoftWindowsServer:WindowsServer:2016-Technical-Preview-with-Containers:2016.0.20151118 --azure-winrm-password $(pass azure-winrm-password) vhost-win-test
$ eval $(./docker-machine-Darwin-x86_64 env vhost-win-test)
$ docker version
Client:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Fri Feb 12 22:11:40 UTC 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.10.0-dev
 API version:  1.22
 Go version:   go1.5.1
 Git commit:   59a341e
 Built:        Fri Nov 13 17:23:07 UTC 2015
 OS/Arch:      windows/amd64

[StefanScherer]

@ppadala And what's even more fantastic is that a docker-machine rm -f vhost-win-test cleans up EVERYTHING - I struggled so many times cleaning up all the parts of a VM created with "deploy to azure", always missed a "undeploy from azure" button 😄

[ppadala]

@ppadala @ahmetalpbalkan I don't know if you already incorporated this, but this article on securing the windows daemon may also be good: https://github.com/Microsoft/Virtualization-Documentation/tree/master/windows-server-container-tools/DockerTLS

Thanks for the link @friism. Yes, the docker-engine is already secured with TLS similar to Linux. docker-machine create workflow generates the credentials (cert, key and ca) and I use WinRM to upload them to Windows Server.

[ppadala]

@ppadala Thanks, this is totally awesome. I've just tried the sample command to create a TP4 machine in Azure. And yeah, it worked with your prebuilt binaries right out of the box. (ok, you have to enter a more complex password to fit the windows password complexity requirements)

Thanks for trying @StefanScherer. The plaintext password is the only part I don't like here (I like your pass approach). We can switch to a cert based auth later. Unfortunately packer library doesn't support that yet.

[ppadala]

@ppadala And what's even more fantastic is that a docker-machine rm -f vhost-win-test cleans up EVERYTHING - I struggled so many times cleaning up all the parts of a VM created with "deploy to azure", always missed a "undeploy from azure" button 😄

Credit goes to @ahmetalpbalkan for moving to Azure ARM and writing a clean "cleanup".

[ahmetb]

@ppadala this is feakin' amazing. Great work! I guess the question now will be how to merge this to azure driver in the short and to other drivers (such as google/aws) in the long term.

[ppadala]

@ahmetalpbalkan the pull request #3329 is already synced with master and merged with azure driver, will be great if you can review and provide comments. Currently, only Azure has Windows server 2016 VMs, but the code is written in such a way that support for GCP, AWS, vSphere and others easily.

[srounce]

Currently, only Azure has Windows server 2016 VMs, but the code is written in such a way that support for GCP, AWS, vSphere and others easily.

• @ppadala
  Does this mean that supporting local vm providers is on the horizon?

[ahmetb]

@ppadala @srounce I would say somebody should take a look at WinRM and Startup Script execution capabilities of AWS/GCE/others APIs to find out if the functionality can be ported to other drivers with minimal set of changes.

[nilleb]

@ppadala @ahmetalpbalkan Could the docker-machine 'azure' driver be used with AzureStack?

[ahmetb]

@nilleb I'm afraid not yet. If you need it, please open a separate issue.

[SharadGumaste]

I just downloaded the docker for windows that does have the 1.12 RC2 that has the 0.8.0-rc1 but when i try the "azure-os" "windows" option it does not seem to work. Please let me know where i could acquire a docker-machine version that can create TP5 Windows 2016 docker hosts on Azure.

Thanks,

[ahmetb]

@SharadGumaste what's the error when you say "it does not work"? have you tried specifying the image with --azure-image?

[ppadala]

I just downloaded the docker for windows that does have the 1.12 RC2 that has the 0.8.0-rc1 but when i try the "azure-os" "windows" option it does not seem to work. Please let me know where i could acquire a docker-machine version that can create TP5 Windows 2016 docker hosts on Azure.

@SharadGumaste I am guessing you were trying the released version. The patch for supporting Windows (azure-os windows option) is not merged yet.

[SharadGumaste]

Pradeep, yes I did the update from the docker software. What is the best
way to get binaries of your work? I am very close to doing a demo for a
client of ours except I can't create a barebone Windows server and install
docker host related stuff from scratch and add that machine into
docker-machine and build and deploy from Visual Studio.
I am stuck at TLS certs etc...

Thank you in advance for any assistance.

On Thursday, June 23, 2016, Pradeep Padala notifications@github.com wrote:

I just downloaded the docker for windows that does have the 1.12 RC2 that
has the 0.8.0-rc1 but when i try the "azure-os" "windows" option it does
not seem to work. Please let me know where i could acquire a docker-machine
version that can create TP5 Windows 2016 docker hosts on Azure.

@SharadGumaste https://github.com/SharadGumaste I am guessing you were
trying the released version. The patch for supporting Windows (azure-os
windows option) is not merged yet.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2907 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AH3YqqQLlXWUnSdw4i2TzsqXa4gHmTVYks5qOxt1gaJpZM4HKwbT
.

"A common mistake people make when trying to design something

completely foolproof is to underestimate the ingenuity of complete fools."

– Douglas Adams, author (1952 – 2001)

[ppadala]

@SharadGumaste can you send a mail to pradeep@containerx.io? I have a separate repo for docker-machine for Windows, I can send you instructions on setting up.

[bazzilic]

Are there any news regarding spinning up a Windows Docker host in a local VirtualBox?

[djanosik]

Does anyone have a prototype able to spin up Windows VMs on Azure?

[davidarcher]

@djanosik I am working on trying to get docker-machine to spin up autoscaling Windows OpenStack and EC2 instances for my GitLab CI builds. I took @ppadala 's PR branch, got it rebased up to the latest master, and am studying it now. It's in https://github.com/davidarcher/machine/tree/azure_winrm .

[thejohnfreeman]

I followed @jen20's instructions to build a Windows AMI with SSH. It works with Docker Machine up to the point of provisioning, where it fails because there is no /etc/os-release file. Now that SSH on Windows is available (with some effort), can we get an SSH-based Docker Machine provisioner for Windows?