Add Docker Machine driver plugins

[nathanleclaire]

Docker Machine Driver Plugins

Motivation

Many folks have expressed interest in writing or maintaining a driver for Machine but @ehazlett and I are incapable of reviewing and maintaining all of them ourselves. Additionally, we are a bottleneck for when people want to add features and release new versions of existing drivers. Therefore, we are moving to a plugin-based architecture.

Usage

From the end user's perspective, the Machine CLI will be pretty similar to how it always has, with the additional caveat that binaries for the desired plugins must be installed as well as the core docker-machine binary.

For the plugin developer and/or distributor, there are some considerations to keep in mind. All plugins must be Go binaries located in the end user's PATH with the name of docker-machine-driver-name. Once present, they will be usable.

To implement a plugin, the driver developer must make a new repository with the plugin code, which:

• includes a properly named module (e.g. virtualbox) which contains a struct that fulfills the Driver interface from github.com/docker/machine/libmachine/drivers
• includes a main module (I've chosen to make this in a directory called bin in the virtualbox example presented here for this) which calls plugin.RegisterPlugin() with an instantiated instance of the struct.

Example from VirtualBox:

package main

import (
        "github.com/docker/machine/drivers/virtualbox"
        "github.com/docker/machine/libmachine/drivers/plugin"
)

func main() {
        plugin.RegisterDriver(virtualbox.NewDriver("", ""))
}

Migrating an Existing Driver to a Plugin

This is for the time of writing and is likely to change, but I think through documenting things we can hopefully the most mundane details of this process through a script.

The most important / likely to trip up migrators details are:

• A lot of paths have changed, usually because the modules have been moved inside of libmachine. Some examples:
  • github.com/docker/machine/log => github.com/docker/machine/libmachine/log
  • github.com/docker/machine/utils => github.com/docker/machine/libmachine/mcnutils
  • github.com/docker/machine/drivers => github.com/docker/machine/libmachine/drivers
• The usage of cli.Flag in the drivers has changed to a Machine-specific model which is more RPC-friendly. Essentially:
  • GetCreateFlags is now a required method on the Driver itself, not a module-level function, which returns []mcnflag.Flag (a custom type encompassing all types of flags). All instances of cli.Flag should be replaced by the corresponding instance of mcnflag.Flag.
• It is no longer necessary to have the init block calling Register in the driver module
• The NewDriver "factory" method in the driver module is used to populate default settings.

Under The Hood

The way that plugins work is heavily inspired by Hashicorp's Terraform.

When a Machine command is invoked, the following happens:

1. Host struct(s) is loaded up from disk, and instead of loading the literal driver struct like in the past, a RpcClientDriver is loaded into the Driver field. This struct fulfills the Driver interface but is agnostic about the provider, it simply proxies methods to the RPC Driver server listening in the plugin binary.
2. An RPC plugin server (registered using the wrapper method detailed above) is spun up listening on an available TCP port on localhost, and the created RPC client connects to it.
3. Using the RPC client the configuration for the actual driver (running in the plugin binary) is set remotely.
4. When calls to the driver are made in the Machine code, they are sent over the wire to get a response from the plugin server.
5. When the host is saved to disk again, first it is ensured that the RawDriver is set as []byte to ensure that the configuration can be set over the network again on subsequent invocations.

Tooling

We don't have any tooling / default Machine plugin repository yet, but there are several things I'd like to see:

1. A base / example project at somewhere like https://github.com/docker/docker-machine-example which is easily forkable and has an example driver
2. A Makefile and some basic automation around compiling your driver and releasing the binaries to Github.
3. Detailed documentation explaining how to develop a driver and what the expectations are for each driver. For instance, we need to be explicit that ports should be open and available for 22, 2376, 3376 in the case of Swarm master, etc.

Caveat Emptor

Some of the underlying API / mechanics are likely to change, such as libmachine details. Ultimately, however, it is our goal to make migrating existing work that has been done on drivers to the plugin-based model as smooth as possible. To that end, we really want to get people playing with this ASAP, so feel free to reach out, comment, and attempt some implementations.

cc @ehazlett @dmp42 @tianon @SvenDowideit @bfirsh @mchiang @JeffDM

• N

[posita]

cc @legal90 (re: Parallels driver)

[thaJeztah]

All plugins must be Go binaries located in the end user's PATH

IIRC, for the Docker plugins some locations were defined that were checked in order of priority. Would this be something to consider (I.e., explicitly define locations where the plugins should be installed, instead of relying on PATH)?

[nathanleclaire]

IIRC, for the Docker plugins some locations were defined that were checked in order of priority. Would this be something to consider (I.e., explicitly define locations where the plugins should be installed, instead of relying on PATH)?

Maybe, but simple is always better. We don't really want to push the envelope too much for the initial implementation. If I understand correctly the Docker plugins will search for a socket listening at a particular path, and dictating a specific location for your program's socket seems more reasonable to me than using some kind of custom "machine PATH" for our binaries.

[thaJeztah]

Ah, you're right; it's the sockets that should be at a specific path for the Docker plugins. TL;DR my main objective is to keep consistency across the docker projects, where reasonably possible 👍

[dmp42]

C'mon Gordon, show some enthusiasm and be supportive for a change :-)

[zchee]

@nathanleclaire @posita FYI, I develop parallel driver port new libmachine mecanisum.
If @legal90 is busy to other work, it will publish my repository.

and, Also close have been some of the driver. #1626

[tianon]

Any particularly compelling reason you're vendoring codegangsta in such a strange way? Is there some change you wanted to make that the upstream project isn't keen on?

[nathanleclaire]

I have made a change to some of the command / flag parsing logic to address a bug in the library and this is intended as a temporary step. Ideally I'd like to submit the fix upstream and get it merged in.

[nathanleclaire]

urfave/cli#278

I do wish I could leave it out of the diff in this PR, though :(

[dnephin]

I think you can push to master on your fork and point at that instead, right?

github.com/nathanleclaire/cli

[nathanleclaire]

I could definitely do that, and arguably that'd be the right way to handle it. If the PR I mentioned above doesn't get merged in, that's likely what I'll end up doing.

[zchee]

@nathanleclaire @ehazlett
This is a proposal, What do you think of write the link to other driver repository in the docker/machine README?
Update will be hard, but more people that use the docker(docker-machine).

[nathanleclaire]

This is a proposal, What do you think of write the link to other driver repository in the docker/machine README?

I intend to have a list of drivers and links to their repositories in this repo, although I don't know if having them in the README directly is the right choice. Maybe a separate document that the README links to.

[legal90]

@zchee Thank you for suggestion, but we've just created an initial version of Parallels Driver: https://github.com/Parallels/docker-machine-parallels
It works fine with Docker Machine built from nathanleclaire@2f5b645

@rickard-von-essen, thanks for help!

[xiaohui]

@nathanleclaire I am also create an docker-machine plugin driver for UCloud(an aws-like IaaS provider in China), https://github.com/ucloud/docker-machine-ucloud .

It works well now with docker-machine built with nathanleclaire@0b4755f except the occasionally panic of ls command.

The callstack info of panic can be found at http://pastebin.com/99eTiK2w#, it looks like using nil pointer of SwarmOptions in HostListItem struct, any ideas?

[robhaswell]

Machine plugins are an excellent idea, thank you for this.

The Hashicorp RPC method is very advanced, but it is sadly very different to the existing Docker Engine plugin implementation (JSON over HTTP). Beautiful as the Hashicorp method is, I think it would be sad to have multiple plugin implementation methods across the Docker projects.

[nathanleclaire]

@robhaswell While I feel your pain on diverging plugin architectures, in our case we have an urgent need to (1) Ship a solution to this problem quickly and (2) Not require many updates to the existing drivers and/or driver development model, and neither of those requirements plays particularly nicely with making changes in upstream Docker. I'm trying to orient the plugin architecture to be extensible in the future and not too picky about implementation, so hopefully it's not out of the question to converge the two someday when they each mature a bit.

[nathanleclaire]

@legal90 @xiaohui Thanks for trying things out! Please report issues as they come up like you have been doing. I am also working on / publishing a boilerplate project which will cover some of the things you've already been following along with, such as compiling the plugins and releasing the software.

[nathanleclaire]

If you mean for the flags, they will just have to be updated to be typed as mcnflag.StringFlag etc. in GetCreateFlags. I will post the changes to enable this shortly, they're not on the remote yet.

[nathanleclaire]

LGTM

[thaJeztah]

It's merged! 🎉

[nathanleclaire]

🎊 🎉 😄

[zchee]

today is a beautiful day. 🎉

[hairyhenderson]

!!!! woo-hoo! awesome @nathanleclaire :D

[xiaohui]

🎉 awesome

[denverdino]

Cheers!

The Aliyun ECS driver has been migrated to the new plugin architecture and works well.
One more question: Is there any plan to provide the plugin repository? Thanks a lot

[nathanleclaire]

@denverdino Do you mean a curated list of links to plugin repositories?

[denverdino]

@nathanleclaire yes, if there is one repo/community to organize the plugins, that will be great.

[nathanleclaire]

@denverdino The current plan is to have some sort of PLUGINS.md file in the repo which plugin authors can fork and add as a link to their plugin's repository, if desired.

[denverdino]

@nathanleclaire, that will be good start. Looking forward to it. Thanks a lot

[nathanleclaire]

@denverdino 👍

[frapposelli]

🎉

[baldwinSPC]

@nathanleclaire @ehazlett how would contributors contribute documentation on their drivers whereby that documentation gets published to a spot like this on the docs site?

https://docs.docker.com/machine/drivers/aws/

[nathanleclaire]

@baldwinSPC We do not intend to host any documentation for drivers outside of the core drivers -- that will be up to the plugin maintainers (most cases a markdown file on Github should suffice). However, we most likely will add a list of available non-core plugins (with links) somewhere in this repo soon.