CLI Plugins Design

[ijc]

I am about to start work on a plugin mechanism for the Docker CLI and wanted to share the intended design here first.

Naming

The name plugin is already in long standing and well established use for engine side plugins and so a new name must be found for CLI plugins.

This document uses cli-plugin(s) throughout (e.g. for command names, packages, paths to search etc), but I expect that may change.

High-Level Overview

The Docker CLI will support a plugin/extension mechanism which should be familiar to users of git or kubectl or many other similar commands.

The top-level docker CLI will search a specific set of paths for binaries with names matching the regexp docker-[a-z][a-z0-9]* (that is, anything with a docker- prefix and one or more alphanumerics (lowercase) in the name, with a leading letter). For any such binary which is found and which is deemed to be acceptable a new top-level command will be exposed by the CLI so a binary named docker-foo would produce a foo command named at the top-level. This would then be callable as docker foo «further subcommands and options».

In this document when we refer to the “plugin name” we mean the unprefixed name unless otherwise specified (so “foo” in the example above).

Top-Level CLI Behaviour and Integration

Discovery and Validation of Installed Plugins

To enumerate all available plugins the following paths (decreasing order of priority) are scanned:

• Unix-like OSes :
  • $HOME/.docker/cli-plugins
  • /usr/local/lib/docker/cli-plugins & /usr/local/libexec/docker/cli-plugins
  • /usr/lib/docker/cli-plugins & /usr/libexec/docker/cli-plugins
• On Windows:
  • %USERPROFILE%\.docker\cli-plugins
  • C:\ProgramData\Docker\cli-plugins

(NB1: unlike git and similar this is not the same as $PATH, for our use cases we don't have the historical precedent of supporting invoking subcommands as docker-foo directly, and we have other preexisting commands, such as docker-compose, which will be in $PATH but are not (yet?) plugins)
(NB2: lib vs libexec represents a split in what distros use for this purpose)

Any files or symbolic links found which match the regex docker-.+ (basically anything at all with the docker- prefix) are considered potential plugin "candidates". Anything found which is not a regular file or symbolic link or which does not match the pattern is ignored and is not considered as a plugin candidate. On Windows files which do not have a .exe suffix are not considered candidates and are ignored.

To be considered a valid plugin a candidate must pass each of these “plugin candidate tests”:

• Must have a name which matches the regex docker-[a-z][a-z0-9]*.
  • (NB: The reason for having a broad candidate list of files matching docker-.+ and then restricting here to just docker-[a-z][a-z0-9]* (instead of having that be the candidate list to start with) is that it causes docker cli-plugins list (see below) to list them and the reason they are not valid, which is useful for the user)
• Must not have a name which conflicts with an existing builtin top-level command.
• Must, where relevant, have appropriate OS “execute” permissions (e.g. Unix x bit set) for the current user.
• Must actually be executed successfully and when executed with the subcommand docker-cli-plugin-metadata must produce a valid JSON document (and nothing else) on its standard output (schema to be discussed later).

A candidate in a higher priority directory path always shadows a candidate in a lower priority directory path, even if the higher priority candidate is found to be lacking in some way (e.g. is not executable).

Running a Plugin

When the Docker CLI is run with an unknown top-level command it will try each of the paths enumerated in “Discovery and Validation of Installed Plugins” looking for the first candidate plugin with the correct name (with the docker- prefix).

If no such candidate is found then this is considered the same as if a builtin subcommand is not found today. That is it will print an error message and exit with code 1.

$ docker unknowncommand
docker: 'unknowncommand' is not a docker command.
See 'docker --help'
$ echo $?
1

The candidate will then be subjected to the same set of “plugin candidate tests” as described above. If it fails any of these tests then this is considered an error and the CLI will exit with an appropriate error message. Note: This will include executing the plugin with the docker-cli-plugin-metadata subcommand in order to retain consistency with enumeration:

$ docker invalidplugin
CLI plugin “invalidplugin” is invalid: some reason
$ echo $?
1

At this point the plugin is executed, all parameters (including global parameters) will be passed in the same order as they appeared on the top-level CLI’s command line. This includes repeating the plugin name, so running docker --global foo --bar results in docker-foo --global foo --bar being executed.

Management of Plugins

No new commands will be added to the CLI binary for the management of CLI plugins. However some modifications will be made to existing top-level commands to accommodate them.

docker help (with no arguments) and docker --help

These existing top-level builtin commands will be modified to integrate valid plugins into the existing lists of builtin commands which they output. Plugins will be listed in the existing “Commands” category and not separated out into “plugins” (a list specifically of plugins can be obtained with “docker plugin ls”). The description will be taken from the plugin metadata.

The existing columnar listing of «command» «description» will be augmented with a new vendor column: «command» «vendor» «description» where the vendor column will either read Builtin or be
taken from the plugin metadata. The vendor will be truncated at 12 characters.

If there are any broken plugins then a new section Invalid plugins will be present in the docker help output which will list them and the reason they are invalid. If there are no invalid plugins then this section is omitted.

docker help «command»

If «command» is not a built-in command then this will be handled by looking up «command» as described in Running a Plugin and the invoking as docker-«command» help «command».

docker info

A list of valid plugins (including vendor, version and short description) will be incorporated into the "pretty" output. A warning will be issued for any invalid plugin, including the reason the plugin was considered invalid.

When using docker info -f «format» an array of all (valid and invalid) plugins will be included as a new CLIPlugin top-level entry. For valid plugins an entry will contain the full set of plugin metadata. For an invalid plugin only Name, Path and Err (the reason the plugin is invalid) are guaranteed be present.

Support for Installing Plugins

Users may install a new plugin by simply dropping the binary into an appropriate path (and, where necessary making it executable).

Requirements Placed on Plugins

A library will be provided which takes care of much of the boring detail of what is required here in order to avoid duplicating code in every plugin. See Plugin Support Framework/Libraries below.

The high-level requirements for a plugin are:

• All plugins must support the docker-cli-plugin-metadata subcommand as described in Discovery and Validation of Installed Plugins. The subcommand should be hidden (omitted from help and similar). The schema is described below in Plugin Metadata.
• All plugins must accept the global Docker CLI options and related environment variables and, where those options are relevant to its operation, honour them.
• All plugins should use the same configuration and authorization mechanisms as the top-level CLI (i.e. read from the same configuration files, honour the same top-level configuration options etc).
• Plugins are responsible for supporting different endpoint versions on servers they interact with (so e.g. are responsible for negotiating Docker API version themselves).

Plugin Metadata

When run with the docker-cli-plugin-metadata subcommand a plugin must produce a JSON document containing metadata about the plugin. This shall contain at least:

• Plugin Metadata Schema Version
• Name
• Version
• Author/Vendor
• URL
• Short description

The full schema will be determined (and documented) as part of the implementation work.

Configuration

The model to be used here is that all plugins (and the main CLI binary) have access to the same set of configuration files and each plugin will be able to read and use that configuration independently. Plugins will search for a config file in the same places as the monolithic CLI.

Plugins are expected to make use of existing global configuration where it makes sense and likewise to consider extending the global configuration where that is sensible. In particular for endpoint configuration they are expected to use the context store.

Where plugins unavoidably require specific configuration the .plugins.«name» key in config.json is reserved for their use.

Plugin Support Framework/Libraries

In order to make it easy to write plugins without reinventing numerous wheels and as an aid to consistency various Go helper packages will be provided. The need for some of these will only become apparent as work progresses to produce various plugins so it is expected that this will iterate.

Overall these helpers are expected to fall into two broad categories.

Plugin Specific

Some library functionality will be specific to the needs of plugins. These will live in github.com/docker/cli/plugin.

Functions will include:

func New(cmd cobra.Command, m Metadata)

This is intended to be call from a plugin's main function will take care of the scaffolding required to register the command, handling supporting the required docker-cli-plugin-metadata subcommand as well as the global arguments.

The cmd argument represents the plugins toplevel command (the name will thus be found in cmd.Use). The m argument will be a struct containing other useful metadata (versioning information etc)

CLI Specific

Some library functionality will not be especially specific to plugins but will be common to the main top-level CLI binary as well. For example:

• configuration file parsing, lookup of options by name.
• credential cache / registry authentication.
• contexts and endpoints (e.g. context store added by fast context switching, see ). In this context it is expected that the various server client libraries will provide integration with the context store, so that e.g. the same endpoint key names are used by all plugins (or builtins) which want to talk to a given service
• Table formatting.

It is expected that much of this code already exists within the current CLI but that it may not be structured in a way which makes it easily or cleanly usable by plugins, which basically amounts to ensuring that relevant bits of the CLI code are arranged as a cleanly consumable packages. Where necessary this refactoring will be done and/or new code written. Most likely ending up in subpackages of github.com/docker/cli details TBD as part of implementation but expect e.g. config and auth and similar.

[thaJeztah]

@ijc discussing this with @tonistiigi and @cpuguy83 and we're wondering what is done with global flags; are those passed to the plugin? (e.g. docker -H foobar <some plugin command>)

[andrewhsu]

Would be good to have a working public example or mock up of what this can help accomplish. To help walk through the process of the interaction between the cli and the plugin. It will help validate the interface provides the needed information.

[thaJeztah]

@ijc also (question from Tonis); what's the purpose of the docker cli-plugin ls and docker cli-plugin inspect commands? (given that there's no actual management commands (yet)?

If you want info about a plugin, docker <plugin> --help could provide information

[cpuguy83]

I'm interested in how is the thing that's plugging in going to benefit from being a plugin? What does the core docker CLI provide to it that makes it beneficial to be a plugin?

[ijc]

we're wondering what is done with global flags; are those passed to the plugin? (e.g. docker -H foobar )

The design states:

All parameters (including global parameters) will be passed in the same order as they appeared on the top-level CLI’s command line. This includes repeating the plugin name, so running docker --global foo --bar results in docker-foo --global foo --bar being executed.

---

Would be good to have a working public example or mock up of what this can help accomplish. To help walk through the process of the interaction between the cli and the plugin. It will help validate the interface provides the needed information.

The PR will have both real and mocked example plugins (needed for unit and e2e testing also). I think it would be counterproductive to spend a lot of time trying to nail down the specifics in the abstract at this phase, since there is lots we can and will learn during the implementation about the exact interactions. We should focus on getting the basics in, making use of them in real plugins and then iterating where we find issues.

---

what's the purpose of the docker cli-plugin ls and docker cli-plugin inspect commands? (given that there's no actual management commands (yet)?

I think ls is useful/necessary even in the absence of management commands, but in particular docker cli-plugin ls is also (uniquely) valuable in that it is the place which will include invalid plugins to help diagnose why an expected plugin is not showing up (e.g it will be where it says things like "not executable" or "invalid metadata").

While bunch of the metadata exposed by docker cli-plugin inspect could be exposed by docker <plugin> --help I think it is best to reserve the latter primarily for information on actual usage of the plugin rather than metainformation about the plugin.

---

I'm interested in how is the thing that's plugging in going to benefit from being a plugin? What does the core docker CLI provide to it that makes it beneficial to be a plugin?

For something like docker/app (which we intend to be one early consumer of this functionality) it means that it integrates into the CLI UX more nicely but without tying the development cycles of app and cli together. It's basically the same as 3rd party plugins to any similar system such as git or kubectl.

(edit: added breaks between the different questions, mainly because when rendered my quote of the document ran into my quote of Andrew's question in an unhelpfully confusing way)

[thaJeztah]

The design states

Ah. sorry for that; I was multi-tasking, and typing that during the maintainers meeting, and couldn't find it at a glance

[tonistiigi]

I think ls is useful/necessary even in the absence of management commands, but in particular docker cli-plugin ls is also (uniquely) valuable in that it is the place which will include invalid plugins to help diagnose why an expected plugin is not showing up (e.g it will be where it says things like "not executable" or "invalid metadata").

Presumably, we would list them in docker --help and plain docker anyway. And could show the errors there the same way.

While bunch of the metadata exposed by docker cli-plugin inspect could be exposed by docker --help I think it is best to reserve the latter primarily for information on actual usage of the plugin rather than metainformation about the plugin.

We could just put the metainformation in the beginning of <plugin> --help (if this info is needed at all).

It might we worth considering using the new dial-stdio command for providing the connection to the daemon in plugins to reduce the duplicate code and make sure all plugins always work with all connection types and are forward compatible in this regard.

[ijc]

We could just put the metainformation in the beginning of <plugin> --help

When I said "exposed by docker --help" I actually meant docker plugin --help, so I think my preference to keep that for the plugin usage and metadata elsewhere still applies, the metadata will mostly be a distraction to people who just want the help info.

(if this info is needed at all).

It has version and vendor info etc, so I think it is. It also lists the actual path to the plugin, plus any other paths which were shadowed by it, which is useful for diagnosing issues where a user isn't using the plugin they think they are.

It might we worth considering using the new dial-stdio command for providing the connection to the daemon in plugins to reduce the duplicate code and make sure all plugins always work with all connection types and are forward compatible in this regard.

Interesting, yes that seems like a useful thing to pursue. It would mostly just mean a wrapper in some library to help plugins with calling it consistently.

[tonistiigi]

the metadata will mostly be a distraction to people who just want the help info.

Or we could just make a convention that docker <plugin> --version shows the metadata if we want to keep it separate from --help. Doesn't seem that much different from docker cli-plugin inspect <plugin> from user's point of view.

[ijc]

I suppose we could, but inspect and ls are available for most other objects the CLI deals with and I'm not sure why CLI plugins should be special. Also by forcing it into some other output (like --help or --version) you would lose useful things which inspect gives you like machine readable JSON output, --format etc.

[tonistiigi]

I suppose we could, but inspect and ls are available for most other objects the CLI deals with and I'm not sure why CLI plugins should be special.

I guess the issue here is about why cli plugins are considered objects instead of like any other command that is just loaded differently. My assumption is that users would not make this distinction and would want to use them using same logic they use for other commands.

Also by forcing it into some other output (like --help or --version) you would lose useful things which inspect gives you like machine readable JSON output, --format etc.

docker version supports format as well and can provide json output so isn't a unique concept to inspect.

[ijc]

I guess the issue here is about why cli plugins are considered objects instead of like any other command that is just loaded differently.

The main difference is that the set of commands available as plugins is not fixed and in particular unlike a builtin command their installation/availability can be broken in subtle ways (e.g. by just failing to set the x permission bit). Treating them as objects allows users to inspect and diagnose this (and I don't expect users will interact with these commands very much at all unless something is not working for them).

My assumption is that users would not make this distinction and would want to use them using same logic they use for other commands.

For using them this is true a plugin foo is used with docker foo and it has help docker foo --help and docker help foo which tell you about how to use them, like any other command.

But aside from actually using them there are other operations on CLI plugins which I think are useful but not related to using them which shouldn't be conflated -- i.e. inspecting their properties and validating their availability for which I think it makes sense to treat them as objects.

Right now what I have is something like:

$ docker cli-plugin list
NAME                VENDOR              DESCRIPTION                            STATUS
badmeta                                                                        invalid metadata: invalid character 'i' looking for beginning of object key string
helloworld          Docker Inc.         A basic Hello World plugin for tests   ok
unexec                                                                         unable to execute: exit status 127
$ docker cli-plugin inspect helloworld
[
    {
        "Version": "0.1.0",
        "Vendor": "Docker Inc.",
        "ShortDescription": "A basic Hello World plugin for tests",
        "SchemaVersion": "",
        "Name": "helloworld",
        "Path": "/home/ijc/development/.../docker-helloworld"
        // If there were other `docker-helloworld`'s in the plugin search path, then
        // they would be listed here as being shadowed.
    }
]
$ docker help 
Usage:	docker [OPTIONS] COMMAND

A self-sufficient runtime for containers
... (nb: badmeta doesn't appear here because it is invalid) ...
  exec                      Run a command in a running container
  export                    Export a container's filesystem as a tar archive
  helloworld  (Docker Inc.) A basic Hello World plugin for tests
  history                   Show the history of an image
  images                    List images
... (nb: neither does unexec appear) ...
$ docker helloworld
Hello World!
$ docker help helloworld
A basic Hello World plugin for tests

Usage:
  docker helloworld [flags]

Flags:
  -h, --help   help for helloworld

This all seems quite straight forward and natural to me and is analogous to how other things work.
What are the downsides of treating them as objects? What problems or issues are we trying to avoid by not doing so?

[cpuguy83]

I have some unfinished thoughts on this I'd like to get back to likely after kubecon.

I can see the usefulness of this, particularly to ship some experimental thing that you don't want to have to give out a custom docker cli binary for.

Would this support adding new flags to an existing command?

What about adding a command that is not a top-level one?

Basically, what all would a cli plugin support?

Can we think about the UX for this a bit more? A "cli-plugin" subcommand does not look right and is horribly inconsistent.

[ijc]

Would this support adding new flags to an existing command?

I can't see a sane way to make add a new --foo flag to an existing command in the general case, so by extension not in the case of this framework either.

What about adding a command that is not a top-level one?

That is not currently supported but one of the main reasons I outlawed - in a plugin name is to keep a path to supporting docker-command-subcommand available as a mechanism for implementing docker command subcommand as a plugin.

A "cli-plugin" subcommand does not look right and is horribly inconsistent.

I am not at all wedded to the name, if that's what you mean, its pretty awful and I'm happy to take the maintainer's decision on what else to call it if you all can come to a consensus.

If it's not just the name you don't like then you'll need to be much more explicit about what doesn't "look right" and is "horribly inconsistent" I'm afraid.

[tonistiigi]

The main difference is that the set of commands available as plugins is not fixed and in particular unlike a builtin command their installation/availability can be broken in subtle ways (e.g. by just failing to set the x permission bit).

Like I mentioned before we already have a view of showing all supported commands. And they are even already grouped into different sections atm (plugins can be another group). I see no issue with showing these errors in that view as well. In fact, I think it is better because if there is a configuration issue that user isn't aware of it is very unlikely that user would ever run cli-plugin ls while users already know how to list all available commands.

What are the downsides of treating them as objects? What problems or issues are we trying to avoid by not doing so?

I think it would be better to avoid introducing new concepts if there isn't a reason why they are mandatory for making the use cases work. Atm we are not sure if this evolves to some more complex plugin system (that would need a concept of "plugin object") so it's hard to know if these parts of the proposal are forward compatible.

[ijc]

OK, I'll list failed plugin candidates in docker help.

plugins can be another group

I think that would be a mistake for "successful" plugins, they should be integrated into the normal lists of commands, but for failed ones it mights make sense, I'll play with both (I'm leaning towards separate though).

I think it would be better to avoid introducing new concepts if there isn't a reason why they are mandatory for making the use cases work

They aren't mandatory but I do think they would be useful for supportability and maintainability purposes. However since it seems I'm in the minority, or at least there are significant objections from maintainers, I will just drop this side of things. We can revisit another time if there is appetite.

[cpuguy83]

I can't see a sane way to make add a new --foo flag to an existing command in the general case, so by extension not in the case of this framework either.

That's fair, I just want to make sure we have a clear scope for such cases.

---

In terms of a protocol for interacting with plugins the proposal seems rather light.
A simple protocol like (e.g. "run this binary and exit code != 0 means error") is nice, but also limiting.

Some thoughts on some commands that could be implemented by a plugin (naming is whatever):

• info - Get plugin metadata
• run - Executes the actual plugin code

Mainly worried about painting ourselves into a corner in terms of the kind of plugins that can be supported and how to add new kinds in the future... e.g. maybe we want to add support for longer running plugins that interact with built-in commands (in which case there could be an init and a close command on the plugin)

Also needs more details on how things like config are passed through.
Will the docker CLI pass in a file descriptor(s) for, e.g., accessing the daemon, cred store, etc?

[ijc]

me thoughts on some commands that could be implemented by a plugin

What you describe is exactly what the proposal contains (see Discovery and Validation of Installed Plugins and Requirements Placed on Plugins in the proposal). What you call info is docker-cli-plugin-metadata and for run we use the name of the plugin itself (which simplifies various things implementationwise with general frameworks e.g. cobra since then their usual help text generation etc just does the right thing and fits in). There is plenty of subcommand namespace left for future evolution.

Please have a play with #1564 which implements the core bits of this.

Also needs more details on how things like config are passed through.
Will the docker CLI pass in a file descriptor(s) for, e.g., accessing the daemon, cred store, etc?

No, the CLI will provide (in many cases already provides) library packages which covers this functionality to allow plugins to interact in a consistent way with the core facilities, including daemon connection, creds and config etc. See Configuration and Plugin Support Framework/Libraries in the proposal.

In the future we could choose to make those libraries instead be clients for some sort of other scheme based on RPC but there seems to be no immediate need for that level of complexity.

[cpuguy83]

What you describe is exactly what the proposal contains

Yes, I understand this, but it looks underspecified here.

[tonistiigi]

They aren't mandatory but I do think they would be useful for supportability and maintainability purposes. However since it seems I'm in the minority, or at least there are significant objections from maintainers, I will just drop this side of things.

@ijc Can you update the first post if this is out now

[ijc]

@tonistiigi I've reworked the ## Management of Plugins section to reflect that change.

I also incorporated the changes to docker info which have been discussed with @silvin-lubecki and @thaJeztah into that section.

[dweomer]

To avoid confusion when talking/writing about this, I like the idea of calling them CLI Drop-ins or just dropins.

[thaJeztah]

To avoid confusion when talking/writing about this, I like the idea of calling them CLI Drop-ins or just dropins.

"extensions" could've been an alternative, but naming is hard, so I'm not sure what's the best name for them ("plugins" work, but I can see the possible confusion)

[ijc]

hopefully "what a cli extension is" can be narrowed down to something that doesn't require more than just a process cmdline convention?

docs/extend/cli_plugins.md describes just such a process cmdline convention.

The CLI repo provides a library/framework to (hopefully) make following the convention easier (at the expense of a raft of vendoring) if you are writing in Go and using Cobra but there is no actual requirement to use it (or Go, or Cobra) if you don't like that tradeoff.

That's not to say we should refactor things to make the dependency set for the helpers smaller, that's a worthwhile thing irrespective of the above.

[SvenDowideit]

oh wow, i looked for docs, and failed to find it - thanks @ijc

I guess the next thing will be figuring out how to build a docker-cli that has support - its been many many years since I built Docker myself :)

[SvenDowideit]

and to follow up - building the docker cli is so very much less of a deal than it was.

[SvenDowideit]

and for the trivial bash example - https://github.com/SvenDowideit/docker-helloworld/blob/master/docker-env

I wonder if it would be nice to have an absolute path in DOCKER_CLI_PLUGIN_ORIGINAL_CLI_COMMAND=build/docker-linux-amd64 - IDK.

[lukaszlach]

Thank you for this feature! I have created a plugin for creating, building, installing and managing other plugins. Shipps with plugins like dive (explore image layers), expose (expose container on the Internet), publish (publish port on the running container), showcontext (display contents of the build context).
https://github.com/lukaszlach/clip

[solvingj]

This is indeed a very exciting project for people making tools that would interop with docker.

Another tools related project just added a "Plugins" feature which they later renamed to "Hooks", because it's essence was running custom tools before and after various steps in the tools process. It seems a reasonable suggestion here. "Docker Hooks".

[AkihiroSuda]

The issue is closable?

[solvingj]

So, I found this list of CLI plugins in a PDF from dockercon, but can't find a proper web page which holds a similar listing anywhere. Furthermore, I can't figure out how to find documentation for several of these plugins. Googling docker cli plugin jump and docker cli plugin pipeline isn't enough.

So, I would suggest discussing the discoverability model in this issue before closing. I know it's still super early and experimental, but since there are already 5 released in GA with docker enterprise 3, maybe it's not too early for this.

https://goto.docker.com/rs/929-FJL-178/images/DockerCon19-CLI-Plugin-Availability.pdf

[errordeveloper]

@thaJeztah @tonistiigi should this issue remain open?