Fix broken swarm commands with Kubernetes defined as orchestrator #1137

silvin-lubecki · 2018-06-21T16:58:13Z

- What I did

All swarm commands (node, service, etc...) were erroring out if the orchestrator was defined as Kubernetes. I moved the orchestrator flag, the environment variable and the config file value to more locally values regarding stack command.

- How I did it

Renamed DOCKER_ORCHESTRATOR to DOCKER_STACK_ORCHESTRATOR
Renamed config file option "orchestrator" to "stackOrchestrator"
"--orchestrator" flag is no more global but local to stack command and subcommands
Cleaned all global orchestrator code
Replicated Hidden flags in help and Supported flags from root command to stack command

- How to verify it

$ DOCKER_STACK_ORCHESTRATOR=kubernetes docker node ls
ID                            HOSTNAME                STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
1u26o5yll4mdqxfr1o86xphb5 *   linuxkit-025000000001   Ready               Active              Leader              18.06.0-ce-dev

- Description for the changelog

Fixed broken swarm commands with Kubernetes defined as orchestrator
Renamed DOCKER_ORCHESTRATOR to DOCKER_STACK_ORCHESTRATOR
Renamed config file option "orchestrator" to "stackOrchestrator"
"--orchestrator" flag is no more global but local to stack command and subcommands

- A picture of a cute animal (not mandatory but encouraged)

silvin-lubecki · 2018-06-21T16:59:13Z

Please review with attention stack/cmd.go, and if you can make some manual tests too, it would be great 😄

mat007

A few comments!

mat007 · 2018-06-21T17:31:48Z

cli/command/stack/cmd.go

+			return
+		}
+		if _, ok := f.Annotations["kubernetes"]; ok && !orchestrator.HasKubernetes() {
+			errs = append(errs, fmt.Sprintf("\"--%s\" is only supported on a Docker cli with kubernetes features enabled", f.Name))


nit: back quotes would remove the need to escape the double quotes:

errs = append(errs, fmt.Sprintf(`"--%s" is only supported on a Docker cli with kubernetes features enabled`, f.Name))

Yep, I just moved the code but you're right I'll fix it.

mat007 · 2018-06-21T17:35:29Z

cli/command/stack/ps.go

@@ -10,7 +10,7 @@ import (
 	"github.com/spf13/cobra"
 )

-func newPsCommand(dockerCli command.Cli) *cobra.Command {
+func newPsCommand(dockerCli command.Cli, common *commonOptions) *cobra.Command {


I don't think common needs to be a pointer here as it's always non-nil?

Well, it's because newPsCommand is called during command instantiation, but the common.orchestrator value will be updated in the stack/cmd.go:PersistentPreRunE. That's why it is passed as a pointer and not as a copied value.

mat007 · 2018-06-21T17:36:21Z

cli/command/stack/list.go

@@ -13,7 +13,7 @@ import (
 	"vbom.ml/util/sortorder"
 )

-func newListCommand(dockerCli command.Cli) *cobra.Command {
+func newListCommand(dockerCli command.Cli, common *commonOptions) *cobra.Command {


Same here, it doesn't seem common can ever be nil.

mat007 · 2018-06-21T17:37:50Z

cli/command/stack/cmd.go

 )

 var errUnsupportedAllOrchestrator = fmt.Errorf(`no orchestrator specified: use either "kubernetes" or "swarm"`)

+type commonOptions struct {
+	orchestrator command.Orchestrator
+}


Why do we need this type? Can't we use command.Orchestrator directly?

I need to update the value while resolving the orchestrator (conf file -> env -> flag), but we need to wait for the PersistentPreRun to be able to read the orchestrator flag value.

mat007 · 2018-06-21T17:39:15Z

cli/command/stack/list_test.go

@@ -14,6 +15,10 @@ import (
 	"gotest.tools/golden"
 )

+var (
+	orchestrator = commonOptions{orchestrator: command.OrchestratorSwarm}


orchestrator is actually a commonOptions?

Yes I know, I was lazy... It was a first attempt with orchestrator, didn't want to rename 😅

mat007 · 2018-06-21T17:40:09Z

cli/command/system/version.go

@@ -121,13 +121,19 @@ func reformatDate(buildTime string) string {
 	return buildTime
 }

+// nolint: gocyclo


Yeah, don't know why...

mat007 · 2018-06-21T17:42:21Z

cli/command/system/version.go

-		kubeVersion := getKubernetesVersion(dockerCli, opts.kubeConfig)
+		var kubeVersion *kubernetesVersion
+		if orchestrator.HasKubernetes() {
+			kubeVersion = getKubernetesVersion(opts.kubeConfig)


nit: or we could pass orchestrator to getKubernetesVersion instead of kubeConfig

Why instead of? I think getKubernetesVersion needs the kubeConfig to instantiate the kube client.

Sorry got mixed up, I meant instead of the dockerCli which was passed before.

mat007 · 2018-06-21T17:53:38Z

cli/command/stack/kubernetes/cli.go

@@ -46,7 +46,7 @@ func AddNamespaceFlag(flags *flag.FlagSet) {
 }

 // WrapCli wraps command.Cli with kubernetes specifics
-func WrapCli(dockerCli command.Cli, opts Options) (*KubeCli, error) {
+func WrapCli(dockerCli command.Cli, opts Options, orchestrator command.Orchestrator) (*KubeCli, error) {


Can't orchestrator be put inside Options somehow?

I'll try that!

silvin-lubecki · 2018-06-21T18:27:52Z

The metalinter is not happy, nor the unit testing... I'll fix this!

silvin-lubecki · 2018-06-21T19:31:19Z

PTAL

antonybichon17 · 2018-06-21T20:25:57Z

I see all those testOrchestrator being removed but nothing replacing them... or am I missing something?

silvin-lubecki · 2018-06-21T20:41:03Z

Tests were moved here

thaJeztah

left some comments for discussion; trying to give it a spin, but nuked my k8s, lol

thaJeztah · 2018-06-21T17:14:23Z

cli/command/orchestrator.go

@@ -17,10 +17,25 @@ const (
 	OrchestratorAll   = Orchestrator("all")


Looks like these constants can be un-exported? (and the GoDoc removed)

Nevermind; I see they're used in a test; can be fixed, but no urgency

thaJeztah · 2018-06-21T17:17:50Z

cli/command/stack/cmd.go

+	return command.GetStackOrchestrator(orchestratorFlag, config.StackOrchestrator)
+}
+
+func hideFlag(cmd *cobra.Command, orchestrator command.Orchestrator) {


hideOrchestrationFlags perhaps

thaJeztah · 2018-06-21T17:36:08Z

cli/command/system/testdata/docker-client-version.golden

+ Built:              Wed May 30 22:21:05 2018
+ OS/Arch:            linux/amd64
+ Experimental:       true
+ Stack Orchestrator: swarm


Looking at this now, a bit concerned how this will scale if support for more options is added 😞

Talked with @thaJeztah @dhiltgen and it seems more desirable to just get rid of the Orchestrator line for now and revisit.

thaJeztah · 2018-06-21T20:40:33Z

cli/command/orchestrator.go

@@ -17,10 +17,25 @@ const (
 	OrchestratorAll   = Orchestrator("all")
 	orchestratorUnset = Orchestrator("unset")

-	defaultOrchestrator      = OrchestratorSwarm
-	envVarDockerOrchestrator = "DOCKER_ORCHESTRATOR"


We may have to think about printing a warning if someone has this variable set.

I realize so far it's been experimental, so we're allowed to change things, but it's definitely possible people had this option set, and now it's being ignored.

Of course, to further complicate stuff; some day it may find its way back, once k8s integration on the cli is complete 😅

Just added the warning!

thaJeztah · 2018-06-21T20:42:55Z

cli/command/stack/cmd.go

-		newPsCommand(dockerCli),
-		newRemoveCommand(dockerCli),
-		newServicesCommand(dockerCli),
+		newDeployCommand(dockerCli, &opts),


It's a pity the change has such a deep impact. Don't see another solution at this time though 😕

thaJeztah · 2018-06-21T20:46:28Z

cli/command/system/version.go

@@ -29,7 +29,7 @@ Client:{{if ne .Platform.Name ""}} {{.Platform.Name}}{{end}}
 Built:	{{.BuildTime}}
 OS/Arch:	{{.Os}}/{{.Arch}}
 Experimental:	{{.Experimental}}
- Orchestrator:	{{.Orchestrator}}
+ Stack Orchestrator:	{{.StackOrchestrator}}


Still on the fence wether we should always print this, or only if any orchestrator is enabled

Looks like we may want to; here's against a host that doesn't have swarm (or kube) enabled;

Client: Version: 18.06.0-dev API version: 1.37 (downgraded from 1.38) Go version: go1.10.3 Git commit: ba061224 Built: Thu Jun 21 20:48:25 2018 OS/Arch: linux/amd64 Experimental: false Stack Orchestrator: swarm Server: Engine: Version: 18.05.0-ce API version: 1.37 (minimum version 1.12) Go version: go1.10.1 Git commit: f150324 Built: Wed May 9 22:20:16 2018 OS/Arch: linux/amd64 Experimental: true

Output for all may be a bit confusing as well, but don't have better suggestions for that at this point

Stack Orchestrator: all

thaJeztah · 2018-06-21T22:31:05Z

Giving this a spin;

Using "all" orchestrator:

$ docker stack deploy --orchestrator=swarm -c docker-compose.yml foobar
Creating network foobar_default
Creating service foobar_web

$ docker stack ls
NAME                SERVICES            ORCHESTRATOR       NAMESPACE
foobar              1                   Swarm

$ docker stack deploy -c docker-compose.yml foobar 
no orchestrator specified: use either "kubernetes" or "swarm"

$ docker stack deploy --orchestrator=kubernetes -c docker-compose.yml foobar
Waiting for the stack to be stable and running...
web: Ready		[pod status: 1/1 ready, 0/1 pending, 0/1 failed]

Stack foobar is stable and running

$ docker stack ls
NAME                SERVICES            ORCHESTRATOR        NAMESPACE
foobar              1                   Kubernetes          default
foobar              1                   Swarm               
something           1                   Swarm

This is something we can improve (i.e.; if it's non-ambiguous, show the result);

$ docker stack ps something
no orchestrator specified: use either "kubernetes" or "swarm"

$ docker stack ps --orchestrator=kubernetes foobar
ID                  NAME                          IMAGE               NODE                 DESIRED STATE       CURRENT STATE           ERROR               PORTS
1fc7d698-75a        foobar_web-789778cdbf-8tshh   nginx:alpine        docker-for-desktop   Running             Running 3 minutes ago                       *:0->80/tcp

$ docker stack ps --orchestrator=swarm foobar
ID                  NAME                IMAGE               NODE                    DESIRED STATE       CURRENT STATE           ERROR               PORTS
otpw084mgv9f        foobar_web.1        nginx:alpine        linuxkit-025000000001   Running             Running 6 minutes ago

(actually curious how we deal with overlaps in published ports)

$ docker stack services foobar --orchestrator=kubernetes

ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
1fd9a4d4-75a        foobar_web          replicated          1/1                 nginx:alpine        *:8080->80/tcp

$ docker stack services foobar --orchestrator=swarm

ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
9x0iam28mlsl        foobar_web          replicated          1/1                 nginx:alpine        *:8080->80/tcp

* Renaming DOCKER_ORCHESTRATOR to DOCKER_STACK_ORCHESTRATOR * Renaming config file option "orchestrator" to "stackOrchestrator" * "--orchestrator" flag is no more global but local to stack command and subcommands * Cleaning all global orchestrator code * Replicating Hidden flags in help and Supported flags from root command to stack command Signed-off-by: Silvin Lubecki <silvin.lubecki@docker.com>

thaJeztah · 2018-06-22T01:10:48Z

I squashed @silvin-lubecki's commits, and added an extra commit that (at least for now) removes the "Orchestration" information from docker info output until we have a better design for that

ping @dhiltgen @andrewhsu @vdemeester PTAL

thaJeztah

LGTM

andrewhsu

LGTM

dhiltgen

lgtm

dhiltgen · 2018-06-22T04:09:41Z

cli/command/system/version.go

@@ -78,7 +77,7 @@ type clientVersion struct {
 	Arch              string
 	BuildTime         string `json:",omitempty"`
 	Experimental      bool
-	Orchestrator      string `json:",omitempty"`
+	StackOrchestrator string `json:",omitempty"`


@thaJeztah it looks like this got left in (presumably to keep the changes to a minimum) - seems OK, but we should clean this up as part of a follow up pass.

Good one; was actually doubting whether or not we wanted to keep it; perhaps better to remove it yes, as it will show up in docker version --format '{{json .}}'

I'll grab my laptop and update

The output of this information can be confusing, so removing until we have a better design for this. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

The `-k` shorthand was alreaady removed from other commands, so best to be consistent. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah · 2018-06-22T06:41:19Z

@dhiltgen @silvin-lubecki @vdemeester I pushed some additional updates;

Removed the StackOrchestrator from the version struct (as discussed above Fix broken swarm commands with Kubernetes defined as orchestrator #1137 (comment))
Updated the documentation, and fixed some minor issues with the USAGE command
- note that the command-reference documentation on docs.docker.com is generated from a YAML file, so changes the markdown for individual commands is only to make GitHub match reality
Removed the -k shorthand on docker version; this shorthand was already removed (or not added) to any of the other commands. this is a new change

Remaining changes

documentation likely needs additional work (already being discussed on slack)
Bash/Fish/zsh completion scripts do not have all the options

thaJeztah · 2018-06-22T06:46:36Z

cli/command/stack/cmd.go

 	cmd := &cobra.Command{
-		Use:   "stack",
+		Use:   "stack [OPTIONS]",


this output isn't actually used for commands that have subcommands, but good to have in case that situation changes in future

vdemeester

SGTM 💦

vdemeester · 2018-06-22T07:44:38Z

cli/cobra.go

@@ -102,7 +102,7 @@ func managementSubCommands(cmd *cobra.Command) []*cobra.Command {
 var usageTemplate = `Usage:

 {{- if not .HasSubCommands}}	{{.UseLine}}{{end}}
-{{- if .HasSubCommands}}	{{ .CommandPath}} COMMAND{{end}}
+{{- if .HasSubCommands}}	{{ .CommandPath}}{{- if .HasAvailableFlags}} [OPTIONS]{{end}} COMMAND{{end}}


nit: Isn't this change unrelated ? 😝

The docker stack command itself now has options, but there was no [OPTIONS] printed 😞

vdemeester · 2018-06-22T07:54:56Z

cli/command/stack/cmd.go

-		newPsCommand(dockerCli),
-		newRemoveCommand(dockerCli),
-		newServicesCommand(dockerCli),
+		newDeployCommand(dockerCli, &opts),


silvin-lubecki · 2018-06-22T14:27:50Z

PTAL

vdemeester

SGTM 🐸

andrewhsu · 2018-06-22T18:43:54Z

@silvin-lubecki talked with @dhiltgen and @thaJeztah ... would prefer to have this PR without 8fb2e7d. the DOCKER_ORCHESTRATOR env var was in experimental mode so we don't have to treat that as a supported interface that needs migration.

if still strongly desired for inclusion, i'd suggest putting into a new PR.

thaJeztah · 2018-06-22T18:57:03Z

cli/command/stack/cmd.go

@@ -43,13 +42,7 @@ func NewStackCommand(dockerCli command.Cli) *cobra.Command {
 	}
 	defaultHelpFunc := cmd.HelpFunc()
 	cmd.SetHelpFunc(func(cmd *cobra.Command, args []string) {
-		config := cliconfig.LoadDefaultConfigFile(dockerCli.Err()) // dockerCli is not yet initialized, but we only need config file here


Modified the last commit, and removed the warning, but kept these changes

Discussing with @dhiltgen - dropping this commit for now, and keeping it for #1139

thaJeztah · 2018-06-22T19:00:33Z

Moved the warning to #1139

dhiltgen

lgtm

silvin-lubecki requested a review from vdemeester as a code owner June 21, 2018 16:58

GordonTheTurtle added the status/0-triage label Jun 21, 2018

silvin-lubecki requested review from thaJeztah, mat007 and dhiltgen June 21, 2018 16:58

silvin-lubecki force-pushed the stack-orchestrator branch from 0e358b6 to ba06122 Compare June 21, 2018 17:09

vdemeester added status/2-code-review and removed status/0-triage labels Jun 21, 2018

mat007 reviewed Jun 21, 2018

View reviewed changes

silvin-lubecki force-pushed the stack-orchestrator branch from 8c67022 to a935af8 Compare June 21, 2018 19:22

thaJeztah requested changes Jun 21, 2018

View reviewed changes

thaJeztah force-pushed the stack-orchestrator branch 3 times, most recently from 312408d to 87c39da Compare June 22, 2018 00:53

thaJeztah approved these changes Jun 22, 2018

View reviewed changes

andrewhsu approved these changes Jun 22, 2018

View reviewed changes

dhiltgen approved these changes Jun 22, 2018

View reviewed changes

thaJeztah added 3 commits June 21, 2018 22:02

Remove "Stack Orchestrator" from "docker version"

a63252b

The output of this information can be confusing, so removing until we have a better design for this. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Update command usage and documentation

0f07b9f

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Remove shorthand -k option from docker version

f0a8598

The `-k` shorthand was alreaady removed from other commands, so best to be consistent. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah force-pushed the stack-orchestrator branch from 87c39da to f0a8598 Compare June 22, 2018 06:35

thaJeztah reviewed Jun 22, 2018

View reviewed changes

thaJeztah added impact/changelog impact/documentation area/stack docs/revisit area/kubernetes labels Jun 22, 2018

vdemeester approved these changes Jun 22, 2018

View reviewed changes

thaJeztah force-pushed the stack-orchestrator branch from 8fb2e7d to c2c317c Compare June 22, 2018 18:56

thaJeztah reviewed Jun 22, 2018

View reviewed changes

thaJeztah mentioned this pull request Jun 22, 2018

Warn if DOCKER_ORCHESTRATOR is still used but not DOCKER_STACK_ORCHESTRATOR #1139

Merged

thaJeztah force-pushed the stack-orchestrator branch from c2c317c to f0a8598 Compare June 22, 2018 20:12

dhiltgen approved these changes Jun 22, 2018

View reviewed changes

andrewhsu merged commit 151990d into docker:master Jun 22, 2018

GordonTheTurtle added this to the 18.06.0 milestone Jun 22, 2018

albers mentioned this pull request Aug 1, 2018

Add bash completion for experimental CLI commands (manifest) #1252

Merged

thaJeztah mentioned this pull request Nov 29, 2018

[18.09 backport] Add bash completion for experimental CLI commands (manifest) #1542

Merged

silvin-lubecki deleted the stack-orchestrator branch January 23, 2019 10:17

		@@ -17,10 +17,25 @@ const (
		OrchestratorAll = Orchestrator("all")

Fix broken swarm commands with Kubernetes defined as orchestrator #1137

Fix broken swarm commands with Kubernetes defined as orchestrator #1137

Conversation

silvin-lubecki commented Jun 21, 2018

silvin-lubecki commented Jun 21, 2018

mat007 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

silvin-lubecki commented Jun 21, 2018

silvin-lubecki commented Jun 21, 2018

antonybichon17 commented Jun 21, 2018

silvin-lubecki commented Jun 21, 2018

thaJeztah left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thaJeztah Jun 21, 2018 • edited Loading

Choose a reason for hiding this comment

thaJeztah commented Jun 21, 2018

thaJeztah commented Jun 22, 2018

thaJeztah left a comment

Choose a reason for hiding this comment

andrewhsu left a comment

Choose a reason for hiding this comment

dhiltgen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thaJeztah commented Jun 22, 2018

Remaining changes

Choose a reason for hiding this comment

vdemeester left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

silvin-lubecki commented Jun 22, 2018

vdemeester left a comment

Choose a reason for hiding this comment

andrewhsu commented Jun 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thaJeztah commented Jun 22, 2018

dhiltgen left a comment

Choose a reason for hiding this comment

thaJeztah Jun 21, 2018 •

edited

Loading