Docker client version 19.03.09 doesn't automatically downgrade Docker API version correctly

[lokst]

Description

The 19.03.09 version of the docker client doesn't automatically downgrade the API version it uses to be compatible with an older daemon. This causes docker push and other Docker commands like docker version to fail if the maximum API version supported by the Docker daemon is lower than the API version used by the docker client.

Steps to reproduce the issue:

1. Install docker client 19.03.09
2. Run Docker daemon 18.09.3
3. Run "docker push"

Describe the results you received:
docker push fails with the error message Error response from daemon: client version 1.40 is too new. Maximum supported API version is 1.39

When running docker version with version 19.03.09 of the CLI, it fails with:

Client: Docker Engine - Community
 Version:           19.03.9
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        9d988398e7
 Built:             Fri May 15 00:22:47 2020
 OS/Arch:           linux/amd64
 Experimental:      false
Error response from daemon: client version 1.40 is too new. Maximum supported API version is 1.39

But with version 19.03.08 of the client and the same version of the Docker daemon (18.09.3), docker version produces:

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.39 (downgraded from 1.40)
 Go version:        go1.12.17
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 01:22:56 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.3
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       774a1f4
  Built:            Thu Feb 28 05:59:55 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Describe the results you expected:
I expected Docker client 19.03.09 to downgrade its API version to 1.39 , similar to what Docker client 19.03.08 does.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

See above

Output of docker info:

Client:
 Debug Mode: false

Server:
ERROR: Error response from daemon: client version 1.40 is too new. Maximum supported API version is 1.39
errors pretty printing info

Additional environment details (AWS, VirtualBox, physical, etc.):
This happens when the docker CLI is run in a Docker container that communicates with a remote Docker daemon

[silvin-lubecki]

Hello @lokst and thank you for filling this issue. I managed to reproduce the bug you found, let me check why we have this regression.

[lokst]

@silvin-lubecki Thank you for looking into this! 🙂

[Gaff]

Am able to reproduce with centos:

docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock centos 
$ yum install -y yum-utils
$ yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
$ yum install -y docker-ce-cli
$ docker version
Client: Docker Engine - Community
  Version:           19.03.9
  API version:       1.40
  Go version:        go1.13.10
  Git commit:        9d988398e7
  Built:             Fri May 15 00:25:27 2020
  OS/Arch:           linux/amd64
  Experimental:      false
 Error response from daemon: client version 1.40 is too new. Maximum supported API version is 1.39

Workaround is to pin to a specific package in yum:

yum install -y docker-ce-cli-1:19.03.8-3.el7

[sarun87]

@silvin-lubecki thanks for looking into this swiftly.
I confirm that we are seeing this with newer installation of docker across multiple linux flavors IUbuntu 18, CentOS 7.7).

Interestingly some commands continue to work (docker inspect , docker run )

[srolija]

Hoping this finds anyone searching for the same issue. :)

Yesterday CircleCI updated some of their common images (in our case circleci/python:3.7-stretch-node) to the affected 19.03.9 version.

Which started to cause the builds with the explicit version of remote docker:

      - setup_remote_docker:
          version: 18.06.0-ce

to fail with the aforementioned error:

Error response from daemon: client version 1.39 is too new. Maximum supported API version is 1.37

As far as I can find, the issue does not seem to occur on the default remote 17.09.0-ce, I am assuming that it didn't validate the API version at the same time as the newer remotes do or that this bug only affects newer versions for some other reason.

While this isn't fixed I would recommend people experiencing the same issue to either downgrade to previous versions fo the build images (in our case 3.7.6-stretch-node), to define the explicit API version that is compatible with both local CLI and remote docker in our example the DOCKER_API_VERSION=1.37 or to remove the explicit remote version in YAML (as default seems to work).

[ocofaigh]

@silvin-lubecki Any idea when your fix will be available in a release? Will it part of v19.03.10 and any ETA for that?

[silvin-lubecki]

@ocofaigh this fix is planned for v19.03.10 release yes. I don't have precise ETA for the release but it's in our ASAP list 👍

[phil-davis]

Suddenly today we are getting this problem on cloud.drone.io
cs3org/reva#1083 (comment)

It is a free-to-open-source CI environment. I don't think that we are going to have control of the various versions of stuff in that underlying infrastructure.