Add support for Docker 18.09.3. #6347

tsuna · 2019-01-16T01:30:36Z

Starting from Docker 18.09.0, the Docker distribution has been split in
3 packages: the Docker daemon, the Docker CLI, and for containerd. This
adds a twist to how to upgrade Docker from the base image as the daemon
and CLI packages must be installed at the same time, otherwise dpkg/rpm
will refuse to upgrade (the new CLI is incompatible with the old package
and the daemon can't be installed without first installing the CLI and
the new containerd, so the upgrade MUST happen in a single transaction).

This code change thus adds the possibility to specify sources for the
CLI and containerd in nodeup, and the ability to apply the multi-package
upgrade atomically with dpkg/rpm.

Note: I tested this on one of my kops clusters in AWS.

This fixes #5747

k8s-ci-robot · 2019-01-16T01:30:43Z

Hi @tsuna. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tsuna · 2019-01-17T21:26:16Z

/assign @justinsb @andrewsykim @chrisz100

mikesplain · 2019-01-18T14:00:19Z

/ok-to-test

tsuna · 2019-01-22T03:03:52Z

I reworked my change a bit by instead adding a more generic provision for specifying extra packages to install at the same time as Docker. I noticed that RHEL/CentOS has needed this for a while to deploy SELinux policies anyway, so I figured it made more sense to unify this existing need with the new one (the small difference is that the SELinux policies could be installed on their own, which is why the existing kludge in the code was working fine, whereas now to upgrade Docker with the newer 3-package distribution everything needs to happen in a single dpkg transaction).

Let me know what you think.

tsuna · 2019-01-25T01:02:43Z

I just rebased my change, can someone take a look please?

tsuna · 2019-02-04T22:53:02Z

Ping, pretty please :)

chrisz100 · 2019-02-04T22:58:27Z

upup/pkg/fi/nodeup/nodetasks/package.go

@@ -162,7 +169,7 @@ func (e *Package) findDpkg(c *fi.Context) (*Package, error) {
 			installed = true
 			installedVersion = version
 			healthy = fi.Bool(true)
-		case "iF":
+		case "iF", "iU":


What’s this iU?

This happened in an earlier version of this change that attempted to install the packages one by one (which, ultimately, doesn't work). iF means the package is selected to be installed and is currently in half-configured state, while iU means it's selected to be installed and is currently merely unpacked.

It might not be needed anymore since this iteration of the code installs all 3 packages in a single transaction, but having the code craps out if it sees the package in iU state instead of continuing to try/wait until the install is complete seems like a bad idea. So I figured I'd leave this here as it makes sense to handle iU in the same way that iF is handled.

tsuna · 2019-02-04T23:52:05Z

Just rebased my change one more time, to drop a commit that had become redundant since commit 24222ff merged 8 days ago.

tsuna · 2019-02-05T00:23:26Z

/retest
Looks like a spurious infrastructure failure:

error reading s3://k8s-kops-prow/e2e-8592-ff1eb.test-cncf-aws.k8s.io/config: Unable to list AWS regions: AuthFailure: AWS was not able to validate the provided access credentials

tsuna · 2019-02-07T07:17:10Z

/retest (it was the same infrastructure failure I mentioned above)
🤞

coreypobrien · 2019-02-11T15:36:56Z

Can we bump this to 18.09.2 to address CVE-2019-5736 at the same time?

tsuna · 2019-02-28T21:14:54Z

I rebased again and adjusted the change for Docker 18.09.3 that came out today.

I didn't understand the e2e failure that hit previously. It says should support inline execution and attach [It], dunno what that means.

Starting from Docker 18.09.0, the Docker distribution has been split in 3 packages: the Docker daemon, the Docker CLI, and for containerd. This adds a twist to how to upgrade Docker from the base image as the daemon and CLI packages must be installed at the same time, otherwise dpkg/rpm will refuse to upgrade (the new CLI is incompatible with the old package and the daemon can't be installed without first installing the CLI and the new containerd, so the upgrade MUST happen in a single transaction). This code change thus adds the possibility to specify additional packages to install in the same dpkg/yum transaction, such as the Docker CLI and containerd in nodeup, and the ability to apply the multi-package upgrade atomically with dpkg/rpm. We also use this new mechanism for the SELinux policy on RHEL/CentOS.

tsuna · 2019-03-05T19:10:46Z

Just rebased again. Please, pretty please, help me push this through the finish line this week. 🙏

tsuna · 2019-03-08T21:44:35Z

Pretty please, can I haz LGTM & approvalz? 😢

chrisz100 · 2019-03-08T21:46:50Z

/lgtm

tsuna · 2019-03-08T21:50:27Z

Thanks @chrisz100! Almost in time for me to check this off my list for our weekly standup in 10min. How can I get the approval now? If you know who I need to bribe, let me know 😄

montanaslim · 2019-03-11T17:01:46Z

any idea when this will be approved?

tsuna · 2019-03-12T18:08:24Z

I don't know, I don't even understand the difference between LGTM and approval, but if anyone knows who I need to bribe, let me know.

tsuna · 2019-03-13T21:27:24Z

@mikesplain @justinsb can one of you guys please approve this PR? I'll pay you 100 KubeCoin on the day of the ICO. Just need to finish my business plan for a blockchain-backed k8s control plane. etcd is for losers. TIA.

montanaslim · 2019-03-13T22:12:05Z

@tsuna lol'd irl

chrisz100 · 2019-03-14T07:04:24Z

@tsuna it’s the two step review process - lgtm is technically does the code look good and technically doable by anyone in the Kubernetes GitHub org- approver is a higher level that is supposes to check for does it add to our roadmap when we approve it right now or is later a better time.

As it seems like it’s targeted for 1.12 now so rather soon!

justinsb · 2019-03-15T00:52:41Z

Sorry this took so long @tsuna - it hit in the middle of the test account problems followed by the docker CVE, and we had to prioritize the CVE.

I think the package version changed in one place, but I'm just going to send another PR rather than risk forcing another rebase on you. Plus I want those kubecoins :-)

/approve
/lgtm

k8s-ci-robot · 2019-03-15T00:52:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justinsb, tsuna

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [justinsb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Follow up to kubernetes#6347 - add a test for some of the names based on some heuristics, and fix some of the problems that popped up.

shahbhavik01 · 2019-03-26T02:43:36Z

@tsuna
I did:
kops edit cluster

Under cluster spec:

spec:
    docker:
        version: 18.09.3

kops version: 1.11.1
kubernetes version 1.10.12

and then I ran:

kops update cluster <CLUSTER-NAME> --yes
kops rolling-update cluster --yes

Even then the docker-version remains 17.3.2. Any help would be appreciated!

kubectl describe nodes | grep docker
 Container Runtime Version:  docker://17.3.2
 Container Runtime Version:  docker://17.3.2

rdrgmnzs · 2019-03-26T06:46:55Z

@shahbhavik01 this new version will not be in kops until 1.12 is released.

tsuna · 2019-03-26T18:10:44Z

You can look at #6448 (edit: more specifically, this comment) if you wanna try a version you build yourself from master, I posted some instructions there :)

marek-obuchowicz · 2019-04-18T16:40:23Z

@tsuna / @shahbhavik01 I tried kops 1.12.0-beta.2 but for some reason it uses still docker://18.6.3 with kubernetes v1.12.7. Is this expected (looking on the fact that this pr was merged already)?

update: all good, 18.06.3 was chosen as default, but it's possible to select 18.09.3 (as value for spec.docker.version). Is it also possible when using kops 1.12 with older k8s (like 1.11)?

tsuna · 2019-04-18T17:29:49Z

Cześć Marek, the default version used by kops remains unchanged. I'm not sure why, TBH, but you need to edit your cluster manifest to pick the Docker version you want.

apiVersion: kops/v1alpha2
kind: Cluster
[...]
spec:
[...]
  docker:
    version: 18.09.3

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 16, 2019

k8s-ci-robot requested review from rdrgmnzs and robinpercy January 16, 2019 01:30

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 16, 2019

tsuna force-pushed the master branch from 8b53a12 to 28d2723 Compare January 16, 2019 07:25

k8s-ci-robot assigned andrewsykim, chrisz100 and justinsb Jan 17, 2019

tsuna force-pushed the master branch from 28d2723 to 3f9f7d5 Compare January 19, 2019 00:22

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 19, 2019

tsuna force-pushed the master branch from 3f9f7d5 to 2b48e90 Compare January 22, 2019 03:01

tsuna force-pushed the master branch from 2b48e90 to 912a0b3 Compare January 25, 2019 01:02

chrisz100 reviewed Feb 4, 2019

View reviewed changes

tsuna mentioned this pull request Feb 4, 2019

Weird syntax to delete the dockerconfig secret #6447

Closed

tsuna force-pushed the master branch from 912a0b3 to 7e51fa8 Compare February 4, 2019 23:50

tsuna force-pushed the master branch from 7e51fa8 to 0cc4229 Compare February 11, 2019 21:50

tsuna changed the title ~~Add support for Docker 18.09.2.~~ Add support for Docker 18.09.3. Feb 28, 2019

tsuna mentioned this pull request Feb 28, 2019

"error finding the launch configuration or template for instance" #6558

Closed

tsuna force-pushed the master branch from 0a3c172 to e4691cd Compare March 5, 2019 19:09

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 8, 2019

justinsb added this to the 1.12 milestone Mar 14, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 15, 2019

justinsb added a commit to justinsb/kops that referenced this pull request Mar 15, 2019

Fix some of the docker package names & versions

30196e5

Follow up to kubernetes#6347 - add a test for some of the names based on some heuristics, and fix some of the problems that popped up.

justinsb mentioned this pull request Mar 15, 2019

Fix some of the docker package names & versions #6620

Merged

k8s-ci-robot merged commit 16e846d into kubernetes:master Mar 15, 2019

justinsb added a commit to justinsb/kops that referenced this pull request Mar 15, 2019

Fix some of the docker package names & versions

057c19f

Follow up to kubernetes#6347 - add a test for some of the names based on some heuristics, and fix some of the problems that popped up.

alexfrancavilla mentioned this pull request Jul 16, 2019

kops v1.12.1 still shipped with vulnerable docker-ce? #7216

Closed

CarpathianUA mentioned this pull request Aug 23, 2019

Docker 18.09.3 for Centos #7463

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Docker 18.09.3. #6347

Add support for Docker 18.09.3. #6347

tsuna commented Jan 16, 2019

k8s-ci-robot commented Jan 16, 2019

tsuna commented Jan 17, 2019

mikesplain commented Jan 18, 2019

tsuna commented Jan 22, 2019

tsuna commented Jan 25, 2019

tsuna commented Feb 4, 2019

chrisz100 Feb 4, 2019

tsuna Feb 4, 2019

tsuna commented Feb 4, 2019

tsuna commented Feb 5, 2019

tsuna commented Feb 7, 2019

coreypobrien commented Feb 11, 2019

tsuna commented Feb 28, 2019

tsuna commented Mar 5, 2019

tsuna commented Mar 8, 2019

chrisz100 commented Mar 8, 2019

tsuna commented Mar 8, 2019

montanaslim commented Mar 11, 2019

tsuna commented Mar 12, 2019

tsuna commented Mar 13, 2019

montanaslim commented Mar 13, 2019

chrisz100 commented Mar 14, 2019

justinsb commented Mar 15, 2019

k8s-ci-robot commented Mar 15, 2019

shahbhavik01 commented Mar 26, 2019 •

edited

Loading

rdrgmnzs commented Mar 26, 2019

tsuna commented Mar 26, 2019 •

edited

Loading

marek-obuchowicz commented Apr 18, 2019 •

edited

Loading

tsuna commented Apr 18, 2019

Add support for Docker 18.09.3. #6347

Add support for Docker 18.09.3. #6347

Conversation

tsuna commented Jan 16, 2019

k8s-ci-robot commented Jan 16, 2019

tsuna commented Jan 17, 2019

mikesplain commented Jan 18, 2019

tsuna commented Jan 22, 2019

tsuna commented Jan 25, 2019

tsuna commented Feb 4, 2019

chrisz100 Feb 4, 2019

Choose a reason for hiding this comment

tsuna Feb 4, 2019

Choose a reason for hiding this comment

tsuna commented Feb 4, 2019

tsuna commented Feb 5, 2019

tsuna commented Feb 7, 2019

coreypobrien commented Feb 11, 2019

tsuna commented Feb 28, 2019

tsuna commented Mar 5, 2019

tsuna commented Mar 8, 2019

chrisz100 commented Mar 8, 2019

tsuna commented Mar 8, 2019

montanaslim commented Mar 11, 2019

tsuna commented Mar 12, 2019

tsuna commented Mar 13, 2019

montanaslim commented Mar 13, 2019

chrisz100 commented Mar 14, 2019

justinsb commented Mar 15, 2019

k8s-ci-robot commented Mar 15, 2019

shahbhavik01 commented Mar 26, 2019 • edited Loading

rdrgmnzs commented Mar 26, 2019

tsuna commented Mar 26, 2019 • edited Loading

marek-obuchowicz commented Apr 18, 2019 • edited Loading

tsuna commented Apr 18, 2019

shahbhavik01 commented Mar 26, 2019 •

edited

Loading

tsuna commented Mar 26, 2019 •

edited

Loading

marek-obuchowicz commented Apr 18, 2019 •

edited

Loading