Document required manual calico 2.6.3 -> calico 3.1.1 upgrade when upgrading from < 0.17.0-provisioned clusters

[oivindoh]

What this PR does / why we need it:
Upgrading from clusters deployed by acs-engine < 0.17.0 and calico enabled had calico 2.6.3. When upgrading such a cluster with 0.17.0 and later, calico addon manifest is 3.1.x, and a migration is not supported for releases prior to 2.6.5, so we need to perform some manual steps to get up and running on 3.1.x. See Issue #3191

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
fixes #3191

Special notes for your reviewer:
I'm not entirely certain where this belongs (whether in examples/networkpolicy or examples/k8s-upgrade.

If applicable:

• documentation
• unit tests
• tested backward compatibility (ie. deploy with previous version, upgrade with this branch)

Release note:

[acs-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: oivindoh
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: cecilerobertmichon

Assign the PR to them by writing /assign @cecilerobertmichon in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[msftclas]

All CLA requirements met.

[codecov]

Codecov Report

Merging #3208 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3208   +/-   ##
=======================================
  Coverage   52.31%   52.31%           
=======================================
  Files         103      103           
  Lines       15458    15458           
=======================================
  Hits         8087     8087           
  Misses       6643     6643           
  Partials      728      728

[CecileRobertMichon]

Thank you @oivindoh! Looks great. Could you please add something like "As per the Calico v0.3.1 release notes, Calico v0.3.x includes breaking changes:

Some highlights include:
You must upgrade to Calico v2.6.5 before you can upgrade to v3.0.1 (see https://docs.projectcalico.org/v3.0/getting-started/kubernetes/upgrade/)
Calico deployments that access the etcd datastore directly must complete
a one-time migration.
You must convert any customized Calico manifests via calicoctl convert
before you can use them with v3.0.1.

Here some instructions to get up and running with the new version of Calico […]" here?

[CecileRobertMichon]

So people can have context on the breaking changes (and know it's a calico breaking change, not acs-engine)

[oivindoh]

Good point! I added some text to be more explicit about this being due to calico breaking changes and not acs-engine 👍

[oivindoh]

@dtzar could you sanity-check this?

[dtzar]

@oivindoh Thanks for the docs. Although I'm sure the steps you list work, this does not follow the upgrade guidance provided by Calico. I know there are significant changes to the RBAC config as well as the manifest - see the changes in my PR which did the upgrade to get an idea.

Soo... I wouldn't approve this guidance as-is. It should map directly to what is on Calico's webpage. You could truncate what you have and just punt to the Calico upgrade webpage listing the fact we use the Kuberntes Datastore, policy only configuration - or be more specific following the flow/guidance they have which will be specific to acs-engine deployments.

[oivindoh]

@dtzar I guess I'm not entirely clear on where I diverge from the linked upgrade guidance - applying the 3.x manifest with node/cni changed to 2.6.10 and 2.0.6 effectively performs step 1, and handles upgrade to 2.6.5+ in the process. Applying that manifest again with node/cni 3.1.1 performs the rest of the steps required, keeping cluster-cidr as appropriate.

What I wasn't clear on after reading Calico docs (and what I wanted to document here) was the how of upgrading to 2.6.5+, given that I now had a cluster with a 3.1.1 manifest to be managed as an addon and calico 2.6.3 actually running in the cluster.

[dtzar]

Let me know if this is more clear now or if you still have questions

[dtzar]

The place where you diverge from the upgrade guidance is here because you are only replacing the image versions, not the entire template/manifest. The calico-daemonset.yaml is a merge of the Calico RBAC + manifest files and needs more changes than just bumping the image versions.

[dtzar]

Same thing here.

[dtzar]

@oivindoh I read through the document again end-to-end and I think I realize what you're suggesting now. The missing explanation is that I think you're saying to push through with the actual acs-engine upgrade (which will break Calico) and then go into each master node and do the following steps, correct?

If so, then yes the only thing I'm unsure of is if there are any potential problems with the temporary upgrade step you have of 2.6.10 using a 3.x RBAC/Policy template. Feels like a safer route would be to upgrade to 2.6.10 on the existing cluster (so the manifest is much closer), then upgrade acs-engine, then the new 3.x template might just work as-is then.

[oivindoh]

@dtzar Exactly - however upgrading the cluster shouldn’t break calico because it will happily continue to run using its old 2.6.3 definitions until the moment you apply the add on manifest manually, since it’s not set to anything but ensureexists, effectively keeping addon manager off the table.

I should update the document and remove the propagate to each master comment, since the masters will already have the updated manifest anyway (we have opted for 3.1.3 instead of the supplied 3.1.1 for richer network policy support, so needed to propagate).

I didn’t do any form of extended validation after applying 2.6.10 other the quick manual tests to see or services were still available and no alerts were triggering before updating to 3.x, so I can’t really vouch for zero negative effects during that step, but I could not detect anything FWIW.

3.1.1 and 3.1.3 have definitely been chugging along happily after the upgrade process.

[dtzar]

@oivindoh - I wouldn't recommend using the new yaml manifest with 2.6.10 images since it does have significant manifest/rbac changes. Would you be willing to test out this flow and update the document?

1. Update the daemonset yaml (pre-acs-engine-upgrade) to 2.6.10 with the steps as you described.
2. Upgrade / deploy to newer 3.x calico using acs-engine (it should just work at this point...)

Also - I just issued #3257 which upgrades to the latest 3.1.3 so you can update your document accordingly and not have to do another step.

[jackfrancis]

I think given that acs-engine itself does not provide cluster lifecycle configuration management (such that this would be taken care of more elegantly), and that the versions of Calico will continue to grow, let's just accept this doc as-is. It might not be 100% precise, but it has a good chance of helping someone out in the future who is running an acs-engine-built Calico 2 cluster and who is unable to tear down and recreate a new cluster.