Upgrade to Kubernetes 1.14

[roberthbailey]

1.15 is still not available on EKS (see aws/containers-roadmap#380) but 1.13 has now been deprecated on AKS (https://github.com/Azure/AKS/releases/tag/2020-02-03) and customers on AKS are being told to upgrade off of 1.13.

So we are in a bit of a conundrum. While our guidance has been to wait for the next release to be available on all major cloud providers, we haven't yet encountered a time when that wasn't yet the case before the version we supported has been deprecated and will no longer be supported on one of the providers.

In this case, it seems like we should start thinking about upgrading to 1.14, since it's been available on EKS for a long time (and is presumably somewhat stable) and there is no concrete ETA for 1.15 being available there.

List of items to do for upgrading to 1.14 (this is copied from the 1.13 issue and should be updated):

• Update e2e cluster to run against 1.14
  • Update deployment manage script
  • Recreate cluster with new scripts
• Update prow cluster to use 1.14 (even though we aren't using it yet, we should keep it in sync)
  • Update deployment script
  • Recreate cluster with new scripts
• Update the dev tooling to create 1.14 clusters
  • GKE
  • Minikube
  • Kind
  • Update kubectl
• Update terraform submodules
  • GKE
  • Azure
• Update documentation for creating clusters to 1.14
  • Usage requirements
  • GKE
  • Minikube
  • EKS
  • AKS
  • Helm documentation
• Update links to k8s documentation
  • examples/fleet.yaml
  • examples/fleetautoscaler.yaml
  • examples/gameserver.yaml
  • site/content/en/docs/Reference/fleet.md
  • site/content/en/docs/Reference/fleetautoscaler.md
  • site/content/en/docs/Reference/gameserver.md
• Update to client-go 11.0 (based on compatibility matrix)

[markmandel]

I 100% agree. I think we're reaching a point where we can reach consensus to break the standard rule, since AWS has taken it's time.

Also 1.14 has been baked for a long time at this point, so I don't think there are any stability concerns.

I'm 100% on board. Shall we say that if we don't have objections to move to 1.14 by Thursday (13th of Feb), we make the move?

[roberthbailey]

In the past we've tried to move k8s versions early in the release cycle to have more time for the tests to run. If we wait too long to make the switch, then we start to get too close to the 1.4.0 release candidate / 1.4.0 release. I think we should either do it ASAP or wait until after 1.4.0.

[markmandel]

Excellent point. Give it 24 hours then?

I threw a message in Slack as well asking if anyone had objections.

I've got none, I'm definitely onboard.

[markmandel]

Nobody seems to be objecting. I say let's move forward with it (if we can fit it into this release). I know @heartrobotninja will be happy 😄

[aLekSer]

Also we would finally remove replace k8s.io/apimachinery => ./vendor_fixes/k8s.io/apimachinery with 1.14 version.

[heartrobotninja]

That is why this will make me happy 😁

…

On Tue, Feb 11, 2020, 11:54 AM Alexander Apalikov ***@***.***> wrote: Also we would finally remove replace k8s.io/apimachinery => ./vendor_fixes/k8s.io/apimachinery with 1.14 version. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1329?email_source=notifications&email_token=ABM5DZ3UITJP2OZQHJE3ABTRCL7ANA5CNFSM4KSIYZQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELN2TFY#issuecomment-584821143>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABM5DZ7BZL4V5HF2KQCNXV3RCL7ANANCNFSM4KSIYZQA> .

[steven-supersolid]

Great news :)

AKS supports N-2 and aims to release N within around 30 days of the upstream release
https://docs.microsoft.com/en-us/azure/aks/supported-kubernetes-versions
As 1.17 is now available and in preview on AKS we might find ourselves in an unfortunate situation where 1.14 is deprecated on AKS but 1.15 not available on EKS.

If this happens, should we consider a new process where we ensure that Agones works on version N-1, but is also compatible with N without using any of the new features? This would rely on N not introducing any breaking changes of course.

[markmandel]

If this happens, should we consider a new process where we ensure that Agones works on version N-1, but is also compatible with N without using any of the new features? This would rely on N not introducing any breaking changes of course.

This sounds like a separate ticket, as it's a discussion of Kubernetes support matrix and has far ranging consequences.

[roberthbailey]

Current status (I've also filled in the checkboxes at the top of the ticket):

• e2e and prow clusters have been recreated at 1.14.
• Verified the dev tooling changes create 1.14 clusters for: GKE, Minikube, Kind

My PR for terraform went in but since I haven't manually verified the changes I haven't checked it off the list yet.

@aLekSer - if you have a chance to verify the Azure terraform changes I'd appreciate it.

I'd update the documentation closer to the next release (there isn't as much rush to do that part), and @heartrobotninja volunteered to update client-go (and I think this is the update where we get to remove our vendor_fixes directory!).

[aLekSer]

Yes, I will check Azure terraform configs tomorrow morning and will tell the results.

[aLekSer]

There was a number of issues with terraform, one of which is that Kubernetes Terraform provider 1.11 is not working due to hashicorp/terraform-provider-kubernetes#759
So I need to limit upper bound of versions.

Error: Error creating Managed Kubernetes Cluster "test-cluster" (Resource Group "agonesRG"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="AgentPoolK8sVersionNotSupported" Message="Version 1.14.10 is not supported in this region. Please use [az aks get-versions] command to get the supported version list in this region. For more information, please check https://aka.ms/supported-version-list"

  on .terraform/modules/aks_cluster/install/terraform/modules/aks/aks.tf line 47, in resource "azurerm_kubernetes_cluster" "test":                                                                         
  47: resource "azurerm_kubernetes_cluster" "test" {

Will fix them soon and make a PR

[roberthbailey]

@aLekSer's PR to fix the issues with terraform is #1353.

[guitarrapc]

EKS now supports Kubernetes 1.15.

https://aws.amazon.com/about-aws/whats-new/2020/03/amazon-eks-now-supports-kubernetes-
version-1-15/

[roberthbailey]

Now that #1404 has merged this task is finished.