Merge remote-tracking branch 'upstream/master'

* upstream/master: (21 commits)
  Improve the installation script (kubermatic#1253)
  Update README.md (kubermatic#1250)
  Add the changelog for the v1.2.0-beta.1 release (kubermatic#1249)
  Fix credentials in addons (kubermatic#1248)
  fix(docs): fix broken master documentation link (kubermatic#1246)
  Add encryption providers proposal (kubermatic#1213)
  Use Docker for restarting API server on Flatcar (kubermatic#1245)
  Add the Kubernetes 1.20 jobs (kubermatic#1244)
  Restart unhealthy API servers when provisioning/upgrading clusters (kubermatic#1243)
  Add rsync on CentOS and Amazon Linux (kubermatic#1240)
  Update machine-controller to v1.25.0 (kubermatic#1238)
  Update the kubeone-e2e image (kubermatic#1239)
  Update KubeOne CI jobs (kubermatic#1237)
  Disallow and deprecate the PodPresets feature (kubermatic#1236)
  Fix confusing default in OpenIDConnect (kubermatic#1235)
  Add debian support (kubermatic#1233)
  Drop mounting flexvolume plugins into the openstack CCM (kubermatic#1234)
  Remove CoreOS (kubermatic#1232)
  Add the changelog for the v1.2.0-beta.0 release (kubermatic#1230)
  Add containerRuntime API to the full config (kubermatic#1229)
  ...

CHANGELOG.md

@@ -1,5 +1,124 @@
 # Changelog
 
+# [v1.2.0-beta.1](https://github.com/kubermatic/kubeone/releases/tag/v1.2.0-beta.1) - 2021-02-17
+
+## Attention Needed
+
+* [**Breaking**] Support for CoreOS has been removed from KubeOne and machine-controller
+  * CoreOS has reached End-of-Life on May 26, 2020
+  * As an alternative to CoreOS, KubeOne supports Flatcar Linux
+  * We recommend migrating your CoreOS clusters to the Flatcar Linux or other supported operating system
+* [**Breaking**] Default values for OpenIDConnect has been corrected to match what's advised by the example configuration
+  * Previously, there were no default values for the OpenIDConnect fields
+  * This might only affect users using the OpenIDConnect feature
+* [**Breaking**] Disallow and deprecate the PodPresets feature
+  * [**Action Required**] If you're upgrading a cluster that uses the PodPresets feature from Kubernetes 1.19 to 1.20, you have to disable the PodPresets feature in the KubeOne configuration manifest
+  * The PodPresets feature has been removed from Kubernetes 1.20 with no built-in replacement
+  * It's not possible to use the PodPresets feature starting with Kubernetes 1.20, however, it currently remains possible to use it for older Kubernetes versions
+  * The PodPresets feature will be removed from the KubeOneCluster API once Kubernetes 1.19 reaches End-of-Life (EOL)
+  * As an alternative to the PodPresets feature, Kubernetes recommends using the MutatingAdmissionWebhooks.
+
+## Added
+
+* Add support for Kubernetes 1.20
+  * Previously, we've shared that there is an issue affecting newly created clusters where the first control plane node is unhealthy/broken for the first 5-10 minutes. We've investigated the issue and found out that the issue can be successfully mitigated by restarting the first API server. We've implemented a task that automatically restarts the API server if it's affected by the issue ([#1243](https://github.com/kubermatic/kubeone/pull/1243), [#1245](https://github.com/kubermatic/kubeone/pull/1245))
+* Add support for Debian on control plane and static worker nodes ([#1233](https://github.com/kubermatic/kubeone/pull/1233))
+  * Debian is currently not supported by machine-controller, so it's not possible to use it on worker nodes managed by machine-controller
+
+## Changed
+
+### API Changes
+
+* [**Breaking**] Default values for OpenIDConnect has been corrected to match what's advised by the example configuration ([#1235](https://github.com/kubermatic/kubeone/pull/1235))
+  * Previously, there were no default values for the OpenIDConnect fields
+  * This might only affect users using the OpenIDConnect feature
+* [**Breaking**] Disallow and deprecate the PodPresets feature ([#1236](https://github.com/kubermatic/kubeone/pull/1236))
+  * [**Action Required**] If you're upgrading a cluster that uses the PodPresets feature from Kubernetes 1.19 to 1.20, you have to disable the PodPresets feature in the KubeOne configuration manifest
+  * The PodPresets feature has been removed from Kubernetes 1.20 with no built-in replacement
+  * It's not possible to use the PodPresets feature starting with Kubernetes 1.20, however, it currently remains possible to use it for older Kubernetes versions
+  * The PodPresets feature will be removed from the KubeOneCluster API once Kubernetes 1.19 reaches End-of-Life (EOL)
+  * As an alternative to the PodPresets feature, Kubernetes recommends using the MutatingAdmissionWebhooks.
+
+### General
+
+* Add rsync on CentOS and Amazon Linux ([#1240](https://github.com/kubermatic/kubeone/pull/1240))
+
+### Bug Fixes
+
+* Drop mounting Flexvolume plugins into the OpenStack CCM. This fixes the issue with deploying the OpenStack CCM on the clusters running Flatcar Linux ([#1234](https://github.com/kubermatic/kubeone/pull/1234))
+* Ensure all credentials are available to be used in addons. This fixes the issue with the Backups addon not working on non-AWS providers ([#1248](https://github.com/kubermatic/kubeone/pull/1248))
+
+### Updated
+
+* Update machine-controller to v1.25.0 ([#1238](https://github.com/kubermatic/kubeone/pull/1238))
+
+## Removed
+
+* [**Breaking**] Support for CoreOS has been removed from KubeOne and machine-controller ([#1232](https://github.com/kubermatic/kubeone/pull/1232))
+  * CoreOS has reached End-of-Life on May 26, 2020
+  * As an alternative to CoreOS, KubeOne supports Flatcar Linux
+  * We recommend migrating your CoreOS clusters to the Flatcar Linux or other supported operating system
+
+# [v1.2.0-beta.0](https://github.com/kubermatic/kubeone/releases/tag/v1.2.0-beta.0) - 2021-01-27
+
+## Attention Needed
+
+* Kubernetes has announced deprecation of the Docker (dockershim) support in
+  the Kubernetes 1.20 release. It's expected that Docker support will be
+  removed in Kubernetes 1.22
+  * All newly created clusters running Kubernetes 1.21+ will be provisioned
+    with containerd instead of Docker
+  * Automated migration from Docker to containerd is currently not available,
+    but is planned for one of the upcoming KubeOne releases
+  * We highly recommend using containerd instead of Docker for all newly
+    created clusters. You can opt-in to use containerd instead of Docker by
+    adding `containerRuntime` configuration to your KubeOne configuration
+    manifest:
+    ```yaml
+    containerRuntime:
+      containerd: {}
+    ```
+    For the configuration file reference, run `kubeone config print --full`.
+
+
+## Known Issues
+
+* Provisioning Kubernetes 1.20 clusters results with one of the control plane
+  nodes being unhealthy/broken for the first 5-10 minutes after provisioning
+  the cluster. This causes KubeOne to fail to create MachineDeployment objects
+  because the `machine-controller-webhook` service can't be found. Also, one of
+  the NodeLocalDNS pods might get stuck in the crash loop.
+  * KubeOne currently still doesn't support Kubernetes 1.20. We do **not**
+    recommend provisioning 1.20 clusters or upgrading existing clusters to
+    Kubernetes 1.20
+  * We're currently investigating the issue. You can follow the progress
+    in the issue [#1222](https://github.com/kubermatic/kubeone/issues/1222)
+
+## Added
+
+* Add support for containerd container runtime ([#1180](https://github.com/kubermatic/kubeone/pull/1180), [#1188](https://github.com/kubermatic/kubeone/pull/1188), [#1190](https://github.com/kubermatic/kubeone/pull/1190), [#1205](https://github.com/kubermatic/kubeone/pull/1205), [#1227](https://github.com/kubermatic/kubeone/pull/1227), [#1229](https://github.com/kubermatic/kubeone/pull/1229))
+  * Kubernetes has announced deprecation of the Docker (dockershim) support in
+    the Kubernetes 1.20 release. It's expected that Docker support will be
+    removed in Kubernetes 1.22
+  * All newly created clusters running Kubernetes 1.21+ will default to
+    containerd instead of Docker
+  * Automated migration from Docker to containerd is currently not available,
+    but is planned for one of the upcoming KubeOne releases
+
+## Changed
+
+### Bug Fixes
+
+* Fix wrong legacy Docker version on RPM systems ([#1191](https://github.com/kubermatic/kubeone/pull/1191))
+
+### Terraform Configs
+
+* Replace GoBetween load-balancer in vSphere Terraform example by keepalived ([#1217](https://github.com/kubermatic/kubeone/pull/1217))
+
+### Addons
+
+* Fix DNS resolution issues for the Backups addon ([#1179](https://github.com/kubermatic/kubeone/pull/1179))
+
 # [v1.2.0-alpha.0](https://github.com/kubermatic/kubeone/releases/tag/v1.2.0-alpha.0) - 2020-11-27
 
 ## Added

README.md

@@ -8,16 +8,9 @@ Kubermatic KubeOne automates cluster operations on all your cloud, on-prem,
 edge, and IoT environments. KubeOne can install high-available (HA) master
 clusters as well single master clusters.
 
-## KubeOne User Survey
-
-**We're organizing the [KubeOne User Survey][survey]!**
-This survey is intended to shape the future roadmap of KubeOne. Your answers
-will help us determine future features and schedules. We’re raffling one 10€
-Amazon gift card among the respondents of our KubeOne Survey.
-
 ## Getting Started
 
-All user documentation is available at the
+All user documentation for the latest stable version is available at the
 [KubeOne docs website][docs].
 
 Information about the support policy (natively-supported providers, supported
@@ -40,8 +33,9 @@ curl -sfL get.kubeone.io | sh
 ```
 
 The installation script downloads the release archive from GitHub, installs the
-KubeOne binary in your `/usr/local/bin` directory and unpacks the example
-Terraform configs in your current working directory.
+KubeOne binary in your `/usr/local/bin` directory, and unpacks the example
+Terraform configs, addons, and helper scripts in your current working
+directory.
 
 For other installation methods, check the
 [Getting KubeOne guide][docs-install] on our documentation website.
@@ -118,12 +112,12 @@ See [the list of releases][changelog] to find out about feature changes.
 [upstream-supported-versions]: https://kubernetes.io/docs/setup/release/version-skew-policy/#supported-versions
 [cluster-api]: https://github.com/kubernetes-sigs/cluster-api
 [machine-controller]: https://github.com/kubermatic/machine-controller
-[docs]: https://docs.kubermatic.com/kubeone/master/
-[docs-compatibility]: https://docs.kubermatic.com/kubeone/master/compatibility_info/
-[docs-prerequisistes]: https://docs.kubermatic.com/kubeone/master/prerequisites/
-[docs-infrastructure]: https://docs.kubermatic.com/kubeone/master/infrastructure/
-[docs-provisioning]: https://docs.kubermatic.com/kubeone/master/provisioning/
-[docs-install]: https://docs.kubermatic.com/kubeone/master/getting_kubeone/
+[docs]: https://docs.kubermatic.com/kubeone/
+[docs-compatibility]: https://docs.kubermatic.com/kubeone/v1.2/compatibility_info/
+[docs-prerequisistes]: https://docs.kubermatic.com/kubeone/v1.2/prerequisites/
+[docs-infrastructure]: https://docs.kubermatic.com/kubeone/v1.2/infrastructure/
+[docs-provisioning]: https://docs.kubermatic.com/kubeone/v1.2/provisioning/
+[docs-install]: https://docs.kubermatic.com/kubeone/v1.2/getting_kubeone/
 [contributing-guide]: https://github.com/kubermatic/KubeOne/blob/master/CONTRIBUTING.md
 [k8s-slack-kubeone]: https://kubernetes.slack.com/messages/CNEV2UMT7
 [k8s-slack]: http://slack.k8s.io/

docs/proposals/20210112-encryption-roviders.md

@@ -0,0 +1,115 @@
+# Encryption Providers for encrypted secrets at rest
+
+**Auther**: Mohamed Elsayed (@moelsayed)
+**Status**: Draft
+
+
+## Abstract
+
+By default, all Kubernetes secret objects are stored on disk in plain text inside etcd. The [Encryption Providers](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/
+) feature was added to Kubernets starting with version 1.13. 
+
+At rest data encryption is a requirement for security compliance and adds an additional layer of security for secret data, especially when etcd nodes are separated from the control plan and in off-node backups. 
+
+KubeOne needs to support this feature natively. Meaning the user should be able to enable, disable the feature and rotate keys when needed without having to apply any actions manually. 
+
+## Goals
+
+* Provide a safe path to enable/disable Encryption Providers.
+* Support atomic(?) rotation for existing keys.
+* Rewriting all secret resources (no just secrets) after enable/disable/rotate operations.
+
+## Non-Goals
+
+* Deploy External KMS.
+* Safely manage (disable/enable/rotate) configuration when a custom configuration file is used. 
+
+## Challenges
+
+The feature has a lot of moving parts; as it requires performing a specific sequence of actions, including changing the KubeAPI configuration, restarting KubeAPI and rewriting all secret resources to apply the encryption. This requires the implementation to be as idempotent as possible with ability to rollback on failure, with out breaking the cluster. 
+
+## Implementation
+
+Unfortunately, it's not possible to simply update the KubeAPI configuration and expect the configuration to reconcile. KubeOne will have to _read_ the _current_ configuration on the cluster, _mutate_ it based on the _required_ state and then apply it. Additionally, KubeOne will have to be able to revert changes on any errors and recover safely if the process is interrupted at any point.
+
+The configuration for this will be added under `features` in the KubeOneCluster spec:
+
+```yaml
+apiVersion: kubeone.io/v1beta1
+kind: KubeOneCluster
+features:
+  encryptionProviders:
+    enabled: true
+    customProvidersFile: |
+      apiVersion: apiserver.config.k8s.io/v1
+      kind: EncryptionConfiguration
+      resources:
+      - resources:
+        - secrets
+        providers:
+        - identity: {}
+        - aescbc:
+            keys:
+            - name: key1
+            secret: <BASE 64 ENCODED SECRET>
+```
+
+To allow users to rotate the keys, a new flag will be added to the `apply` command:
+
+```bash
+--rotate-encryption-key      automatically rotate encryption provider key
+```
+
+### pre-flight checks
+
+ * Cluster is healthy.
+ * Current Encryption Providers state/configuration is valid and identical on all control plane nodes.
+
+### Enable Encryption Providers for new cluster
+
+* Generate a valid configuration file with the `identity` provider set last.
+* Sync the configuration file to all Control Plane nodes. 
+* Set the required KubeAPI configuration and deploy KubeAPI.
+
+### Enable Encryption Providers for existing cluster
+
+* Ensure there is no Encryption Provider Config (manually added by the user, broken previous enable process, etc..) present.
+* Generate a valid configuration file with the `identity` provider set last.
+* Sync the configuration file to all Control Plane nodes. 
+* Update and restart KubeAPI on all nodes.
+* Rewrite secrets to ensure they are encrypted successfully.
+
+### Disable Encryption Providers for existing cluster 
+
+* Read the current active Encryption Provider configuration from control plane nodes.
+* Mutate the configuration to add `identity` provider first and the active provider last.
+* Sync the configuration file to all Control Plane nodes. 
+* Restart KubeAPI on all control plane nodes.
+* Rewrite secrets to ensure they are decrypted successfully.
+* Update KubeAPI configuration to remove the Encryption Provider configuration and restart KubeAPI on all control plane nodes. 
+* Remove the old configuration file from all control plane nodes.
+
+### Rotate Encryption Provider Key for existing cluster
+
+* Read the current active Encryption Provider configuration from control plane nodes.
+* Generate a new encryption key.
+* Mutate the configuration file to include the new key first, current key second and `identity` last.
+* Sync the updated configuration file to all control plane nodes and restart KubeAPI.
+* Rewrite all secrets to ensure they are encrypted with the new key.
+* Mutate the configuration file again to remove the old key.
+* Sync the updated configuration file to all control plane nodes and restart KubeAPI.
+
+### Apply Custom Encryption Provider file
+This use case is useful for users who would like to utilize an external KMS provider or specify resources other than secrets for encryption. In this case, KubeOne will not manage the content of the file, it will only validate it to make sure it's syntactically valid. Additionally, KubeOne will not rewrite the resources in this case. 
+
+* Ensure the configuration file is valid. 
+* Sync the configuration file to all control plane nodes.
+* Restart KubeAPI on all nodes. 
+
+## Tasks & effort
+
+* Implement the needed pre-flight checks.
+* Implement validation for Encryption Provider configuration files. 
+* Implement the workflow for each use case. 
+* Add e2e tests for each workflow.
+* Add documentation for the feature. 

examples/terraform/azure/variables.tf

@@ -24,7 +24,6 @@ variable "worker_os" {
   # valid choices are:
   # * ubuntu
   # * centos
-  # * coreos
   default = "ubuntu"
 }
 

examples/terraform/digitalocean/variables.tf

@@ -24,7 +24,6 @@ variable "worker_os" {
   # valid choices are:
   # * ubuntu
   # * centos
-  # * coreos
   default = "ubuntu"
 }
 

examples/terraform/gce/variables.tf

@@ -24,7 +24,6 @@ variable "worker_os" {
   # valid choices are:
   # * ubuntu
   # * centos
-  # * coreos
   default = "ubuntu"
 }
 

examples/terraform/hetzner/variables.tf

@@ -24,7 +24,6 @@ variable "worker_os" {
   # valid choices are:
   # * ubuntu
   # * centos
-  # * coreos
   default = "ubuntu"
 }
 

examples/terraform/openstack/variables.tf

@@ -24,7 +24,6 @@ variable "worker_os" {
   # valid choices are:
   # * ubuntu
   # * centos
-  # * coreos
   default = "ubuntu"
 }
 

examples/terraform/packet/variables.tf

@@ -24,7 +24,6 @@ variable "worker_os" {
   # valid choices are:
   # * ubuntu
   # * centos
-  # * coreos
   default = "ubuntu"
 }
 

examples/terraform/vsphere/variables.tf

@@ -24,7 +24,6 @@ variable "worker_os" {
   # valid choices are:
   # * ubuntu
   # * centos
-  # * coreos
   default = "ubuntu"
 }
 

hack/images/kubeone-e2e/Dockerfile

@@ -14,13 +14,13 @@
 
 # building image
 
-FROM golang:1.15.2 as builder
+FROM golang:1.15.7 as builder
 
 RUN apt-get update && apt-get install -y \
     unzip \
     upx-ucl
 
-ENV TERRAFORM_VERSION "0.12.29"
+ENV TERRAFORM_VERSION "0.12.30"
 RUN curl -fL https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip | funzip >/usr/local/bin/terraform
 RUN chmod +x /usr/local/bin/terraform
 
@@ -37,11 +37,11 @@ RUN /opt/install-kube-tests-binaries.sh
 
 # resulting image
 
-FROM golang:1.15.2
+FROM golang:1.15.7
 
 ARG version
 
-LABEL "io.kubeone"="Loodse GmbH"
+LABEL "io.kubeone"="Kubermatic GmbH"
 LABEL version=${version}
 LABEL description="Set of kubernetes binaries to conduct kubeone E2E tests"
 LABEL maintainer="https://github.com/kubermatic/kubeone/blob/master/OWNERS"

hack/images/kubeone-e2e/install-kube-tests-binaries.sh

@@ -17,10 +17,9 @@
 set -euox pipefail
 
 declare -A full_versions
-full_versions["1.16"]="v1.16.15"
-full_versions["1.17"]="v1.17.12"
-full_versions["1.18"]="v1.18.9"
-full_versions["1.19"]="v1.19.2"
+full_versions["1.18"]="v1.18.15"
+full_versions["1.19"]="v1.19.7"
+full_versions["1.20"]="v1.20.2"
 
 root_dir=${KUBETESTS_ROOT:-"/opt/kube-test"}
 tmp_root=${TMP_ROOT:-"/tmp/get-kube"}

hack/images/kubeone-e2e/release.sh

@@ -16,7 +16,7 @@
 
 set -euox pipefail
 
-TAG=v0.1.12
+TAG=v0.1.13
 
 docker build --build-arg version=${TAG} --pull -t kubermatic/kubeone-e2e:${TAG} .
 docker push kubermatic/kubeone-e2e:${TAG}