[ci:component:github.com/gardener/gardener:v1.44.6->v1.50.1]

[gardener-robot-ci-3]

Release Notes:

It is no longer possible to perform the following shoot operations when it is hibernated: `rotate-{credentials,etcd-encryption-key,serviceaccount-key}-{start,complete}`.

In case at least one shoot cluster CA certificate is about to expire in less than `1y`, a new constraint of type `CACertificateValiditiesAcceptable` will be visible in the `.status.constraints` to make end-users aware that a rotation should be performed.

Images in image vector now support a new field `architectures`. It is a list of CPU architecture of machines on which one image can be used. If not specified images are considered to support both `amd64` and `arm64` CPU architecture.

Files in `./extensions/test` have been moved to `./test` package, Please adapt the import paths accordingly:
- `https://github.com/gardener/gardener/tree/master/extensions/test/testmachinery` has been moved to `https://github.com/gardener/gardener/tree/master/test/testmachinery/extensions`
- `https://github.com/gardener/gardener/tree/master/extensions/test/integration` has been moved to `https://github.com/gardener/gardener/tree/master/test/integration/extensions/controller`

`gardener-apiserver`, `gardener-controller-manager`, `gardener-scheduler`, `gardener-admission-controller`, `gardener-seed-admission-controller` and `gardener-resource-manager` are now using `gcr.io/distroless/static-debian11:nonroot` instead of versions of `alpine` as a base image.

The following images are updated:
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler:` `v0.19.0` -> `v1.20.1` (for Kubernetes `< 1.20`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler:` `v1.20.0` -> `v1.20.1` (for Kubernetes `1.20`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler:` `v1.21.0` -> `v1.21.1 `(for Kubernetes `1.21`)
- `eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler`: `v1.21.0` -> `v1.22.1` (for Kubernetes `>= 1.22`)

It is now possible to render charts from embedded file systems (`embed.FS`). The `Render` method of the `chartrenderer.Interface` in favour of `RenderEmbeddedFS`. The `Apply`/`Delete` methods of the `kubernetes.ChartApplier` interfaces are deprecated and in favor of `{Apply,Delete}FromEmbeddedFS`. They will be removed in a future version. You should consider adapting your code to the newly introduced methods.

The `WorkerPoolKubernetesVersion` feature gate has been promoted to GA and is now unconditionally enabled. Make sure that all provider extensions registered to your system support this feature before upgrading to this Gardener version.

The `--secure-port` flag of the Gardener API Server can now be configured through the helm chart by setting `.Values.global.apiserver.securePort`. The default value is `8443`. The service exposing the Gardener API Server deployment will continue to listen on port `443`.

Upgrade `node-exporter` to `v1.3.1`

`SeedKubeScheduler`: `gardenlet` does now support the `SeedKubeScheduler` feature gate to be enabled for K8s `1.24` Seed clusters.

A bug has been fixed which could prevent `gardenlet` pods from coming up in case the `projected-token-mount` webhook served by `gardener-resource-manager` is unavailable or broken.

`CloudProfile`s now supports two new fields `.spec.machineImages[].architectures` and `.spec.machineTypes[].architecture`. 
`.spec.machineImages[].architectures` - It is a list of CPU architectures of the machine image supported by the particular machine image version.
`.spec.machineTypes[].architecture` - It specifies the CPU architecture of the given machine type.

Worker now supports a new field `.spec.pools[].architecture`. It specifies the CPU architecture of the machine in the given worker pool.

In order to trigger control-plane migration using the `shoots/binding` subresource, please see https://github.com/gardener/gardener/blob/master/docs/usage/control_plane_migration.md#triggering-the-migration

Patches to `spec.seedName` field in the shoot manifest will be rejected. Please use the `shoots/binding` subresource instead.

Seed prometheus: allow to overwrite scheme per annotation for job garden.

Adapt `allow-to-dns` networkpolicy to also work with node local dns in cilium case.

A bug has been fixed which prevented the etcd defragmentation from running properly. This fix will cause a restart of all etcd instances during the next maintenance time window.

Activate the `diskstats` collector for the node_exporter

Gardenlet now manages fine-granular `PriorityClasses` that are supposed to be used by all components in order to improve the overall robustness of the system.
- Find out more in the related [documentation](https://github.com/gardener/gardener/blob/master/docs/development/priority-classes.md).
- Extensions need to migrate all their extension controller pods as well as their shoot control plane and shoot system components to the newly defined `PriorityClasses` and drop custom ones.
- Legacy `PriorityClass` `gardener-shoot-controlplane` is deprecated and will be removed in a future release.

The GA-ed `CachedRuntimeClients`,  `AdminKubeconfigRequest`, `DenyInvalidExtensionResources` and `UseDNSRecords` feature gates are removed and can no longer be specified via the `--feature-gates` flags.

A bug has been fixed which prevented the assignment of the `ERR_CLEANUP_CLUSTER_RESOURCES` error code to `Shoot`s.

The default value for `--audit-log-path` of Gardener API Server was changed from `/var/lib/audit.log` to `/tmp/audit.log` so that a `nonroot` user can access it without additional permissions.

The `Shoot` API now supports a new field `spec.provider.workers[].machine.architecture`. It specifies the CPU architecture of the machine in a given worker pool of shoot. It must match the architecture of the used machine type and machine image as defined in the referenced `CloudProfile`.

Pause container image is now used from `k8s.gcr.io/pause` instead of `gcr.io/google_containers/pause-amd64`.

The recent changes to the "github.com/gardener/gardener/extensions/pkg/controller/healthcheck/config".HealthCheckConfig type that added client configuration settings are now reverted.

New condition `SeedSystemComponentsHealthy` is added to the `seed` object to indicate the status of the system components.

Updates istio components to v1.12.5

Adds retry handling in case of errors that can happen when the gardener controller manager attempts to hibernate shoot clusters according to the hibernation schedules configured in `shoot.spec.hibernation.schedules`

A new API diff check has been added to ensure PRs are not changing exported interfaces, types, or method signatures in incompatible ways.

The generic error code mapping in Gardener is deprecated. Extensions should use their own error code mappings and should return corresponding error codes to Gardener.

Additional reconciliations for resources after adding the finalizer are prevented using an early exit approach.

VPA binaries and dependency have been upgraded to 0.10.0.

Fixed an issue that could cause the cloud-config-downloader to invalidate its credentials token if the node that it is currently running on has issues with the file system where the credentials token is stored (for example when the node runs out of disk space).

The `ShootMaxTokenExpirationOverwrite` feature gate has been promoted to beta and is now enabled by default.

Gardener resource manager can now properly deploy v1beta1 `CronJobs` if they are part of a `ManagedResource`'s referenced `Secret`

Support for Terraformer `v1` was finally dropped.

Update alpine image to version `3.15.3` and update dependencies.

Gardenlet will now update its kubeconfig if `gardenClientConnection.gardenClusterCACert` is specified and contains a different CA cert than the one currently used in the kubeconfig.

Update api-server-proxy to `v0.3.0`.

Fix kube-proxy switch from IPVS to IPTables mode.

`RotateSSHKeypairOnMaintenance` feature gate in gardener-controller-manager has been promoted to `beta` and is now enabled by default.

Introduce feature gate `HAControlPlanes` in alpha state for gardenlet and gardener-scheduler. :warning: This comes with a change to the certs used, which will cause a restart of the etcds.

Increase the QPS and burst values for `kube-apiserver` requests for the `vpa-recommender` of Seed and Shoot clusters to better cope with large cluster sizes.

Fix defragmentation fail issue which occurs due to x509: failed to validate certificate for 0.0.0.0 because it doesn't contain any IP SANs.

Loki memory limit is increased to 3Gi.

`CachedRuntimeClients` feature gate in `gardener-controller-manager`, `gardenlet` is promoted to GA and cannot be disabled.

An issue preventing nodes from updating their downloaded cloud config checksum annotation has been fixed.

A bug has been fixed which prevented the migration of existing basic auth secrets without CSV data to the new secrets manager.

In order to save network I/O and costs, the `cloud-config-downloader` script running every `30s` on each shoot worker node now first performs a metadata-only request for the cloud config `Secret`. It only downloads the full secret (including data containing the `executor` script) if the checksum annotation has changed.

Fix a bug in the `PodDisruptionBudget` of the Gardener API server that was not allowing maintenance operations with the hosting cluster when the HVPA is enabled the replicas are set to 1.

The local Gardener development setup now uses `calico` instead of `kindnetd` as CNI plugin for the seed and shoot clusters. This enables support for `NetworkPolicy`s and rolling updates of shoot worker nodes.

Newly created shoot clusters now get a dedicated CA certificate which is used for signing client certificates. Note that this client CA is different from the cluster CA. For existing clusters, the client CA is the same like the cluster CA to ensure backwards compatibility.

The seed cluster CA certificate is now auto-rotated each `30d`.

Temporarily no longer allow changing container runtime on existing workers due to an open bug: [#4415](https://github.com/gardener/gardener/issues/4415).

The `DenyInvalidExtensionResources` feature gate in the `seed-admission-controller` has been promoted to GA and can no longer be disabled.

In case `gardener-resource-manager` fails to be bootstrapped because its client certificate has expired, `gardenlet` does now automatically generate a new client certificate and re-triggers the bootstrap process.

A bug has been fixed which can result in `Shoot`s stuck in deletion when the `ShootMaxTokenExpiration{Overwrite,Validation}` feature gates are enabled.

Extensions need to prepare for supporting the [Shoot CA rotation feature](https://github.com/gardener/gardener/issues/3292) ([GEP-18](https://github.com/gardener/gardener/blob/master/docs/proposals/18-shoot-CA-rotation.md)). Please see [CA Rotation in Extensions](https://github.com/gardener/gardener/blob/master/docs/extensions/ca-rotation.md) and [Conventions](https://github.com/gardener/gardener/blob/master/docs/extensions/conventions.md) for detailed descriptions of the requirements.

The extension library has been adapted to support the [Shoot CA rotation feature](https://github.com/gardener/gardener/issues/3292) ([GEP-18](https://github.com/gardener/gardener/blob/master/docs/proposals/18-shoot-CA-rotation.md)). Please see [gardener/gardener#5803](https://github.com/gardener/gardener/pull/5803) for a detailed description on how to adapt to the breaking changes.

Worker's `RollingUpdate` condition is removed as it was not used.

The Golang version is updated to `1.17.9`.

The lastActivityTimestamp of the project is now updated every time a plant, backupEntry or shoot is created or a quota or secret in the project namespace is referred by a secretbinding. The timestamp is also updated when these resources are updated or deleted.

`addons-nginx-ingress-controller`, `kubernetes-dashboard`, `blackgox-exporter` no longer have lower memory limits when VPA is enabled.

The HVPA controller now respects `controlledResources` and `controlledValues` parameters that have been newly introduced in `autoscaling.k8s.io/v1`.

`autoscaling.k8s.io/v1` is now being used instead of `autoscaling.k8s.io/v1beta2` in HVPA resources. This enables using `controlledValues: RequestsOnly` in `spec.vpa.template.spec.resourcePolicy`

Fixed a bug that caused `make gardener-up` to fail.

Only requests but not limits of an existing `kube-apiserver` deployment are copied when HVPA is enabled to allow limits to be removed from existing deployments.

The hostname can be inserted into the log label stream via configuration.

Remove some security vulnerabilities by re-vendoring.

Loki's curator does fewer slice allocations when deleting files.

Loki's curator closes the opened directories after each deletion.

Loki's curator profiling is available via HTTP `pprof` API open on `2718` port.

Documentation for accessing the shoot cluster is added [here](https://github.com/gardener/gardener/blob/master/docs/usage/shoot_access.md).

Fixes an issue that occurs during the control plane migration flow when the shoot's control plane namespace on the source seed is being terminated and the flow is restarted before the namespace has been completely deleted.

The `WorkerPoolKubernetesVersion` feature gate has been promoted to beta and is now enabled by default. Make sure that all provider extensions registered to your system support this feature before upgrading to this Gardener version.

Throw Fatal error to avoid edge case potential deadlocks.

ETCD won't restart from the PVC if it is wrongly mounted to the pod.

The webhook for auto-mounting projected service account tokens now also considers init containers.

An issue causing gardener-resource-manager to not be scaled up (and afterwards the Shoot reconciliation to be stuck) after a failed hibernation attempt is now fixed.

A potential issue causing control plane Secrets to be wrongly deleted due to a failed (or not yet executed) task is now fixed.

The `ShootMaxTokenExpirationValidation` feature gate has been promoted to beta and is now enabled by default.

Container images are now being build and published also for `arm64` platforms.

Changing the default `ServiceAccount` issuer to a custom issuer for shoot clusters is now supported.

There is a [new document](https://github.com/gardener/gardener/tree/master/docs/usage/shoot_serviceaccounts.md) explaining the various configurations (and caveats) regarding the `ServiceAccount` configuration for shoot clusters.

It is now possible to remove the CA bundle from the gardenlet kubeconfig by setting `gardenClientConnection.gardenClusterCACert` to either `none` or `null`.

The golang version is now updated to `1.18.1`.

Make sure that all your shoots have been reconciled with Gardener `v1.45` before upgrading to this version. Generally, make sure that you are following the [supported upgrade order](https://github.com/gardener/gardener/blob/master/docs/deployment/version_skew_policy.md#supported-component-upgrade-order).

An issue causing the istiod validating webhook's `clientConfig.caBundle` to be not populated is now fixed.

The status of `ManagedResources` now contains a new condition of type `ResourcesProgressing` which can be used to detect whether updates to managed resources have been fully rolled out.

An issue causing the controlplane migration integration tests to always fail is now fixed.

Gardenlet memory limit was removed, according to measured usage, to prevent OOMKills due to reaching the limits.

Update envoy-proxy to v1.21.2

Remove resource limits from etcd resources for existing clusters. In conjunction with the etcd-druid changes in https://github.com/gardener/etcd-druid/pull/342, this can lead to a etcd-pod RESTART (!).

provider-local now supports `Ingress` objects in the Seed cluster and now enables using the [shoot node logging feature](https://github.com/gardener/gardener/blob/master/docs/deployment/configuring_logging.md#enable-logs-from-the-shoots-node-systemd-services).

New annotation `gardener.cloud/operation=rotate-observability-credentials` is introduced to enable end user to rotate the credentials to access the observability stack via Grafana.

The operator  `observability-ingress` credentials are now auto-rotated every 30 days.

Seed log processor images `fluent-bit` is updated from `1.8.7` to `1.9.3`.
Shoot Loki side car `kube-rbac-proxy` image is updated from `v0.8.0` to `v0.12.0`.
Shoot Loki side car `telegraf-iptables` base image is updated from `1.18.0-alpine` to `1.22.3-alpine`.
Seed log processor plugin `fluent-bit-plugin` base image is updated from `alpine:3.12.3` to `alpine:3.15.4`

The name of the seed node is added to the log stream.

CoreDNS does no longer support wildcard dns queries.

Both the seed's and the shoot's Grafana instances now serve dashboards for the `gardener-resource-manager`.

The version of the `nginx-ingress-controller` addon has been bumped to `0.49.3` for shoots < 1.22, and to `1.2.0` for shoots >= 1.22. The version of the `kubernetes-dashboard` has been bumped to `2.2.0` for shoots < 1.21, to `2.4.0` for shoots = 1.21, and to `2.5.1` for shoots >= 1.22. The version of the `kubernetes-dashboard-metrics-scraper` has been bumped to `1.0.7` for all shoots.

The version of the `nginx-ingress-controller` addon has been bumped to `0.49.3` for seeds < 1.22, and to `1.2.0` for seeds >= 1.22.

provider-local now allows to enable the [`dependency-watchdog-probe`](https://github.com/gardener/gardener/blob/master/docs/usage/seed_bootstrapping.md#dependency-watchdog) in the seed cluster.

The default shoot creation e2e test now also tests for the `AdminKubeconfigRequest` feature.

A [new document](https://github.com/gardener/gardener/tree/master/docs/usage/shoot_credentials_rotation.md#certificate-authorities) related to the rotation of the CA certificate rotation has been added.

Fixed retrieval of credentials during copy operation for backups stored in Swift snapstore.

Update alpine images to version `3.15.4`.

The `UpdateFunc` predicate in the extensions library is modified to allow reconciliation of object on change in timestamp when the `Shoot` is in `Error` state.

Fix a blackbox exporter configuration issue (path to shoot CA) that resulted in false positive "ApiServerNotReachable" alerts

Cluster Autoscaler version of the same major and minor version as K8s running on the shoot, is now deployed in the control plane, starting with k8s >=1.20 shoots. For others v0.19.0 of autoscaler is deployed.

NodeTemplate is not formed using cache in case nodegrp has minimum size zero. This is done to keep nodeTemplate updated even when instance type of nodegrp is updated.

cluster-autoscaler is now synced with upstream v1.21.0

Loki StatefulSet fsGroupChangePolicy is changed from "Always" to "OnRootMissmatch" in order to increase the Loki pod creation when moved from one node to another

Do not re-use resource limits from an existing etcd  stateful set. This will cause a RESTART(!) of the etcd pod for existing clusters that currently have a resource limit set for the etcd stateful-set, but whose etcd resource does not specify a resource limit.

When the owner check fails, `etcd-backup-restore` will restart the `etcd` process right before attempting to take a final snapshot, if the owner check was previously successful.

`gardener-resource-manager` is now (re-)bootstrapped in case its token got invalidated.

A new alpha `ShootSARotation` feature gate (disabled by default) has been introduced which allows the rotation of the service account signing key secrets for shoot clusters.

The delete/modify permissions for `ServiceAccount`s assigned to `Project` members with the `admin` role are now deprecated and will be removed in a future version of Gardener. In order to manage `ServiceAccount`s in the project namespace, use the `serviceaccountmanager` role. Please find more information [here](https://github.com/gardener/gardener/blob/master/docs/usage/projects.md).

The `.spec.kubernetes.kubeAPIServer.serviceAccountConfig.signingKeySecretName` field is deprecated now and will be removed in a future version. If you use this field for `Shoot`s make sure to recreate them as soon as possible since there is no option to migrate away from it.

Upgrade Prometheus to v2.35.0

The `RotateSSHKeypairOnMaintenance` feature gate is now deprecated and disabled by default. It will be removed in a future version of Gardener. If you rely on it then you can implement an equivalent workflow by annotating `Shoot`s with `gardener.cloud/operation=rotate-ssh-keypair` during their respective maintenance time windows.

Added e2e integration test for control plane migration

Added wrapper scripts and `make` targets that can be used to setup the `skaffold` test environment and trigger e2e integration tests: `make ci-e2e-kind` can be used to trigger the default e2e integration tests; `make ci-e2e-kind-migration` can be used to trigger the control plane migration e2e test.

A `ManagedSeed` can now be annotated with `gardener.cloud/operation=renew-kubeconfig` to recreate the gardenlet's kubeconfig secret.

Monitoring dashboards of node local dns should work again.

A corner case in gardenlet's logic about detection of misconfigured webhook is addressed. Previously a webhook for namespaces that properly ignores the kube-system namespace was wrongly considered as "problematic".

Metrics about machine boot times are added to the monitoring stack.

The status of the `SeedSystemComponentsHealthy` and `Bootstrapped` seed conditions is set to `Progressing` at the start of seed reconciliations.

Upgrade blackbox-exporter to v0.20.0

Additional dashboard for node-local-dns errors.

An issue causing the `lastTransitionTime` and `lastUpdateTime` of the `SeedRegistered` condition of a `ManagedSeed` to be unnecessary updated on each reconciliation is now fixed.

The extension controller webhook certificates are now auto-rotated each `30d`.

In order to support auto-rotation of the extension webhook certificates, the following breaking changes were introduced:
- the generic `ControlPlane` actuator no longer accepts `[]admissionregistrationv1.MutatingWebhook` for the shoot webhooks but an `*atomic.Value`
- the `webhookOptions.Completed().AddToManager` function no longer returns two `[]admissionregistrationv1.MutatingWebhook` (one for seed webhooks, one for shoot webhooks) but only one `*atomic.Value` for the shoot webhooks (which can be used for the generic `ControlPlane` actuator)
- It is no longer necessary to call `ReconcileShootWebhooksForAllNamespaces` explicitly via a `Runnable` on start-up/leader-election (hence, this code can be dropped).

Upgrading to this Gardener version is only possible from `v1.47` (as suggested in https://github.com/gardener/gardener/blob/master/docs/deployment/version_skew_policy.md#supported-component-upgrade-order).

It is now possible again to migrate the CRIs for existing worker pools in shoot clusters.

The ingress default backend has been switched to `k8s.gcr.io/defaultbackend-amd64:1.5`.

The `ManagedIstio` and `APIServerSNI` feature gates are now deprecated. They are already turned on by default and will be removed in a future version of Gardener. If you don't use them yet, turn them on now so to ensure a smooth migration to the future Gardener release.

The seed setting for disabling DNS for shoots is now deprecated and will be removed in a future version of Gardener. Make sure to recreate your shoot clusters on such seeds with DNS enabled.

The "Kubernetes Pods" grafana dashboard now allows to select multiple pods at once.

An issue causing Pod creation to fail for the node-local-dns DaemonSet when privileged containers are not allowed is now fixed.

A custom validity of secrets is now properly respected. Earlier, it was overwritten and regenerated in each reconciliation which technically led to the situation in which such secrets were never auto-rotated when their intentional validity was expired.

Changing the `spec.seedName` for `Shoot`s is now possible only via the new `shoots/binding` subresource. Patches to `spec.seedName` in the `Shoot` will not have any effect anymore. Please see [this document](https://github.com/gardener/gardener/tree/master/binding/docs/concepts/scheduler.md#shootsbinding-subresource) for more information.

The shoot node network is no longer allowed to overlap with the seed service network.

Memory limits of various components were updated, based on measured usage, to prevent OOMKills due to reaching the limits. Limit scaling was disabled to prevent limit downscaling during periods of load system load.

It is now possible to trigger the rotation of the ETCD encryption key secret for shoot clusters. Please consult the [documentation](https://github.com/gardener/gardener/tree/master/docs/usage/shoot_credentials_rotation.md#etcd-encryption-key) for more information.

Gardener can now support shoot and seed clusters with Kubernetes version 1.24. In order to allow creation/update of 1.24 clusters you will have to update the version of your provider extension(s) to a version that supports 1.24 as well. Please consult the respective releases and notes in the provider extension's repository.

Gardener can now support seed and shoot clusters with Kubernetes version 1.24. Extension developers have to prepare individual extensions as well to work with 1.24.

fix allow-fluentbit networkpolicy with empty cird

add additionalEgressIpBlock for fluentbit networkpolicy to gardenlet config

making blackbox-exporter on shoots highly-available, to prevent false positive alerts during rollouts of blackbox-exporter, apiserver-proxy and worker nodes

Add `SchedulingConstraints` field to Etcd spec. The currently supported constraints are `Affinity` and `TopologySpreadConstraints`.

Multi-node etcd bootstrapping is now supported. This is an alpha feature intended for initial use and evaluation. Please do not enable this feature for your productive workloads
Multi-node etcd restoration from backup buckets is not supported
Intended to work only with etcd-druid v0.9.x and beyond
Intended to work only with etcd-custom-image v3.4.13-bootstrap-4 and beyond

A bug has been fixed which prevented the `ServiceAccount`'s `automountServiceAccountToken` field from being reconciled.

If Spec.Replicas in ETCD CR is greater than 0 and a even number, then no statefulset for ETCD nodes will be created and so the ETCD cluster won't be setup by Druid

Deployed configmap programmatically as component instead of chart
- configmap configures ETCD config based on the number of nodes in cluster. Number of nodes in cluster is derived from spec.Replicas of ETCD CR

Introduced separate TLS config for client and peer communication with ETCD cluster. The previous Etcd resource field `spec.etcd.tls` is now deprecated and removed.

Fix `make test` and `make test-integration` for M1 Macbooks

When evaluationg the `SeedSystemComponentsHealthy` condition, the `ResourcesProgressing` condition of `ManagedResource`s is now also considered.

An issue causing nil pointer dereference in gardenlet when `shoot.spec.kubernetes.enableStaticTokenKubeconfig` is set to `false` is now fixed.

Keep all the memory metrics in the seed prometheus

Updates to the `extensions.BackupEntry.Spec.Region` field are now allowed.

There are two new `rotate-credentials-{start,complete}` operation annotations for `Shoot`s which can be used to start or complete the rotation of all Gardener-provided/Gardener-generated credentials.

With the new `maintenance.gardener.cloud/operation` annotation for `Shoot`s it is now possible to confine the execution of the respective operation to the shoot cluster's maintenance time window.

Additional step in the shoot deletion flow called `Deleting metrics-server`. This step explicitly deletes the metrics server before the `Cleaning shoot namespaces` step.

Provider extensions using the generic `controlplane` mutator webhook can now mutate the `cluster-autoscaler` Deployment by implementing the `EnsureClusterAutoscalerDeployment` function. This is required in the context of https://github.com/kubernetes/autoscaler/issues/4517 - cluster-autoscaler supports `--feature-gates` flag and provider extensions have to mutate the cluster-autoscaler Deployment to add the CSI related feature flags to it.

The `ShootMaxTokenExpiration{Validation,Overwrite}` feature gates have been promoted to GA and are always enabled.

The deprecated client certificate for Prometheus has been dropped. Extensions still relying on it must now adapt their scrape configurations according to the [documentation](https://github.com/gardener/gardener/blob/master/docs/extensions/logging-and-monitoring.md#extensions-monitoring-integration).

An issue causing the `vpn-seed` container to not be able to connect to the `kube-apiserver` during Shoot CA rotation when `ReversedVPN` feature gate is disabled is now fixed.

An issue preventing the cluster-autoscaler to watch `csidrivers` and `csistoragecapacities` is now fixed.

The generic Worker actuator now scales up machine-controller-manager Deployment when Shoot is hibernating (or waking up) and machine-controller-manager Deployment is already scaled down by external actor (dependency-watchdog).

Combine systemd services logs in one Loki stream except `docker`, `containerd`, `kubelet`, and `kernel`.

Logging usage documentation is updated.

NodeLocalDNS can now be enabled via the shoot specification, nodes are rolled in case NodeLocalDNS is switched. Each node gets an additional label indicating the state of NodeLocalDNS at this node.

Adds ability to create a second seed cluster in the local setup.

A bug has been fixed which could have caused orphaned `ServiceAccount` token `Secret`s after the rotation of the signing key.

A full snapshot of `etcd-main` is now triggered after all `Secret` were encrypted with the new key after ETCD encryption key rotation.

Fixed a bug with the `gardener.cloud/operation: renew-kubeconfig` annotation for `ManagedSeed` resources, which caused the corresponding gardenlet to break when the annotation was set.

When annotating shoots with `gardener.cloud/operation` or `maintenance.gardener.cloud/operation` apiserver now validates if the respective operations are supported.

Update coredns to v1.9.3.

A bug has been fixed which might cause `ServiceAccount`s to still reference old static token `Secret`s after the rotation of the `ServiceAccount` signing key.

A bug has been fixed which could allow the gardenlet performing rotation of certificate authorities or `ServiceAccount` signing keys even if the respective feature gates were disabled.

- An initial-cluster field is now expected in the ETCD config

Introducing a timeout `timeoutToOpenBoltDB` to open boltDB within a given time, so backup-restore won't have to wait for ever.

When the owner check fails, `etcd-backup-restore` will restart the `etcd` process right before attempting to take a final snapshot, if the owner check was previously successful.

Fixed retrieval of credentials during copy operation for backups stored in Swift snapstore.

Throw Fatal error to avoid edge case potential deadlocks.

- ETCD won't restart from the PVC if it is wrongly mounted to the pod

OCS S3 Snapstore now supports supplying access information via a mounted secret.

The base image of etcd has been set to Alpine 3.15.4.

A bug has been fixed that deleted member `lease` objects in all namespaces. With this release member lease renewals are enabled again.

Fixed a bug where druid did not copy etcd labels to configmap

Do not re-used resource limits from an existing etcd  stateful set. This will cause a RESTART(!) of the etcd pod for existing clusters that currently have a resource limit set for the etcd stateful-set, but whose etcd resource does not specify a resource limit.

ETCd backups can now be successfully copied between OCS buckets.

Paths transformations in .docforge/manifest.yaml for simplification

A bug has been fixed which prevented extension controllers to register shoot webhooks only (w/o any seed webhooks).

Fix a bug causing nil pointer exceptions when configuring the webhook server for local development

You can now make the `gardenlet` remediate problematic webhooks in shoot clusters by setting  `.controllers.shootCare.webhookRemediatorEnabled=true` in its configuration file.

Gardener landscape administrators are now provided with `serviceaccountmanager` permissions, i.e. they can manage service accounts and issue tokens for them.

A disruption free CA rotation is now being supported for HA shoot clusters.

A bug was fixed which caused current, accidental resource limit values for the loki container of the loki component, to be established as fixed limits, in place of the correct absolute limit value.

`k8s.io/*`  is now upgraded to `v0.24.1` and `sigs.k8s.io/controller-runtime` is now upgraded to `v0.12.1`.

If you use multi-zonal seed clusters (those labelled with `seed.gardener.cloud/multi-zonal`), then the Gardener Scheduler will only consider them for multi-zonal shoot clusters. Normal or single-zonal shoot clusters will not be scheduled there.

Node local dns components will stay in the cluster until after the node roll out of a node local dns switch and be cleaned up in the next reconcile.

Increased the VPA Recommender log level to v=3.

A bug has been fixed which can cause the `gardener-resource-manager` deployment in the shoot namespaces to mount a `ServiceAccount` token secret from a different namespace.

There are now client warnings for `Shoot` resources when credentials rotation is due or when the static token kubeconfig is used.

Allow passing custom REST configuration settings (QPS, Burst, RateLimiter, Timeout) to extension shoot clients.

The version of the `nginx-ingress-controller` addon has been bumped to `1.2.1` for shoots and seeds >= 1.22.

Upgrade grafana to 7.5.16

Fix an issue where the HVPA would set Requests higher than Limits if `ControlledValues: RequestsOnly` is set

It is now possible to override the maximum delay seconds for the cloud-config user-data execution on shoot worker nodes by specifying the `shoot.gardener.cloud/cloud-config-execution-max-delay-seconds` annotation on the `Shoot` resource (default: `300`).

Fix the "Target" variable of the vpa-recommendations dashboard

Allow cilium to be used on seeds with SNI enabled.

Improve the CPU and memory usage calculation on the Node Details dashboard

Update node local dns to v1.22.5

Allow updates of old shoot clusters that were already created with an invalid default domain before the validation was introduced.

The blackbox exporter scrape probe logs are also written to stdout

Bump prometheus to v2.36.1

The `DisableDNSProviderManagement` feature gate has been promoted to beta and is now enabled by default. If you are using the Gardener extension `shoot-dns-service` make sure to deploy version >= `v1.20.0` and to set `providerConfig.values.dnsProviderManagement.enabled=true` in its [controller deployment](https://github.com/gardener/gardener-extension-shoot-dns-service/blob/master/example/controller-registration.yaml#L9). The shoot DNS service  admission controller (`gardener-extension-admission-shoot-dns-service`) must be deployed on the garden cluster.

`NetworkPolicy/allow-to-private-networks` now allows access to networks overlapping the shoot networks in case reversed VPN is active.

`kube-apiserver` and `prometheus` pods are no longer allowed to access shoot networks in case reversed VPN is active.

[gardener-robot]

@gardener-robot-ci-3 Thank you for your contribution.