Consider using WaitForFirstConsumer volumeBindingMode for gardener.cloud-fast StorageClass

[ialidzhikov]

How to categorize this issue?

/area storage
/area control-plane
/kind enhancement
/priority 3
/platform aws

What would you like to be added:
#152 outlines advantages of volumeBindingMode WaitForFirstConsumer over Immediate.

By default, the Immediate mode indicates that volume binding and dynamic provisioning occurs once the PersistentVolumeClaim is created. For storage backends that are topology-constrained and not globally accessible from all Nodes in the cluster, PersistentVolumes will be bound or provisioned without knowledge of the Pod's scheduling requirements. This may result in unschedulable Pods.

As part of #152 we changed the default volumeBindingMode to WaitForFirstConsumer for the StorageClasses in the Shoot.

But the gardener.cloud-fast StorageClass that is deployed in the Seed and used for the etcd PVC still uses volumeBindingMode Immediate.

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    resources.gardener.cloud/description: |-
      DO NOT EDIT - This resource is managed by gardener-resource-manager.
      Any modifications are discarded and the resource is returned to the original state.
    resources.gardener.cloud/origin: garden/provider-aws-pmjkz
  creationTimestamp: "2021-06-13T14:30:42Z"
  labels:
    app.kubernetes.io/instance: provider-aws
    app.kubernetes.io/name: gardener-extension-provider-aws
  name: gardener.cloud-fast
  resourceVersion: "273262060"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/gardener.cloud-fast
  uid: d7f132ff-7439-4697-88b3-25e007498392
parameters:
  type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: Immediate

Anything else we need to know?:
This issue should be applicable also for the other provider extensions. I just don't want to duplicate the same issue multiple times.

[ialidzhikov]

/cc @vlerenc @rfranzke