VPA for validator/admission component

[rfranzke]

How to categorize this PR?

/area scalability auto-scaling
/kind enhancement
/priority normal
/platform gcp

What this PR does / why we need it:
VPA for validator/admission component

Release note:

The validator/admission component's Helm chart is now deploying a `VerticalPodAutoscaler` resource by default. If undesired or no VPA is available in the garden cluster then it can be turned of via `.Values.global.vpa.enabled=false`.

[ialidzhikov]

What changed after #141?

[rfranzke]

Well, you know best after you had to manually increase the resource limits yesterday in one of the landscapes, right 😉

From #141 (comment):

I'm mainly worried about downscaling actions performed by VPA. In operations, we observed cases for which a proper upscaling didn't happen after downscaling and in order to recover we needed to delete the VPA object.

Hm, true, OTOH, we use VPA for basically everything and run thousands of clusters, so not sure if this component now is so special. Also, this statement is somewhat older and VPA became better in the meantime. Anyways, the feature is configurable, so if some landscape operator does not like it or it causes too much trouble then it can still be disabled and the old behaviour is preserved. Generally, offering VPA (and also enabling it by default) makes sense.

The admission component directly affects the availability of the Shoot API and thus I'm a bit critical.

True, however, it's more critical if the statically configured resource requirements are not enough anymore and someone gets paged during the night for an incident that could have also auto-resolved via VPA.

/cc @timuthy

[timuthy]

Can we combine this change with a secret sync on webhook start? Then we'd at least avoid a resource spike when the webhooks gets the first request after a while.

[rfranzke]

@timuthy what do you have in mind? Simply listing all secrets on start-up?

[timuthy]

@timuthy what do you have in mind? Simply listing all secrets on start-up?

Yes, I think this will do the trick.

[rfranzke]

@timuthy Here you go: 2645570

/squash

[prashanth26]

/lgtm

Thanks for this.

[vpnachev]

/lgtm