Activate explicit LB/SG deletion again and improve code to reduce API calls

[rfranzke]

How to categorize this PR?

/area scalability robustness ops-productivity
/kind enhancement
/priority 3
/platform aws

What this PR does / why we need it:
With #290 we had disabled the explicit deletion for clusters >= 1.16. However, we see some occasional issues with leaked LB/SG, so let's reactivate it again and improve the code to reduce the number of API calls.

Special notes for your reviewer:
/assign @ialidzhikov
/invite @ialidzhikov
/squash

Earlier, we listed all LBs and executed a DescribeTags call for each load balancer. Now, we first list LB and execute only one DescribeTags call with all the found LB names. This should significantly improve the overall number of API calls (especially, when many clusters in the same AWS account are deleted in parallel).

Release note:

The load balancers and security groups are again explicitly deleted by the AWS provider extension (independent of the Kubernetes version used by the shoot cluster). The number of API calls have been reduced to the absolute minimum.

[gardener-robot]

@rfranzke Label priority/normal does not exist.

[rfranzke]

/test

[testmachinery]

Testrun: e2e-j2xms
Workflow: e2e-j2xms-wf
Phase: Failed

+---------------------+---------------------+--------+----------+
|        NAME         |        STEP         | PHASE  | DURATION |
+---------------------+---------------------+--------+----------+
| infrastructure-test | infrastructure-test | Failed | 25m27s   |
+---------------------+---------------------+--------+----------+

[rfranzke]

/test

[testmachinery]

Testrun: e2e-5wpxs
Workflow: e2e-5wpxs-wf
Phase: Succeeded

+---------------------+---------------------+-----------+----------+
|        NAME         |        STEP         |   PHASE   | DURATION |
+---------------------+---------------------+-----------+----------+
| infrastructure-test | infrastructure-test | Succeeded | 15m7s    |
+---------------------+---------------------+-----------+----------+

[ialidzhikov]

I played with it locally and it looks good to me. Thanks once again for this PR!
Only one minor suggestion inline.

[rfranzke]

/test

[testmachinery]

Testrun: e2e-b9d5g
Workflow: e2e-b9d5g-wf
Phase: Succeeded

+---------------------+---------------------+-----------+----------+
|        NAME         |        STEP         |   PHASE   | DURATION |
+---------------------+---------------------+-----------+----------+
| infrastructure-test | infrastructure-test | Succeeded | 13m59s   |
+---------------------+---------------------+-----------+----------+

[ialidzhikov]

/lgtm

[vlerenc]

we see some occasional issues with leaked LB/SG

Where are those coming from?

In the end, it doesn't matter too much, I guess, because we need this code for the cluster deletion (w/o a control plane) anyway, but out of curiosity: what's happening there, that Kubernetes (?) leaks IaaS resources? Do we know?

[mvladev]

In the end, it doesn't matter too much, I guess, because we need this code for the cluster deletion (w/o a control plane) anyway, but out of curiosity: what's happening there, that Kubernetes (?) leaks IaaS resources? Do we know?

We delete all Services when cluster is in deletion state, but end-users have automation that re-creates them. So after we finish with the delete operations and mark the cluster as free of any resources, the end-users manage to sneak in a Service create call. cloud-controller-manager creates this LB and is scaled down. We then proceed to delete the infrastructure. The VPC / subnet deletion fails due to dependencies.

[ialidzhikov]

In the end, it doesn't matter too much, I guess, because we need this code for the cluster deletion (w/o a control plane) anyway, but out of curiosity: what's happening there, that Kubernetes (?) leaks IaaS resources? Do we know?

We delete all Services when cluster is in deletion state, but end-users have automation that re-creates them. So after we finish with the delete operations and mark the cluster as free of any resources, the end-users manage to sneak in a Service create call. cloud-controller-manager creates this LB and is scaled down. We then proceed to delete the infrastructure. The VPC / subnet deletion fails due to dependencies.

@mvladev , please don't mix issues and cases. We were observing the leaked SGs and LBs mostly for our vpn-shoot LB and nginx-ingress LB. The issue you talk about is a different one.
@vlerenc , I had in my list to open a follow issue to have an investigation. I will do it later..

[ialidzhikov]

Ref #308