Update LB targets when node labels change

[klemenb]

Hi! First, thanks for implementing #514

I am now using this feature, but it appears that the targets only get updated for the first time after I update the Service and they don't update later on if I change labels on the nodes (the same labels that I use in the selector). Is this by design? When would the targets update - e.g. when nodes would get added or removed from the cluster?

What I am trying to achieve is to remove a target from the LB before "kured" reboots the node. This way I should get really close to zero-downtime node reboots as the node would be removed while it is still alive and any connections can probably be cleanly drained (not sure about the internals of Hetzner LB when you remove a target, but at least in theory this could work perfectly). Kured can apply additional labels to the nodes before rebooting and I can use the selector to exclude nodes with that label from the LB targets. The only missing part is now to actually trigger the update of the LB targets when the label gets added to the node.

So - 3 questions:

1. Is it possible to use the existing "load-balancer.hetzner.cloud/node-selector" functionality to update targets when node labels get changed (on existing nodes)?

2. If not, is there an easy way to add this functionality?

3. Is there any other mechanism that can help me achieve what I described above?

Thank you for all the work on the project so far and I appreciate any help you can provide :)

[apricote]

Mh, I tested this by updating the annotation on the Service, this triggered an immediate reconcile of the LB. Looking at k8s.io/cloud-provider it seems that our method is only called when the library sees a change to the full list of nodes, but our filtering only happens afterwards: https://github.com/kubernetes/cloud-provider/blob/73ce03d745d168cd8169b5f9ebfde4120b9ca82f/controllers/service/controller.go#L735-L761

There is nothing we can do here really, the fix needs to come from k/cloud-provider.

[klemenb]

Thanks for the explanation.

I will look into watching these changes manually and triggering an update to the Service.