fix(watcher): reconnect after server or client timeout

[buehler]

This fixes #739.
This closes #771.

Allows the resource watcher to retry the connection until the
cancellation token requests a stop. The watcher caches the
received entities and checks for their keys in a concurrent
dictionary. Recurring "added" events after reconnection
should not trigger a reconciliation.

[buehler]

@duke-bartholomew what do you think about these changes? This should fix #739. Further it has a breaking change, but one that is pretty open. The change should not affect many people.

To deduplicate the events that are generated while reconnecting, the entity cache in the watcher should do the trick.

So, theoretically the breaking change would not be required.

[buehler]

I think I'm going to remove the breaking change, because right now there is no real value in it.

[duke-bartholomew]

@duke-bartholomew what do you think about these changes? This should fix #739. Further it has a breaking change, but one that is pretty open. The change should not affect many people.

To deduplicate the events that are generated while reconnecting, the entity cache in the watcher should do the trick.

So, theoretically the breaking change would not be required.

@buehler sorry for my late reply ..
👍 looks good

I would personally also love to have the ability to get a ResourceVersion from the List actions ...
This would give me the ability to first List resources, build up my local state for all of them, then trigger any side-effects on the total state and then start watches to process changes in "real-time" and trigger further side-effects as they happen.
To do this I currently use the underlying IKubernetes client directly, but this is then bypassing your KubernetesClient abstraction and re-implementing a big part of what you already have present in your KubernetsClient, which feels a bit like a shame.
But anyway this is not related anymore to the original issue of the Watcher dying.