Replace status.observedAt with redis pub/sub channels for resource tree updates

[jessesuen]

Currently, to support the Application watch API, which powers the UI view which displays live changes in a resource tree, the controller updates the k8s application object with a new observedAt timestamp whenever any of the application's resources changes. The Argo CD API server have a K8s watch, and are notified about the update, which forwards the updated application object to the argocd client watchers.

The problem with this approach is:

1. It causes application objects to get updated very frequently (as frequent as an app's resources are updated). This causes a lot of chatter to k8s and unnecessarily bumps the resourceVersion of app objects

2. we unnecessarily involve the k8s api server for notifications about a change in an applications resource tree.

3. for a very active application, sometimes 1-second level granularity of observedAt is not sufficient, and we miss the last event(s) that happened (e.g. a pod going from terminating to missing). A manual refresh fixes this.

The solution is that instead of updating status.observedAt to notify the Argo CD API server about changes in the resource tree view, the API server could subscribe to a redis channel about changes in the resource tree (in addition to the application watch). By doing this, we:

1. no longer need to update the application object which is simply updating a timestamp.
2. we should no longer miss the last event(s) that happens with an application resources

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.