Update observedGeneration in reconcileStatus flow instead of reconcileSpec flow

[shreyas-s-rao]

How to categorize this PR?

/area usability
/kind impediment

What this PR does / why we need it:
Follow-up to #987

This PR enhances the reconciliation flow such that observedGeneration is updated in reconcileStatus flow instead of reconcileSpec flow.

Reason: Upon a spec reconciliation request (spec change or gardener.cloud/operation: reconcile annotation, depending on the reconcile strategy), the observedGeneration was updated in reconcileSpec flow, while the other status fields like conditions, sts-related fields, etc were updated later in reconcileStatus flow. After the user updates the spec, and waits for the spec changes to be reflected in the etcd cluster (which are accurately reflected by conditions), it can so happen that the observedGeneration is updated first, followed by a gap before the conditions are updated. This can lead to instances when the user watches the status and assumes that the spec has been fully reconciled and pods are updated with latest changes, because the previous conditions still applied while the observed generation got updated.

To avoid this, the observedGeneration will now be updated along with the rest of the status fields, in one single status patch call, so that at any given point of time, the user can check both status.observedGeneration and status.conditions to know whether the spec changes have been fully rolled out.

Additionally, the operation annotation is conditionally removed at the end of both spec and status reconciliation, to ensure that any failure in either spec or status reconciliation will result in a fresh spec reconciliation upon requeue, and will not get missed. This will ensure that the observedGeneration gets correctly updated, even if druid restarts in between reconciliation flow.

Which issue(s) this PR fixes:
Fixes #985

Special notes for your reviewer:
/invite @unmarshall
/assign @unmarshall
/cc @timuthy

Release note:

Updation of `status.observedGeneration` and optional removal of the `gardener.cloud/operation: reconcile` annotation on the Etcd resource are now executed after the reconciliation of the Etcd status, to depict accurate state of the cluster at any given point in time. Users must wait for the `status.observedGeneration` field to be updated (and additionally for the removal of the `gardener.cloud/operation: reconcile` annotation is CLI flag `enable-spec-auto-reconcile` is set to false) to confirm completion of reconciliation.

[seshachalam-yv]

/assign

[shreyas-s-rao]

@seshachalam-yv thanks a lot for your inputs. As discussed, I have made the following changes to streamline the reconciliation flow. Changes made in 97282ed:

1. Call canReconcileSpec only once, to ensure that only one event is emitted when suspend-reconcile key is detected. Additionally, in the case where a new Etcd resource is created with operation-annotation set, computing canReconcileSpec only at the beginning ensures that it returns true, as opposed to recalculating it after updating observedGeneration, in which case it returns false and the operation annotation would never be removed.
2. Use operatorContext to pass information such as result of canReconcileSpec and shortCircuitSpecReconcile (both bool values), to avoid recalculating these within each of the other reconciliation steps like reconcileStatus and removeOperationAnnotation
3. Add helper functions with tests, along with some missing tests for other existing helper functions.

[unmarshall]

/lgtm