This commit is intended to address two issues that we diagnosed while
investigating crossplane-contrib/provider-aws#802.

The first issue is that controller-runtime does not guarantee reads from cache
will return the freshest version of a resource. It's possible we could create an
external resource in one reconcile, then shortly after trigger another in which
it appears that the managed resource was never created because we didn't record
its external-name. This only affects the subset of managed resources with
non-deterministic external-names that are assigned during creation.

The second issue is that some external APIs are eventually consistent. A newly
created external resource may take some time before our ExternalClient's observe
call can confirm it exists. AWS EC2 is an example of one such API.

This commit attempts to address the first issue by making an Update to a managed
resource immediately before Create it called. This Update call will be rejected
by the API server if the managed resource we read from cache was not the latest
version.

It attempts to address the second issue by allowing managed resource controller
authors to configure an optional grace period that begins when an external
resource is successfully created. During this grace period we'll requeue and
keep waiting if Observe determines that the external resource doesn't exist,
rather than (re)creating it.

Signed-off-by: Nic Cope <negz@rk0n.org>

The retry logic we use to persist critical annotations makes it difficult to
delete an annotation without potentially also deleting annotations added by
another controller (e.g. the composition logic). This commit therefore changes
the way we detect whether we might have created an external resource but not
recorded the result. Previously we relied on the presence of the 'pending'
annotation to detect this state. Now we check whether the 'pending' annotation
is newer than any 'succeeded' or 'failed' annotation.

Signed-off-by: Nic Cope <negz@rk0n.org>