Package fails re-reconcile on upgrade

[rjferguson21]

Environment

App version: 0.25.0
Kubernetes distro being used: k3d
Other: uds-k3d-demo

We've seen intermittent issues where on upgrades Packages fail to reconcile because Keycloak is temporarily unresponsive.

Steps to reproduce

1. Install uds-k3d-demo
2. Delete the keycloak pod
3. Immediately delete the pepr-watcher pod

Expected result

The operator would re-reconcile all Packages, and they would end up in a status=Ready state.

Actual Result

The operator tries to reconcile the Packages, triggering an HTTP request to Keycloak as part of the reconciliation process for packages that leverage SSO. This request fails and the operator immediately retries 5 times in short succession and subsequently sets the Package to status=Failed.

Visual Proof (screenshots, videos, text, etc)

Severity/Priority

Critical

Additional Context

We debated a couple of solutions:

• Adding a retry mechanism in the operator, specially for the Keycloak requests.
• Adding a check to ensure Keycloak is healthy prior to making requests.
• Adding a global operator delay on retries.