Nightly "production like" testing

[mjnagel]

Is your feature request related to a problem? Please describe.

Currently uds-core has CI smoke testing on k3d, with a "dev" setup (single replicas, single node, minimal external dependencies). When I deploy to my RKE2 or EKS cluster I don't have a good example of an HA setup and low confidence that has been tested before I deploy in my environment.

Describe the solution you'd like

Nightly CI tests against k8s distros, suggest targeting rke2 and eks, with a "production" bundle configuration used for CI testing only (not published).

Beyond just the basic deploy it would also be good to add more full e2e testing as part of this nightly deploy.

Describe alternatives you've considered

End users should have staging environments to test the deploy before they move to prod so this in theory isn't a high risk, but I think it's very valuable for us to test before cutting releases rather than relying on end users to test for us.

We could also run the testing on every PR, but that is likely not desirable since it slows down the fast feedback on PRs.

Additional context

We know of end users interested in RKE2 and EKS so that is part of the rationale for targeting those distros.

We also don't want to reproduce any testing that should be done upstream, so we likely should stick to cross-application functionality (i.e. hitting istio virtual services, SSO logic, etc).

[mjnagel]

Scoping the remaining work here:

• Add an RKE2 nightly test (in AWS?)
• Update nightly testing bundle to have HA configurations for all applications. A few noted gaps:
  • HA for Keycloak - multi replica + postgres (RDS)
  • Multi-replica Loki
• Run validation scripts against deployed clusters: This may require some extra effort to ensure that endpoints are accessible

It may make sense to split out this work into multiple issues as each of these sub-bullets are probably issue-sized in themselves.

[mjnagel]

Intending to break this out into more bite sized tickets. Likely each of the above bullet points will be its own ticket.

[mjnagel]

Closing as the linked tickets provide more consumable chunks here.