kind e2e tests fail randomly

[jan--f]

Hmm I've seen this error in multiple PRs now, seems like we have a race somewhere maybe?

2022-07-20T08:00:20Z	ERROR	controller.monitoringstack	Reconciler error	{"reconciler group": "monitoring.rhobs", "reconciler kind": "MonitoringStack", "name": "tq-ms-combo", "namespace": "e2e-tests", "error": "roles.rbac.authorization.k8s.io \"tq-ms-combo-prometheus\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227

Its usually resolved with a re-run.

Originally posted by @jan--f in #178 (comment)

[sthaha]

@JoaoBraveCoding , @tremes are you looking in this? Else I would like to take a crack at it.

[tremes]

@sthaha I was unsuccessfully trying to reproduce it...feel free to investigate. Thanks

[sthaha]

One of the failures seem to be related to: kubernetes-sigs/controller-runtime#1881

2022-10-11T23:37:26Z	INFO	thanos-querier	watched MonitoringStack changed, checking for matching querier	{"Monitoring Stack": "e2e-tests/valid-loglevel"}
2022-10-11T23:37:26Z	INFO	observability-operator	Failed to update status	{"stack": "e2e-tests/valid-loglevel", "err": "Operation cannot be fulfilled on monitoringstacks.monitoring.rhobs \"valid-loglevel\": StorageError: invalid object, Code: 4, Key: /registry/monitoring.rhobs/monitoringstacks/e2e-tests/valid-loglevel, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 833ce6a7-1b72-4b17-9a3d-86a9a8353fae, UID in object meta: "}
2022-10-11T23:37:26Z	INFO	observability-operator	Reconciling monitoring stack	{"stack": "e2e-tests/valid-loglevel"}

[simonpasquier]

Probably not relevant anymore.