monitoring: support configuration of alerting notifications by application owners #937

simonpasquier · 2021-10-19T08:36:31Z

This document describes a solution that allows OpenShift users to route alert
notifications without cluster admin intervention. It complements the existing
user workload monitoring stack, enabling a full self-service experience for
workload monitoring.

philipgough · 2021-10-19T09:40:13Z

enhancements/monitoring/alert-routing-for-user-workload-monitoring.md

+As an OpenShift cluster admin, I want to exclude certain user namespaces from
+modifying the Plaform Alertmanager configuration so that I can recover in case
+of breakage or bad behavior.
+


@simonpasquier I don't really understand this story. How would excluding specific namespaces allow us to recover from a bad configuration?

say that there's a AlertmanagerConfig resource in namespace foo breaking the Platform Alertmanager. Then the cluster admin can temporarily exclude that particular namespace from UWM to recover the situation and get in touch with the team that provisioned the "bad" config.

Got it, that makes sense. So it is a reactive solution essentially. Drifting off slightly to implementation, would they have any means other than logging to be alerted on what namespace has broken the conf?

I see this is covered later with the AlertmanagerBadConfig alert, somewhat at least. But maybe we can do some improvements to help identify the issue.

Hmmm I see this, but I wonder if we can have a more ambitious goal:

As an OpenShift cluster main I expect Alertmanager to NOT crash/stop refreshing configuration if only one of the AlertmanagerConfig is malformed / invalid.

WDYT?

is that even possible? (:

I think that is the goal from the prometheus-operator perspective yes. We will discard the invalid CR and it will be excluded from the merge.

The operator can only do as much as it can to ensure that it generates a valid Alertmanager configuration but there will always be a theoretical risk that the operator has a bug in certain edge cases. Still I'll try to think about more proactive strategies.

enhancements/monitoring/alert-routing-for-user-workload-monitoring.md

philipgough · 2021-10-19T09:44:49Z

enhancements/monitoring/alert-routing-for-user-workload-monitoring.md

+As a UWM admin, I want to send user alerts to the Platform Alertmanager
+cluster because UWM alerts are managed by an external system (off-cluster Alertmanager for
+instance).
+


If managed by an external system, why would we want to send to platform AM?

I think this is clarified down further, but they would be able to (if desired) send to both right? If using some external system they could forward the alerts to both platform and external AM?

yes it's already possible today (if you configure additionalAlertmanagerConfigs for UWM, alerts will always go to Platform Alertmanager). I'll add another user story to make it clear.

philipgough · 2021-10-19T10:26:01Z

enhancements/monitoring/alert-routing-for-user-workload-monitoring.md

+Alertmanager configuration.
+
+Mitigation
+* The Prometheus operator should expose a validating admission webhook that should prevent invalid configurations.


This would require that the webhook and the operator share the same RBAC to read cluster wide secrets, correct?

the operator implements the webhook endpoint so it should have the proper RBAC but maybe it's better if the webhook doesn't try to resolve external references?
More a discussion for prometheus-operator/prometheus-operator#4338 though ;)

bwplotka

Amazing! Some suggestions, otherwise looking good 👍🏽

bwplotka · 2021-10-21T08:51:23Z

enhancements/monitoring/alert-routing-for-user-workload-monitoring.md

@@ -0,0 +1,411 @@
+---
+title: alert-routing


This is great. I would propose naming this PR and title like Multi-Tenant Alert Routing on OpenShift 🔥

Suggested change

title: alert-routing

title: multi-tenant-alert-routing

bwplotka · 2021-10-21T09:49:26Z

enhancements/monitoring/alert-routing-for-user-workload-monitoring.md

+This document describes a solution that allows OpenShift users to route alert
+notifications without cluster admin intervention. It complements the existing
+[user-workload monitoring stack][uwm-docs], enabling a full self-service experience for
+workload monitoring.
+


Suggested change

This document describes a solution that allows OpenShift users to route alert

notifications without cluster admin intervention. It complements the existing

[user-workload monitoring stack][uwm-docs], enabling a full self-service experience for

workload monitoring.

This document describes a solution that allows OpenShift users to route alert

notifications from shared Alertmanager without cluster admin intervention. It complements the existing

[user-workload monitoring stack][uwm-docs], enabling a full self-service experience for

workload monitoring.

bwplotka · 2021-10-21T09:51:43Z

enhancements/monitoring/alert-routing-for-user-workload-monitoring.md

+cluster admins.
+* Cluster admins can grant permissions to users and groups to manage alert
+routing scoped to individual namespaces.
+* Namespace owners should be able to opt-out from Alertmanager


👍🏽 That was my worry - what if someone created their own AM and what to watch for couple of namespaces on their own. That will do the work

yes we already have a solution in place for service/pod monitors and rules. For consistency it should also apply to AlertmanagerConfigs.

enhancements/monitoring/alert-routing-for-user-workload-monitoring.md

bwplotka · 2021-10-21T09:57:18Z

enhancements/monitoring/alert-routing-for-user-workload-monitoring.md

+
+#### Story 3
+
+As an application owner, I want to know if my AlertmanagerConfig custom


👍🏽 Is this case for DeadMansSnitch?

Not specifically. First of all, I want to know whether or not the Prometheus operator reconciled my AlertmanagerConfig resource (e.g. it won't if it contains invalid secret references for instance): this would be covered by exposing a Status subresource in the CRD.
Secondly, it would be nice if I can have an easy way to trigger a "fake" alert (drill mode) to validate my receivers.

bwplotka · 2021-10-21T09:59:27Z

enhancements/monitoring/alert-routing-for-user-workload-monitoring.md

+As an OpenShift cluster admin, I want to exclude certain user namespaces from
+modifying the Plaform Alertmanager configuration so that I can recover in case
+of breakage or bad behavior.
+


Hmmm I see this, but I wonder if we can have a more ambitious goal:

As an OpenShift cluster main I expect Alertmanager to NOT crash/stop refreshing configuration if only one of the AlertmanagerConfig is malformed / invalid.

WDYT?

bwplotka · 2021-10-21T09:59:33Z

enhancements/monitoring/alert-routing-for-user-workload-monitoring.md

+As an OpenShift cluster admin, I want to exclude certain user namespaces from
+modifying the Plaform Alertmanager configuration so that I can recover in case
+of breakage or bad behavior.
+


is that even possible? (:

openshift-ci · 2021-10-21T10:00:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bwplotka

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~enhancements/monitoring/OWNERS~~ [bwplotka]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jan--f

Yeah this all makes sense to me.

Would it make sense to map out the additional changes we'd need, if we opt for the optional UWM alertmanager cluster? That might inform the decision in the first place. E.g. would we still retain the option to exclude some namespaces, what would the resource overhead look like.

jan--f · 2021-10-21T13:50:40Z

enhancements/monitoring/alert-routing-for-user-workload-monitoring.md

+routing scoped to individual namespaces.
+* Namespace owners should be able to opt-out from Alertmanager
+configuration (similar to what exist for service/pod monitors and rules using the
+`"openshisft.io/user-monitoring: false"` label on the namespace).


Suggested change

`"openshisft.io/user-monitoring: false"` label on the namespace).

`"openshift.io/user-monitoring: false"` label on the namespace).

If it is more than theoretically possible for users to destabilize the Platform AM, then we have to at least consider this second AM being required. How likely are we to hit these cases where the prometheus operator does not correctly handle a user input leading to a Platform AM outage?

enhancements/monitoring/multi-tenant-alert-routing.md

jeremyeder · 2021-10-27T12:59:15Z

@openshift/sre-alert-sme please review (I also sent this to a wide distribution of SRE mailing lists)

dofinn

This is a great feature design @simonpasquier ! I have concerns over meshing resources usage between admins/SREs + users as there are no guard rails from preventing users from pushing AM into an unavailable state. This concerns me at scale where if it became an issue, SRE would be forced to disable this feature leaving the customers back in their original position their application alerting requirements. Big + on UWM having its own AM.

dofinn · 2021-10-28T02:37:37Z

enhancements/monitoring/multi-tenant-alerting.md

+
+This role complements the existing `monitoring-edit`, `monitoring-rules-edit` and `monitoring-rules-view` roles.
+
+#### Resource impacts


How will the alert tree be structured? In the case of sharing resources with customers, are SREs able to prioritise platform alerts over user?

For example:

system/platform alerts

user alerts (1st reconciled/added)

......

user alerts (last reconciled/added)
?

In practice, the routing tree will be structured as:

user AlertmanagerConfig from namespace X

user AlertmanagerConfig from namespace Y

...

user AlertmanagerConfig from namespace Z

Alertmanager configuration defined by the cluster admins

But there's no priority per se in Alertmanager: as soon as an alert matches a routing entry, it gets dispatched to a goroutine and the tree evaluation goes on to check if other entries match.

dofinn · 2021-10-28T02:44:32Z

enhancements/monitoring/multi-tenant-alerting.md

+to a potential outage of the Platform Alertmanager cluster.
+
+Mitigations
+* The `AlertmanagerBadConfig` alert fires when Alertmanager can't reload its configuration.


I presume this alert will be destined for Admin/SRE routes? If so, this is unbound overhead that has potential to cause alot of pain for admins/SREs. Either configs should be somehow prevalidated or automatically rolled-back to prevent this.

Yes since the Platform Alertmanager is managed by the SRE teams, the notification would be routed to them.

As SRE, we should not care about alerts that caused by customer's wrong configs.

cc @mrbarge another candidate for ocm-agent :)

If a dedicated Alertmanager cluster is deployed for user-defined alerts, the platform SREs can decide where the AlertmanagerBadConfig alert for namespace="openshift-user-workload-monitoring" should be routed. I would guess that something similar exists today for alerts about the UWM components (Prometheus + Thanos Ruler).

dofinn · 2021-10-28T02:45:55Z

enhancements/monitoring/multi-tenant-alerting.md

+Mitigations
+* The `AlertmanagerBadConfig` alert fires when Alertmanager can't reload its configuration.
+* Cluster admins can turn off the support for `AlertmanagerConfig` globally so that the Platform Alertmanager cluster can process platform alerts again and the cluster admins have time to identiy the "rogue" `AlertmanagerConfig` resource(s).
+* Cluster admins can exclude specific user namespaces (once the "rogue" `AlertmanagerConfig` resource(s) have been identified) to restore UWM functionality for good citizens.


How will the rouge AlertmanagerConfigbe identified? Will it get a.status` of invalid?

As of today, nothing exists and the operator can't report which one is invalid since it didn't catch it before. The identification has to rely on human intervention.

Eventually the operator will update the status field for AlertmanagerConfig resources that the operator doesn't reconcile (see #### AlertmanagerConfig resources not being reconciled) below.

@simonpasquier will this render the co/monitoring degraded? (it would fail preupgrade health checks if so...)

As of today, CMO doesn't check for alerts so Alertmanager (or Prometheus) failing to reload the configuration doesn't change the co/monitoring conditions.

dofinn · 2021-10-28T02:47:55Z

enhancements/monitoring/multi-tenant-alerting.md

+Users may provide bad credentials for the receivers, the system receiving the
+notifications might be unreachable or the system might be unable to process the requests. These
+situations would trigger the `AlertmanagerFailedToSendAlerts` and/or
+`AlertmanagerClusterFailedToSendAlerts` alerts. The cluster admins have to act


Again, this is unbound support/toil for Admin/SREs. If we open the openshift-monitoring AM up to customers/users then there needs to be harder guard rails against the customers/user breaking the platform AM. Admins/SREs should not be involved in troubleshooting user configuration. This wont scale.

dofinn · 2021-10-28T02:51:34Z

enhancements/monitoring/multi-tenant-alerting.md

+
+### Risks and Mitigations
+
+#### Disruption of the platform Alertmanager


This scares me, giving customers this power to configure unbound alerting thresholds via their configs + applications gives high risk to AM unavailability. Great potential for unintended user generated DoS.

enhancements/monitoring/multi-tenant-alerting.md

simonpasquier

I have concerns over meshing resources usage between admins/SREs + users as there are no guard rails from preventing users from pushing AM into an unavailable state. This concerns me at scale where if it became an issue, SRE would be forced to disable this feature leaving the customers back in their original position their application alerting requirements. Big + on UWM having its own AM.

Thanks for the great feedback @dofinn! I kind of expected these concerns. For organizations where a single team manages everything, it might be less problematic but I agree that they are very valid in the OSD context where the platform and UWM responsibilities are decoupled.
I'll update the proposal to describe the possibility of a standalone UWM Alertmanager.

simonpasquier · 2021-10-28T13:36:40Z

enhancements/monitoring/multi-tenant-alerting.md

+to a potential outage of the Platform Alertmanager cluster.
+
+Mitigations
+* The `AlertmanagerBadConfig` alert fires when Alertmanager can't reload its configuration.


Yes since the Platform Alertmanager is managed by the SRE teams, the notification would be routed to them.

simonpasquier · 2021-10-28T13:38:16Z

enhancements/monitoring/multi-tenant-alerting.md

+Mitigations
+* The `AlertmanagerBadConfig` alert fires when Alertmanager can't reload its configuration.
+* Cluster admins can turn off the support for `AlertmanagerConfig` globally so that the Platform Alertmanager cluster can process platform alerts again and the cluster admins have time to identiy the "rogue" `AlertmanagerConfig` resource(s).
+* Cluster admins can exclude specific user namespaces (once the "rogue" `AlertmanagerConfig` resource(s) have been identified) to restore UWM functionality for good citizens.


As of today, nothing exists and the operator can't report which one is invalid since it didn't catch it before. The identification has to rely on human intervention.

simonpasquier · 2021-10-28T13:38:29Z

enhancements/monitoring/multi-tenant-alerting.md

+Users may provide bad credentials for the receivers, the system receiving the
+notifications might be unreachable or the system might be unable to process the requests. These
+situations would trigger the `AlertmanagerFailedToSendAlerts` and/or
+`AlertmanagerClusterFailedToSendAlerts` alerts. The cluster admins have to act


simonpasquier · 2021-10-28T13:42:41Z

enhancements/monitoring/multi-tenant-alerting.md

+
+This role complements the existing `monitoring-edit`, `monitoring-rules-edit` and `monitoring-rules-view` roles.
+
+#### Resource impacts


In practice, the routing tree will be structured as:

user AlertmanagerConfig from namespace X

user AlertmanagerConfig from namespace Y

...

user AlertmanagerConfig from namespace Z

Alertmanager configuration defined by the cluster admins

But there's no priority per se in Alertmanager: as soon as an alert matches a routing entry, it gets dispatched to a goroutine and the tree evaluation goes on to check if other entries match.

enhancements/monitoring/multi-tenant-alerting.md

david-martin · 2021-10-29T10:44:18Z

enhancements/monitoring/multi-tenant-alerting.md

+
+Even though the Prometheus operator prevents it as much as it can, it may be
+possible for users to create an `AlertmanagerConfig` resource that triggers the
+Prometheus operator to generate an invalid Alertmanager configuration, leading


Would it make more sense for the prometheus operator to reject any AlertmanagerConfig resources that would cause an overall bad config?
Webhook validation?

The operator will do its best to reject invalid configuration but bugs happen and it may happen that the operator generates a configuration that couldn't be parsed by Alertmanager. In short, unlikely to happen but still possible.

enhancements/monitoring/multi-tenant-alerting.md

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

Distinguish between Platform and UWM Alertmanager deployments.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

enhancements/monitoring/multi-tenant-alerting.md

simonpasquier · 2022-03-10T09:02:46Z

/remove-lifecycle stale

The proposal is in good shape now, I think all comments have addressed.
@jeremyeder @dofinn can you give it a final review pls?

simonpasquier · 2022-03-16T12:51:31Z

/remove-lifecycle rotten
/assign @jeremyeder

dofinn · 2022-03-22T03:55:56Z

enhancements/monitoring/multi-tenant-alerting.md

+
+#### Deployment models
+
+##### Leveraging the Platform Alertmanager


Inhibition is a global setting... is it possible that multi tenant AlertManager CRs could unknowingly inhibit each other?

yes because the operator will always append a namespace matcher to the inhibition rules, e.g.

inhibit_rules: - target_matchers: - alertname="HttpErrors" source_matchers: - alertname=~"DatabaseErrors" equal: - service

is turned into

inhibit_rules: - target_matchers: - alertname="HttpErrors" - namespace="foo" source_matchers: - alertname=~"DatabaseErrors" - namespace="foo" equal: - service

dofinn · 2022-03-22T04:38:32Z

enhancements/monitoring/multi-tenant-alerting.md

+      enabled: true
+      enableUserAlertmanagerConfig: true
+      logLevel: info
+      nodeSelector: {...}


@simonpasquier will this be workers by default?

Yes, it should be similar to what happens for the other monitoring components. When nodeSelector and tolerations aren't defined, all worker nodes are eligible and master nodes are excluded (because they are tainted).

Excellent, the current prom-operator from UWM schedules itself on masters :)

$ oc -n openshift-user-workload-monitoring get pods prometheus-operator-68fd48b89d-6tp7s -o json | jq '.spec.tolerations[]' { "effect": "NoSchedule", "key": "node-role.kubernetes.io/master", "operator": "Exists" } { "effect": "NoExecute", "key": "node.kubernetes.io/not-ready", "operator": "Exists", "tolerationSeconds": 300 } { "effect": "NoExecute", "key": "node.kubernetes.io/unreachable", "operator": "Exists", "tolerationSeconds": 300 } { "effect": "NoSchedule", "key": "node.kubernetes.io/memory-pressure", "operator": "Exists" }

yes this is following the OpenShift convention for operators -> https://github.com/openshift/enhancements/blob/master/CONVENTIONS.md#taints-and-tolerations

dofinn · 2022-04-04T23:03:30Z

/remove-lifecycle stale

The proposal is in good shape now, I think all comments have addressed. @jeremyeder @dofinn can you give it a final review pls?

Done ! Awesome work on this @simonpasquier ! /lgtm

dofinn · 2022-04-05T19:50:51Z

/lgtm

openshift-ci · 2022-04-05T19:58:55Z

@simonpasquier: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 19, 2021

openshift-ci bot requested review from markmc and sttts October 19, 2021 08:36

simonpasquier force-pushed the alertmanagerconfig-support branch 2 times, most recently from ddab0db to 597cd13 Compare October 19, 2021 08:52

simonpasquier changed the title ~~[WIP] monitoring: support configuration of alerting settings by application owners~~ [WIP] monitoring: support configuration of alerting notifications by application owners Oct 19, 2021

philipgough reviewed Oct 19, 2021

View reviewed changes

enhancements/monitoring/alert-routing-for-user-workload-monitoring.md Outdated Show resolved Hide resolved

philipgough reviewed Oct 19, 2021

View reviewed changes

simonpasquier force-pushed the alertmanagerconfig-support branch from 597cd13 to 62b4146 Compare October 19, 2021 09:57

philipgough reviewed Oct 19, 2021

View reviewed changes

bwplotka approved these changes Oct 21, 2021

View reviewed changes

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 21, 2021

jan--f reviewed Oct 21, 2021

View reviewed changes

simonpasquier changed the title ~~[WIP] monitoring: support configuration of alerting notifications by application owners~~ monitoring: support configuration of alerting notifications by application owners Oct 21, 2021

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 21, 2021

wanghaoran1988 reviewed Oct 25, 2021

View reviewed changes

enhancements/monitoring/multi-tenant-alert-routing.md Outdated Show resolved Hide resolved

dofinn reviewed Oct 28, 2021

View reviewed changes

simonpasquier commented Oct 28, 2021

View reviewed changes

wanghaoran1988 reviewed Oct 29, 2021

View reviewed changes

enhancements/monitoring/multi-tenant-alerting.md Show resolved Hide resolved

david-martin reviewed Oct 29, 2021

View reviewed changes

enhancements/monitoring/multi-tenant-alerting.md Show resolved Hide resolved

david-martin reviewed Oct 29, 2021

View reviewed changes

philipgough mentioned this pull request Nov 4, 2021

Enable AlertmanagerConfig CR processing with the cluster-monitoring-config ConfigMap openshift/cluster-monitoring-operator#1375

Closed

simonpasquier force-pushed the alertmanagerconfig-support branch from 5012be5 to 2941e40 Compare November 26, 2021 13:51

philipgough reviewed Nov 26, 2021

View reviewed changes

enhancements/monitoring/multi-tenant-alerting.md Outdated Show resolved Hide resolved

philipgough reviewed Nov 26, 2021

View reviewed changes

enhancements/monitoring/multi-tenant-alerting.md Outdated Show resolved Hide resolved

simonpasquier force-pushed the alertmanagerconfig-support branch from ac9a842 to ba3dc21 Compare November 26, 2021 16:15

sttts reviewed Nov 29, 2021

View reviewed changes

enhancements/monitoring/multi-tenant-alerting.md Outdated Show resolved Hide resolved

simonpasquier added 14 commits March 7, 2022 15:50

Rename proposal

b1b7a1a

Address Jan's comments

f774cb6

Rename file

65adefc

Add section about external labels for platform alerts

d5ade6e

Draft a first version of UWM Alertmanager deployment

0fb5b73

Refine UWM Alertmanager design

74fcfd3

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

Add new required sections

abeadb9

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

Add details about validating webhook deployment and justifications

6913f54

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

Fix configuration for sourcing the platform alerts

f848992

Rewrite mitigations section

3021766

Distinguish between Platform and UWM Alertmanager deployments.

Update reviewers and approvers

b046609

Address @dofinn remarks

f370ceb

fix RBAC role name

1551d1a

Add details about OCP console integration

78d6ddb

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

simonpasquier force-pushed the alertmanagerconfig-support branch from 2194aef to 78d6ddb Compare March 7, 2022 17:04

simonpasquier commented Mar 7, 2022

View reviewed changes

enhancements/monitoring/multi-tenant-alerting.md Show resolved Hide resolved

Clear up 'Open questions' section

f556b83

openshift-ci bot assigned jeremyeder Mar 16, 2022

openshift-ci bot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Mar 16, 2022

dofinn reviewed Mar 22, 2022

View reviewed changes

openshift-ci bot assigned dofinn Apr 5, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Apr 5, 2022

openshift-merge-robot merged commit c296f93 into openshift:master Apr 5, 2022

simonpasquier deleted the alertmanagerconfig-support branch May 5, 2022 12:21


		#### Story 3

		As an application owner, I want to know if my AlertmanagerConfig custom

	`"openshisft.io/user-monitoring: false"` label on the namespace).
	`"openshift.io/user-monitoring: false"` label on the namespace).


		This role complements the existing `monitoring-edit`, `monitoring-rules-edit` and `monitoring-rules-view` roles.

		#### Resource impacts


		### Risks and Mitigations

		#### Disruption of the platform Alertmanager


		#### Deployment models

		##### Leveraging the Platform Alertmanager

monitoring: support configuration of alerting notifications by application owners #937

monitoring: support configuration of alerting notifications by application owners #937

Conversation

simonpasquier commented Oct 19, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

philipgough Oct 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simonpasquier Oct 19, 2021 • edited Loading

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci bot commented Oct 21, 2021

jan--f left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeremyeder commented Oct 27, 2021

dofinn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simonpasquier Oct 28, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dofinn Mar 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simonpasquier left a comment

Choose a reason for hiding this comment

simonpasquier Oct 28, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simonpasquier commented Mar 10, 2022

simonpasquier commented Mar 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dofinn commented Apr 4, 2022

dofinn commented Apr 5, 2022

openshift-ci bot commented Apr 5, 2022

philipgough Oct 19, 2021 •

edited

Loading

simonpasquier Oct 19, 2021 •

edited

Loading

simonpasquier Oct 28, 2021 •

edited

Loading

dofinn Mar 22, 2022 •

edited

Loading

simonpasquier Oct 28, 2021 •

edited

Loading