Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

STF won't install without cluster-monitoring-operator installed (workaround within) #306

Closed
csibbitt opened this issue Jan 25, 2022 · 1 comment · Fixed by #465
Closed

Comments

@csibbitt
Copy link
Collaborator

Not usually a problem since it's installed by default, but not on CRC.

TASK [Bind the local prometheus SA to prometheus cluster role] ******************************** 
fatal: [localhost]: FAILED! => {"changed": false, "error": 404, "msg": "Failed to create object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"clusterroles.rbac.authorization.k8s.io \\\\\"prometheus-k8s\\\\\" not found\",\"reason\":\"NotFound\",\"details\":{\"name\":\"prometheus-k8s\",\"group\":\"rbac.authorization.k8s.io\",\"kind\":\"clusterroles\"},\"code\":404}\\n'", "reason": "Not Found", "status": 404}

needs https://paste.opendev.org/show/812354/

 TASK [Bind role] ******************************** 
fatal: [localhost]: FAILED! => {"changed": false, "error": 404, "msg": "Failed to create object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"clusterroles.rbac.authorization.k8s.io \\\\\"alertmanager-main\\\\\" not found\",\"reason\":\"NotFound\",\"details\":{\"name\":\"alertmanager-main\",\"group\":\"rbac.authorization.k8s.io\",\"kind\":\"clusterroles\"},\"code\":404}\\n'", "reason": "Not Found", "status": 404}

needs https://paste.opendev.org/show/812355/

@leifmadsen
Copy link
Member

Noting because I don't trust the pastebins:

812354

apiVersion: rbac.authorization.k8s.io/v1                                                                                                                                        [17/1814]
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get
- apiGroups:
  - authentication.k8s.io
  resources:
  - tokenreviews
  verbs:
  - create
- apiGroups:
  - authorization.k8s.io
  resources:
  - subjectaccessreviews
  verbs:
  - create
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
- apiGroups:
  - security.openshift.io
  resourceNames:
  - nonroot
  resources:
  - securitycontextconstraints
  verbs:
  - use

812355

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: alertmanager-main
rules:
- apiGroups:
  - authentication.k8s.io
  resources:
  - tokenreviews
  verbs:
  - create
- apiGroups:
  - authorization.k8s.io
  resources:
  - subjectaccessreviews
  verbs:
  - create
- apiGroups:
  - security.openshift.io
  resourceNames:
  - nonroot
  resources:
  - securitycontextconstraints
  verbs:
  - use

elfiesmelfie added a commit that referenced this issue Aug 25, 2023
The cluster-monitoring-operator is required for STF to install. It creates the required alertmanager-main and prometheus-k8s ClusterRoles. These are not present when usin CRC, so a workaround is needed.

TODO: Add in check for the ClusterRoles instead of creating them unconditionally
This is a workaround for not having cluster-monitoring-operator
installed: #306
elfiesmelfie added a commit that referenced this issue Aug 28, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s
ClusterRoles, ans STF relies on these being present.
These are not present when using CRC, so they need to be created
explicitly.

TODO: Add in check for the ClusterRoles instead of creating them unconditionally
This is a workaround for not having cluster-monitoring-operator
installed: #306
elfiesmelfie added a commit that referenced this issue Aug 31, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s
ClusterRoles, ans STF relies on these being present.
These are not present when using CRC, so they need to be created
explicitly.

TODO: Add in check for the ClusterRoles instead of creating them unconditionally
This is a workaround for not having cluster-monitoring-operator
installed: #306
elfiesmelfie added a commit that referenced this issue Sep 1, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s
ClusterRoles, ans STF relies on these being present.
These are not present when using CRC, so they need to be created
explicitly.

This is a workaround for not having cluster-monitoring-operator
installed: #306
elfiesmelfie added a commit that referenced this issue Sep 1, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s
ClusterRoles, ans STF relies on these being present.
These are not present when using CRC, so they need to be created
explicitly.

This is a workaround for not having cluster-monitoring-operator
installed: #306
elfiesmelfie added a commit that referenced this issue Sep 1, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s
ClusterRoles, ans STF relies on these being present.
These are not present when using CRC, so they need to be created
explicitly.

This is a workaround for not having cluster-monitoring-operator
installed: #306
elfiesmelfie added a commit that referenced this issue Sep 1, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s
ClusterRoles, ans STF relies on these being present.
These are not present when using CRC, so they need to be created
explicitly.

This is a workaround for not having cluster-monitoring-operator
installed: #306
elfiesmelfie added a commit that referenced this issue Sep 4, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s
ClusterRoles, ans STF relies on these being present.
These are not present when using CRC, so they need to be created
explicitly.

This is a workaround for not having cluster-monitoring-operator
installed: #306
elfiesmelfie added a commit that referenced this issue Sep 4, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s
ClusterRoles, ans STF relies on these being present.
These are not present when using CRC, so they need to be created
explicitly.

This is a workaround for not having cluster-monitoring-operator
installed: #306
elfiesmelfie added a commit that referenced this issue Sep 4, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s
ClusterRoles, ans STF relies on these being present.
These are not present when using CRC, so they need to be created
explicitly.

This is a workaround for not having cluster-monitoring-operator
installed: #306
elfiesmelfie added a commit that referenced this issue Sep 5, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s
ClusterRoles, ans STF relies on these being present.
These are not present when using CRC, so they need to be created
explicitly.

This is a workaround for not having cluster-monitoring-operator
installed: #306
elfiesmelfie added a commit that referenced this issue Sep 5, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s
ClusterRoles, ans STF relies on these being present.
These are not present when using CRC, so they need to be created
explicitly.

This is a workaround for not having cluster-monitoring-operator
installed: #306
elfiesmelfie added a commit that referenced this issue Sep 5, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s
ClusterRoles, ans STF relies on these being present.
These are not present when using CRC, so they need to be created
explicitly.

This is a workaround for not having cluster-monitoring-operator
installed: #306
elfiesmelfie added a commit that referenced this issue Sep 6, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s
ClusterRoles, ans STF relies on these being present.
These are not present when using CRC, so they need to be created
explicitly.

This is a workaround for not having cluster-monitoring-operator
installed: #306
elfiesmelfie added a commit that referenced this issue Sep 6, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s
ClusterRoles, ans STF relies on these being present.
These are not present when using CRC, so they need to be created
explicitly.

This is a workaround for not having cluster-monitoring-operator
installed: #306
elfiesmelfie added a commit that referenced this issue Sep 7, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s
ClusterRoles, ans STF relies on these being present.
These are not present when using CRC, so they need to be created
explicitly.

This is a workaround for not having cluster-monitoring-operator
installed: #306
elfiesmelfie added a commit that referenced this issue Sep 8, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s
ClusterRoles, ans STF relies on these being present.
These are not present when using CRC, so they need to be created
explicitly.

This is a workaround for not having cluster-monitoring-operator
installed: #306
elfiesmelfie added a commit that referenced this issue Sep 11, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s.
ClusterRoles, and STF relies on these being present.
These are not present when using CRC, so ClusterRoles need to be
explicitly created.

The names of the ClusterRoles have been updated, in case there is some
conflict when cluster-monitoring-operator is installed after STF.

This is a workaround for not having cluster-monitoring-operator
installed: #306

resolves #306
elfiesmelfie added a commit that referenced this issue Sep 12, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s.
ClusterRoles, and STF relies on these being present.
These are not present when using CRC, so ClusterRoles need to be
explicitly created.

The names of the ClusterRoles have been updated, in case there is some
conflict when cluster-monitoring-operator is installed after STF.

This is a workaround for not having cluster-monitoring-operator
installed: #306

resolves #306
elfiesmelfie added a commit that referenced this issue Sep 14, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s.
ClusterRoles, and STF relies on these being present.
These are not present when using CRC, so ClusterRoles need to be
explicitly created.

The names of the ClusterRoles have been updated, in case there is some
conflict when cluster-monitoring-operator is installed after STF.

This is a workaround for not having cluster-monitoring-operator
installed: #306

resolves #306
leifmadsen pushed a commit that referenced this issue Sep 19, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s.
ClusterRoles, and STF relies on these being present.
These are not present when using CRC, so ClusterRoles need to be
explicitly created.

The names of the ClusterRoles have been updated, in case there is some
conflict when cluster-monitoring-operator is installed after STF.

This is a workaround for not having cluster-monitoring-operator
installed: #306

resolves #306
leifmadsen pushed a commit that referenced this issue Sep 19, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s.
ClusterRoles, and STF relies on these being present.
These are not present when using CRC, so ClusterRoles need to be
explicitly created.

The names of the ClusterRoles have been updated, in case there is some
conflict when cluster-monitoring-operator is installed after STF.

This is a workaround for not having cluster-monitoring-operator
installed: #306

resolves #306
elfiesmelfie added a commit that referenced this issue Sep 20, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s.
ClusterRoles, and STF relies on these being present.
These are not present when using CRC, so ClusterRoles need to be
explicitly created.

The names of the ClusterRoles have been updated, in case there is some
conflict when cluster-monitoring-operator is installed after STF.

This is a workaround for not having cluster-monitoring-operator
installed: #306

resolves #306
elfiesmelfie added a commit that referenced this issue Sep 20, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s.
ClusterRoles, and STF relies on these being present.
These are not present when using CRC, so ClusterRoles need to be
explicitly created.

The names of the ClusterRoles have been updated, in case there is some
conflict when cluster-monitoring-operator is installed after STF.

This is a workaround for not having cluster-monitoring-operator
installed: #306

resolves #306
elfiesmelfie added a commit that referenced this issue Sep 21, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s.
ClusterRoles, and STF relies on these being present.
These are not present when using CRC, so ClusterRoles need to be
explicitly created.

The names of the ClusterRoles have been updated, in case there is some
conflict when cluster-monitoring-operator is installed after STF.

This is a workaround for not having cluster-monitoring-operator
installed: #306

resolves #306
leifmadsen pushed a commit that referenced this issue Sep 21, 2023
The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s.
ClusterRoles, and STF relies on these being present.
These are not present when using CRC, so ClusterRoles need to be
explicitly created.

The names of the ClusterRoles have been updated, in case there is some
conflict when cluster-monitoring-operator is installed after STF.

This is a workaround for not having cluster-monitoring-operator
installed: #306

resolves #306
leifmadsen added a commit that referenced this issue Sep 21, 2023
* [issue#306] Add missing ClusterRoles

The cluster-monitoring-operator is required for STF to install. It
creates the required alertmanager-main and prometheus-k8s.
ClusterRoles, and STF relies on these being present.
These are not present when using CRC, so ClusterRoles need to be
explicitly created.

The names of the ClusterRoles have been updated, in case there is some
conflict when cluster-monitoring-operator is installed after STF.

This is a workaround for not having cluster-monitoring-operator
installed: #306

resolves #306

* Fix up the RBAC setup for prometheus-stf (#467)

Fix up the RBAC changes to fully get prometheus-stf working and
decoupled from prometheus-k8s. Changes to using a separate
prometheus-stf ClusterRole, ClusterRoleBinding, and ServiceAccount,
along with a Role and RoleBinding, all using prometheus-stf as the
ServiceAccount. Also updates the Alertmanager configuration to use
alertmanager-stf instead of alertmanager-main.

* Fix smoketest to use prometheus-stf for token retrieval

* Refactor smoketest script (#468)

* Refactor smoketest script

Perform a bit of smoketest refactoring and fix up a few bugs.

* Update alert trigger to use startsAt in order to potentially speed up
  delivery of the alerts. Failures in the SNMP_WEBHOOK_STATUS seems to
  be primarily to delayed alert notification through
  prometheus-snmp-webhook.
* Add an alert clean up task as part of the clean up logic at the end.
* Update openssl x509 to not use the -in flag which seems unnecessary
  and on some systems causes a failure.
* Add new SMOKETEST_VERBOSE boolean so local testing can skip massive
  amounts of information dumped to stdout.
* Remove curl pod using label selector for slightly cleaner output.
* Update failure check to combine RET and SNMP_WEBHOOK_STATUS since
  testing seems to show changes are slightly more reliable.

* Show logs from curl

* Remove nodes/metrics permission from ClusterRole

As part of least priviledge work, remove the nodes/metrics permission as
we're not scraping nodes for information. Everything appears to continue
working in STF without this permission.

* Move SCC RBAC from ClusterRole to Role

Working on simplifying and reducing our access scope as much as
possible. It appears moving SCC RBAC from ClusterRole to Role allows
things to continue to work with Prometheus. It's possible further
testing may reveal this will need to reverted.

* Convert alertmanager-stf Role to ClusterRole (#473)

Convert alertmanager-stf Role to ClusterRole as the tokenreviews and
subjectaccessreviews resources need to be accessable at the cluster
scope.

* Create ClusterRoleBinding and Role for alertmanager (#475)

* Create ClusterRoleBinding and Role for alertmanager

Create appropriate ClusterRoleBinding and Role for alertmanager-stf,
breaking out SCC into a Role vs ClusterRole to keep things in alignment
to prometheus-stf RBAC setup.

* Adjust smoketest.sh for SNMP webhook test failures

Adjust the smoketest script to also fail when the SNMP webhook test has
failed. Add a wait condition for the curl pod to complete so logs can be
retrieved.

* Add *RoleBinding rescue capabilities

If changes happen to the ClusterRoleBinding or RoleBinding then
generally the system is not going to allow you to patch the object. Adds
block/rescue logic to remove the existing ClusterRoleBinding or
RoleBinding before creating it when patching the object fails.

---------

Co-authored-by: Leif Madsen <lmadsen@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

2 participants