[atlas-operator] failing to watch some namespaces blocks reconciliation for all

[mbapostol]

This is the setup:

• operator deployed at cluster level
• permissions to watch/list/etc given at namespace level

The issue:

• The operator remains blocked at failing to list resources in the namespaces permissions were not given for (some) atlas resources
• No reconciliation is done, even for namespaces that are having all permissions
• Even one namespace can block reconciliation for all the rest

Wanted behaviour:

• operator should reconcile for the namespaces were all permissions are given and ignore the problematic ones
• or operator should reconcile for all namespaces and ignore only the resources it cannot watch/list/etc.

[mbapostol]

Note: some internal sensitive values were replaced

atlas-operator values:

mongodb-atlas-operator-crds:
enabled: false
watchNamespaces:
- namespace1
- namespace2
-
crossNamespaceRoles: false

role/role-binding deployed via a tenant chart at namespace level:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: mongo-operator-role
namespace: {{ .Release.Namespace }}
rules:

• apiGroups:
  • ""
    resources:
  • events
    verbs:
  • create
  • patch
• apiGroups:
  • ""
    resources:
  • secrets
    verbs:
  • create
  • delete
  • get
  • list
  • patch
  • update
  • watch
• apiGroups:
  • atlas.mongodb.com
    resources:
  • atlasdeployments
    verbs:
  • create
  • delete
  • get
  • list
  • patch
  • update
  • watch
• apiGroups:
  • atlas.mongodb.com
    resources:
  • atlasdeployments/status
    verbs:
  • get
  • patch
  • update

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: mongo-operator-sa-rb
namespace: {{ .Release.Namespace }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: mongo-operator-role
subjects:

• kind: ServiceAccount
  name: mongodb-atlas-operator
  namespace: {{ .Values.mongoOperatorNamespace }}

Everything works until one of the namespaces is listed in the "watchNamespaces" in operator values but either is missing the role/role-binding or is missing some resources in the role (when upgrading the operator version new atlas resources are supported and the operator tries to list them in every namespace)

E.g. after upgrading to 2.7.0 (and atlasipaccesslists CRD added to the cluster) but the new resource not added yet in the role at some namespace(s) level, operator gets stuck trying to list that resource; at this point the reconciliation is not being done for the namespaces that have the role updated:

{"level":"INFO","time":"2025-02-17T11:57:43.034Z","msg":"pkg/mod/k8s.io/client-go@v0.32.1/tools/cache/reflector.go:251: failed to list *v1.AtlasIPAccessList: atlasipaccesslists.atlas.mongodb.com is forbidden: User "system::mongodb-atlas-operator" cannot list resource "atlasipaccesslists" in API group "atlas.mongodb.com" in the namespace "namespace1""}
{"level":"ERROR","time":"2025-02-17T11:57:43.034Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/client-go@v0.32.1/tools/cache/reflector.go:251: Failed to watch *v1.AtlasIPAccessList: failed to list *v1.AtlasIPAccessList: atlasipaccesslists.atlas.mongodb.com is forbidden: User "system::mongodb-atlas-operator" cannot list resource "atlasipaccesslists" in API group "atlas.mongodb.com" in the namespace "namespace2""}

[s-urbaniak]

thank you for the issue @mbapostol, self-assigning and checking.

[s-urbaniak]

@mbapostol from what I see in the helm code the crossNamespaceRoles governs whether per-namespace roles and rolebindings will be generated. Can you please set crossNamespaceRoles: true and check if it works?