Failed to watch *v1.ReplicaSet and Failed to list *v1.ReplicaSet

[thefirstofthe300]

Component(s)

No response

What happened?

Description

Given the OpenTelemetryCollector configuration:

  config: |
    receivers:
      otlp:
        protocols:
          grpc: {}
          http: {}
    processors:
      k8sattributes: {}
      batch: {}
    exporters:
      datadog:
        api:
          site: datadoghq.com
          key: ${env:DD_API_KEY}
    service:
      pipelines:
        metrics:
          receivers:
            - otlp
          processors:
            - k8sattributes
            - batch
          exporters:
            - datadog

The otelcol throws the error:

W0405 22:24:51.394408       1 reflector.go:539] k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:observability-system:opentelemetry-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0405 22:24:51.394454       1 reflector.go:147] k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:observability-system:opentelemetry-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope

over and over again.

Steps to Reproduce

Deploy the above provided configuration

Expected Result

No errors

Actual Result

Errors due to missing rbac permissions

Kubernetes Version

1.28.3

Operator version

0.97.1

Collector version

0.97.0

Environment information

No response

Log output

No response

Additional context

No response

[thefirstofthe300]

@iblancasa I believe you did most of the work getting this feature added?

[jicki]

me too

E0407 06:56:47.446856       1 reflector.go:147] k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:opentelemetry:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
W0407 06:57:22.787483       1 reflector.go:539] k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:opentelemetry:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0407 06:57:22.787518       1 reflector.go:147] k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:opentelemetry:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0407 06:57:59.754005       1 reflector.go:147] k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:opentelemetry:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
W0407 06:57:32.346543       1 reflector.go:539] k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:opentelemetry:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0407 06:57:32.346576       1 reflector.go:147] k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:opentelemetry:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope

[iblancasa]

Are you enabling the creation of the RBAC resources from the operator + giving the operator the permission to create RBAC resources?

[thefirstofthe300]

Yes. This error is thrown and appears to be the final permission needed to be granted to the OTEL collector by the operator.

…

On Mon, Apr 8, 2024, at 3:18 AM, Israel Blancas wrote: Are you enabling the creation of the RBAC resources from the operator + giving the operator the permission to create RBAC resources? — Reply to this email directly, view it on GitHub <#2823 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAYDZ4SDPVU2V6YZDC3636DY4JVF5AVCNFSM6AAAAABFZ3YWFOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBSGM3TMOBZHE>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[iblancasa]

@thefirstofthe300 Can you share more info?

• The RBAC created by the operator
• The RBAC you created to assign the permissions to the SA from the OTEL Operator

[thefirstofthe300]

@iblancasa The above config I provided should be enough to replicate the issue in your own environment AFAICT. The issue only appears if the k8s_attribute processor has no explicit configuration defined.

One thing to note is that this appears to be expected behavior. The OTEL collector helm chart adds replicasets permissions to the clusterrole if the k8sattribute processor is enabled: https://github.com/open-telemetry/opentelemetry-helm-charts/blob/main/charts/opentelemetry-collector/templates/clusterrole.yaml#L18-L28

The RBAC created by the operator is

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: "2024-04-05T22:22:12Z"
  labels:
    app.kubernetes.io/component: opentelemetry-collector
    app.kubernetes.io/instance: observability-system.opentelemetry
    app.kubernetes.io/managed-by: opentelemetry-operator
    app.kubernetes.io/name: opentelemetry-observability-system-cluster-role
    app.kubernetes.io/part-of: opentelemetry
    app.kubernetes.io/version: latest
  name: opentelemetry-observability-system-cluster-role
  resourceVersion: "292396896"
  uid: b9d06eb9-b1db-4158-b4bd-949c95ae846b
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - watch
  - list

[iblancasa]

It seems the documentation from the k8sattributesprocessor is wrong:

If you'd like to set up the k8sattributesprocessor to receive telemetry from across namespaces, it will need get, watch and list permissions on both pods and namespaces resources, for all namespaces and pods included in the configured filters. Additionally, when using k8s.deployment.uid or k8s.deployment.name the processor also needs get, watch and list permissions for replicasets resources.

Created an issue in the contrib repo: open-telemetry/opentelemetry-collector-contrib#32247