filter processor regex broken in 0.108.0 app version

[ceastman-r7]

Component(s)

processor/filter

What happened?

Description

Going from app version 0.107.0 to 0.108.0 breaks the filter processor regex exclude.

Steps to Reproduce

using the following filter:
filter/dropmetrics:
error_mode: ignore
metrics:
exclude:
match_type: regexp
metric_names:
- containerd.*
- coredns.*
- datadog.*
- otel.*
- istio.mesh.*

Expected Result

I am expecting the metrics that start with these names will be filtered out (dropped).

Actual Result

metrics with these names are passed onto the exporters. i.e. otel.system.cpu.system

Collector version

0.107.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

exporters:
      datadog:
        api:
          key: ${env:DD_API_KEY}
          site: datadoghq.com
        metrics:
          resource_attributes_as_tags: true
      debug:
        verbosity: detailed
      otlphttp:
        auth:
          authenticator: basicauth/otlp
        endpoint: ${env:otel_otlp_endpoint}
      prometheusremotewrite/external:
        auth:
          authenticator: basicauth/metrics
        endpoint: ${env:otel_metrics_endpoint}
        resource_to_telemetry_conversion:
          enabled: true
        target_info:
          enabled: false
    extensions:
      basicauth/metrics:
        client_auth:
          password: ${env:otel_metrics_api}
          username: ${env:otel_metrics_user}
      basicauth/otlp:
        client_auth:
          password: ${env:otel_otlp_api}
          username: ${env:otel_otlp_user}
      file_storage:
        directory: /var/lib/otelcol
      health_check:
        endpoint: ${env:MY_POD_IP}:13133
    processors:
      attributes/addtags:
        actions:
        - action: insert
          key: kube_cluster_name
          value: ${env:K8S_CLUSTER_NAME}
        - action: insert
          key: otel_account_alias
          value: kubeplay-staging
      attributes/droptags:
        actions:
        - action: delete
          key: app_kubernetes_io_instance
        - action: delete
          key: app_kubernetes_io_name
        - action: delete
          key: cloud.platform
        - action: delete
          key: cloud.provider
        - action: delete
          key: cloud.region
        - action: delete
          key: cloud_platform
        - action: delete
          key: cloud_provider
        - action: delete
          key: cloud_region
        - action: delete
          key: container.id
        - action: delete
          key: container.image.name
        - action: delete
          key: container.runtime.version
        - action: delete
          key: container_id
        - action: delete
          key: container_image_name
        - action: delete
          key: container_runtime_version
        - action: delete
          key: destination_canonical_revision
        - action: delete
          key: destination_canonical_service
        - action: delete
          key: destination_cluster
        - action: delete
          key: family
        - action: delete
          key: host.id
        - action: delete
          key: host.image.id
        - action: delete
          key: host.name
        - action: delete
          key: host.type
        - action: delete
          key: host_id
        - action: delete
          key: host_image_id
        - action: delete
          key: host_name
        - action: delete
          key: host_type
        - action: delete
          key: http.scheme
        - action: delete
          key: http_scheme
        - action: delete
          key: instance
        - action: delete
          key: internal.ip
        - action: delete
          key: internal_ip
        - action: delete
          key: job
        - action: delete
          key: k8s.cronjob.uid
        - action: delete
          key: k8s.daemonset.uid
        - action: delete
          key: k8s.deployment.uid
        - action: delete
          key: k8s.hpa.uid
        - action: delete
          key: k8s.job.uid
        - action: delete
          key: k8s.namespace.uid
        - action: delete
          key: k8s.node.uid
        - action: delete
          key: k8s.pod.ip
        - action: delete
          key: k8s.pod.uid
        - action: delete
          key: k8s.replicaset.uid
        - action: delete
          key: k8s.statefulset.uid
        - action: delete
          key: k8s_cronjob_uid
        - action: delete
          key: k8s_daemonset_uid
        - action: delete
          key: k8s_deployment_uid
        - action: delete
          key: k8s_hpa_uid
        - action: delete
          key: k8s_job_uid
        - action: delete
          key: k8s_namespace_uid
        - action: delete
          key: k8s_node_uid
        - action: delete
          key: k8s_pod_ip
        - action: delete
          key: k8s_pod_uid
        - action: delete
          key: k8s_replicaset_uid
        - action: delete
          key: k8s_statefulset_uid
        - action: delete
          key: kubelet.version
        - action: delete
          key: kubelet_version
        - action: delete
          key: net.host.name
        - action: delete
          key: net.host.port
        - action: delete
          key: net_host_name
        - action: delete
          key: net_host_port
        - action: delete
          key: os.image
        - action: delete
          key: os.type
        - action: delete
          key: os_image
        - action: delete
          key: os_type
        - action: delete
          key: otel.scope.name
        - action: delete
          key: otel_scope_name
        - action: delete
          key: plugin
        - action: delete
          key: pod_template_hash
        - action: delete
          key: policy.background.mode
        - action: delete
          key: policy.type
        - action: delete
          key: policy_background_mode
        - action: delete
          key: policy_type
        - action: delete
          key: proto
        - action: delete
          key: provider.id
        - action: delete
          key: provider_id
        - action: delete
          key: request_protocol
        - action: delete
          key: resource.request.operation
        - action: delete
          key: resource_request_operation
        - action: delete
          key: response_flags
        - action: delete
          key: rule.execution.cause
        - action: delete
          key: rule.type
        - action: delete
          key: rule_execution_cause
        - action: delete
          key: rule_type
        - action: delete
          key: server
        - action: delete
          key: service.instance
        - action: delete
          key: service.instance.id
        - action: delete
          key: service_instance
        - action: delete
          key: service_instance_id
        - action: delete
          key: source_canonical_revision
        - action: delete
          key: source_canonical_service
        - action: delete
          key: source_cluster
        - action: delete
          key: source_principal
        - action: delete
          key: system.uuid
        - action: delete
          key: system_uuid
        - action: delete
          key: uid
        - action: delete
          key: zone
        - action: delete
          key: zones
      batch:
        send_batch_max_size: 100
        send_batch_size: 10
        timeout: 10s
      filter/dropmetrics:
        error_mode: ignore
        metrics:
          exclude:
            match_type: regexp
            metric_names:
            - containerd.*
            - coredns.*
            - datadog.*
            - otel.*
            - istio.mesh.*
      filter/ignorenamespaces:
        error_mode: ignore
        logs:
          log_record:
          - resource.attributes["namespace"] == "logging"
          - resource.attributes["namespace"] == "istio-addons"
          - resource.attributes["namespace"] == "istio-addons-ingress"
      filter/includemetrics:
        metrics:
          include:
            match_type: strict
            metric_names:
            - coredns_cache_hits_total
            - coredns_dns_request_duration_seconds
            - coredns_dns_requests_total
            - coredns_dns_responses_total
            - istio_tcp_connections_opened_total
            - k8s.container.cpu_request_utilization
            - k8s.container.cpu_request
            - k8s.container.memory_limit_utilization
            - k8s.container.memory_request_utilization
            - k8s.container.memory_request
            - k8s.container.ready
            - k8s.container.restarts
            - k8s.daemonset.desired_scheduled_nodes
            - k8s.daemonset.ready_nodes
            - k8s.deployment.available
            - k8s.deployment.desired
            - k8s.node.cpu.usage
            - k8s.node.memory.usage
            - k8s.node.network.errors
            - k8s.node.network.io
            - k8s.pod.cpu.usage
            - k8s.pod.memory.usage
            - k8s.pod.network.errors
            - k8s.pod.network.io
            - k8s.pod.phase
            - k8s.replicaset.available
            - k8s.replicaset.desired
            - k8s.statefulset.desired_pods
            - k8s.statefulset.ready_pods
            - kube_node_info
            - kube_node_spec_unschedulable
            - kube_node_status_allocatable
            - kube_node_status_capacity
            - kube_node_status_condition
            - kube_pod_container_info
            - kube_pod_container_resource_limits
            - kube_pod_container_resource_requests
            - kube_pod_container_status_ready
            - kube_pod_container_status_running
            - kube_pod_container_status_terminated
            - kube_pod_container_status_waiting_reason
            - kube_pod_container_status_waiting
            - kube_pod_info
            - kube_pod_init_container_info
            - kube_pod_init_container_status_ready
            - kube_pod_init_container_status_running
            - kube_pod_init_container_status_terminated
            - kube_pod_init_container_status_waiting
            - kube_service_info
            - policy_report_result
            - test.countermetric
            - test.gaugemetric
      filter/istiodinfo:
        error_mode: ignore
        logs:
          log_record:
          - IsMatch(body, ".*info.*") and resource.attributes["namespace"] == "istio-system"
            and resource.attributes["container"] == "discovery"
      k8sattributes:
        auth_type: serviceAccount
        extract:
          labels:
          - from: pod
            key: app.kubernetes.io/instance
            tag_name: kube_app_instance
          - from: pod
            key: app.kubernetes.io/version
            tag_name: kube_app_version
          - from: pod
            key: app.kubernetes.io/component
            tag_name: kube_app_component
          metadata:
          - container.image.tag
          - k8s.pod.name
          - k8s.pod.uid
          - k8s.deployment.name
          - k8s.node.name
          - k8s.namespace.name
          - k8s.daemonset.name
          - k8s.job.name
          - k8s.cronjob.name
          - k8s.statefulset.name
          - k8s.container.name
        passthrough: false
        pod_association:
        - sources:
          - from: resource_attribute
            name: k8s.pod.ip
        - sources:
          - from: resource_attribute
            name: k8s.pod.uid
      memory_limiter:
        check_interval: 5s
        limit_percentage: 80
        spike_limit_percentage: 25
      resource/droptags:
        attributes:
        - action: delete
          key: app_kubernetes_io_instance
        - action: delete
          key: app_kubernetes_io_name
        - action: delete
          key: cloud.platform
        - action: delete
          key: cloud.provider
        - action: delete
          key: cloud.region
        - action: delete
          key: cloud_platform
        - action: delete
          key: cloud_provider
        - action: delete
          key: cloud_region
        - action: delete
          key: container.id
        - action: delete
          key: container.image.name
        - action: delete
          key: container.runtime.version
        - action: delete
          key: container_id
        - action: delete
          key: container_image_name
        - action: delete
          key: container_runtime_version
        - action: delete
          key: destination_canonical_revision
        - action: delete
          key: destination_canonical_service
        - action: delete
          key: destination_cluster
        - action: delete
          key: family
        - action: delete
          key: host.id
        - action: delete
          key: host.image.id
        - action: delete
          key: host.name
        - action: delete
          key: host.type
        - action: delete
          key: host_id
        - action: delete
          key: host_image_id
        - action: delete
          key: host_name
        - action: delete
          key: host_type
        - action: delete
          key: http.scheme
        - action: delete
          key: http_scheme
        - action: delete
          key: instance
        - action: delete
          key: internal.ip
        - action: delete
          key: internal_ip
        - action: delete
          key: job
        - action: delete
          key: k8s.cronjob.uid
        - action: delete
          key: k8s.daemonset.uid
        - action: delete
          key: k8s.deployment.uid
        - action: delete
          key: k8s.hpa.uid
        - action: delete
          key: k8s.job.uid
        - action: delete
          key: k8s.namespace.uid
        - action: delete
          key: k8s.node.uid
        - action: delete
          key: k8s.pod.ip
        - action: delete
          key: k8s.pod.uid
        - action: delete
          key: k8s.replicaset.uid
        - action: delete
          key: k8s.statefulset.uid
        - action: delete
          key: k8s_cronjob_uid
        - action: delete
          key: k8s_daemonset_uid
        - action: delete
          key: k8s_deployment_uid
        - action: delete
          key: k8s_hpa_uid
        - action: delete
          key: k8s_job_uid
        - action: delete
          key: k8s_namespace_uid
        - action: delete
          key: k8s_node_uid
        - action: delete
          key: k8s_pod_ip
        - action: delete
          key: k8s_pod_uid
        - action: delete
          key: k8s_replicaset_uid
        - action: delete
          key: k8s_statefulset_uid
        - action: delete
          key: kubelet.version
        - action: delete
          key: kubelet_version
        - action: delete
          key: net.host.name
        - action: delete
          key: net.host.port
        - action: delete
          key: net_host_name
        - action: delete
          key: net_host_port
        - action: delete
          key: os.image
        - action: delete
          key: os.type
        - action: delete
          key: os_image
        - action: delete
          key: os_type
        - action: delete
          key: otel.scope.name
        - action: delete
          key: otel_scope_name
        - action: delete
          key: plugin
        - action: delete
          key: pod_template_hash
        - action: delete
          key: policy.background.mode
        - action: delete
          key: policy.type
        - action: delete
          key: policy_background_mode
        - action: delete
          key: policy_type
        - action: delete
          key: proto
        - action: delete
          key: provider.id
        - action: delete
          key: provider_id
        - action: delete
          key: request_protocol
        - action: delete
          key: resource.request.operation
        - action: delete
          key: resource_request_operation
        - action: delete
          key: response_flags
        - action: delete
          key: rule.execution.cause
        - action: delete
          key: rule.type
        - action: delete
          key: rule_execution_cause
        - action: delete
          key: rule_type
        - action: delete
          key: server
        - action: delete
          key: service.instance
        - action: delete
          key: service.instance.id
        - action: delete
          key: service_instance
        - action: delete
          key: service_instance_id
        - action: delete
          key: source_canonical_revision
        - action: delete
          key: source_canonical_service
        - action: delete
          key: source_cluster
        - action: delete
          key: source_principal
        - action: delete
          key: system.uuid
        - action: delete
          key: system_uuid
        - action: delete
          key: uid
        - action: delete
          key: zone
        - action: delete
          key: zones
      resource/loki:
        attributes:
        - action: upsert
          from_attribute: app.kubernetes.io/component
          key: component
        - action: insert
          from_attribute: k8s.container.name
          key: container
        - action: upsert
          from_attribute: release
          key: service.instance.id
        - action: upsert
          from_attribute: app.kubernetes.io/instance
          key: service.instance.id
        - action: insert
          from_attribute: k8s.namespace.name
          key: namespace
        - action: insert
          from_attribute: k8s.namespace.name
          key: service.namespace
        - action: insert
          from_attribute: app
          key: service.name
        - action: insert
          from_attribute: k8s.node.name
          key: node_name
        - action: insert
          from_attribute: k8s.pod.name
          key: pod
        - action: insert
          key: loki.resource.labels
          value: app, container, pod, namespace, node_name, response_code
        - action: insert
          key: loki.format
          value: raw
      resourcedetection:
        detectors:
        - env
        - eks
        - ec2
        - system
        override: false
        timeout: 2s
      transform/istioproxy:
        error_mode: ignore
        log_statements:
        - context: log
          statements:
          - merge_maps(cache, ParseJSON(body), "upsert") where resource.attributes["k8s.container.name"]
            == "istio-proxy" and IsMatch(body, "^\\{")
          - set(resource.attributes["authority"], cache["authority"])
          - set(resource.attributes["bytes_received"], cache["bytes_received"])
          - set(resource.attributes["connection_termination_details"], cache["connection_termination_details"])
          - set(resource.attributes["downstream_local_address"], cache["downstream_local_address"])
          - set(resource.attributes["downstream_remote_address"], cache["downstream_remote_address"])
          - set(resource.attributes["duration"], cache["duration"])
          - set(resource.attributes["method"], cache["method"])
          - set(resource.attributes["path"], cache["path"])
          - set(resource.attributes["protocol"], cache["protocol"])
          - set(resource.attributes["request_duration"], cache["request_duration"])
          - set(resource.attributes["request_id"], cache["request_id"])
          - set(resource.attributes["requested_server_name"], cache["requested_server_name"])
          - set(resource.attributes["response_code_details"], cache["auresponse_code_detailsthority"])
          - set(resource.attributes["response_code"], cache["response_code"])
          - set(resource.attributes["response_duration"], cache["response_duration"])
          - set(resource.attributes["response_flags"], cache["response_flags"])
          - set(resource.attributes["response_tx_duration"], cache["response_tx_duration"])
          - set(resource.attributes["start_time"], cache["start_time"])
          - set(resource.attributes["upstream_cluster"], cache["upstream_cluster"])
          - set(resource.attributes["upstream_host"], cache["upstream_host"])
          - set(resource.attributes["upstream_local_address"], cache["upstream_local_address"])
          - set(resource.attributes["upstream_transport_failure_reason"], cache["upstream_transport_failure_reason"])
          - set(resource.attributes["user_agent"], cache["user_agent"])
          - set(resource.attributes["x_b3_parentspanid"], cache["x_b3_parentspanid"])
          - set(resource.attributes["x_b3_traceid"], cache["x_b3_traceid"])
          - set(resource.attributes["x_forwarded_for"], cache["x_forwarded_for"])
      transform/log:
        log_statements:
        - context: log
          statements:
          - set(body, attributes["log"])
    receivers:
      datadog:
        endpoint: 0.0.0.0:8127
        read_timeout: 60s
      filelog:
        exclude:
        - /var/log/pods/platform-delivery_otel-ds*_*/opentelemetry-collector/*.log
        include:
        - /var/log/pods/*/*/*.log
        include_file_name: false
        include_file_path: true
        operators:
        - id: container-parser
          max_log_size: 102400
          type: container
        retry_on_failure:
          enabled: true
        start_at: end
        storage: file_storage
      jaeger:
        protocols:
          grpc:
            endpoint: ${env:MY_POD_IP}:14250
          thrift_compact:
            endpoint: ${env:MY_POD_IP}:6831
          thrift_http:
            endpoint: ${env:MY_POD_IP}:14268
      kubeletstats:
        auth_type: serviceAccount
        collection_interval: 60s
        endpoint: https://${env:KUBE_NODE_NAME}:10250
        extra_metadata_labels:
        - k8s.volume.type
        - container.id
        insecure_skip_verify: true
        metric_groups:
        - container
        - pod
        - volume
        - node
        metrics:
          container.cpu.usage:
            enabled: true
          container.uptime:
            enabled: true
          k8s.container.cpu.node.utilization:
            enabled: true
          k8s.container.cpu_limit_utilization:
            enabled: true
          k8s.container.cpu_request_utilization:
            enabled: true
          k8s.container.memory_limit_utilization:
            enabled: true
          k8s.container.memory_request_utilization:
            enabled: true
          k8s.node.cpu.usage:
            enabled: true
          k8s.node.uptime:
            enabled: true
          k8s.pod.cpu.node.utilization:
            enabled: true
          k8s.pod.cpu.usage:
            enabled: true
          k8s.pod.cpu_limit_utilization:
            enabled: true
          k8s.pod.cpu_request_utilization:
            enabled: true
          k8s.pod.memory_limit_utilization:
            enabled: true
          k8s.pod.memory_request_utilization:
            enabled: true
          k8s.pod.uptime:
            enabled: true
        node: ${env:KUBE_NODE_NAME}
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      prometheus:
        config:
          scrape_configs:
          - job_name: opentelemetry-collector
            scrape_interval: 10s
            static_configs:
            - targets:
              - ${env:MY_POD_IP}:8888
      zipkin:
        endpoint: ${env:MY_POD_IP}:9411
    service:
      extensions:
      - basicauth/metrics
      - basicauth/otlp
      - health_check
      - file_storage
      pipelines:
        logs:
          exporters:
          - otlphttp
          processors:
          - k8sattributes
          - resourcedetection
          - transform/istioproxy
          - resource/loki
          - transform/log
          - filter/istiodinfo
          - filter/ignorenamespaces
          receivers:
          - otlp
          - filelog
        metrics:
          exporters:
          - datadog
          processors:
          - attributes/addtags
          - resourcedetection
          - k8sattributes
          - filter/dropmetrics
          - resource/droptags
          - attributes/droptags
          - batch
          receivers:
          - otlp
          - datadog
        metrics/external:
          exporters:
          - prometheusremotewrite/external
          processors:
          - attributes/addtags
          - resourcedetection
          - k8sattributes
          - filter/includemetrics
          - resource/droptags
          - attributes/droptags
          - batch
          receivers:
          - kubeletstats
        metrics/externalotlp:
          exporters:
          - prometheusremotewrite/external
          processors:
          - attributes/addtags
          - resourcedetection
          - k8sattributes
          - filter/dropmetrics
          - resource/droptags
          - attributes/droptags
          - batch
          receivers:
          - otlp
          - datadog
        traces:
          exporters:
          - debug
          processors:
          - memory_limiter
          - batch
          receivers:
          - otlp
          - jaeger
          - zipkin
      telemetry:
        logs:
          encoding: json
        metrics:
          address: ${env:MY_POD_IP}:8888
          level: detailed

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• processor/filter: @TylerHelmuth @boostchicken

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

Try:

      filter/dropmetrics:
        error_mode: ignore
        metrics:
          metric:
            - IsMatch(name, "containerd.*")
            - IsMatch(name, "coredns.*")
            - IsMatch(name, "datadog.*")
            - IsMatch(name, "otel.*")
            - IsMatch(name, "istio\\.mesh.*")

Otherwise it is likely the name not matching the regex. This happens most often because the data as the collector sees it is different than your mental model. You can enable service.telemetry.logs.level: debug or a debug exporter to see exactly how the collector sees that data.

[ceastman-r7]

@TylerHelmuth thank you, that filter appears to be working as expected.