Receiving bogus TLS handshakes when using K8s attributes processor on some namespaces

[gebn]

Component(s)

processor/k8sattributes

What happened?

Description

When monitoring a specific namespace with the k8sattributes processor, every 10s, two messages are printed at the same time from zapgrpc about "received bogus greeting from client". If I change the config to monitor another namespace instead, these messages are not printed. It reads as if something is trying to connect to the collector with TLS every 10s, and failing. There is no indication of source IP in the logs to verify what the client is. Changing the config alone would not trigger this, there must be something else going on.

Steps to Reproduce

Start the collector with

  k8sattributes:
    filter:
      namespace: a

then change to

  k8sattributes:
    filter:
      namespace: b

Expected Result

No "received bogus greeting from client" error messages, or at least the behaviour to be consistent when monitoring both namespaces.

Actual Result

"received bogus greeting from client" error messages when monitoring one namespace, and no such messages when monitoring the other.

Collector version

v0.94.0

Environment information

Official Docker image

OpenTelemetry Collector configuration

processors:
  ...
  k8sattributes:
    filter:
      namespace: <ns>
service:
  telemetry:
    logs:
      level: debug
  extensions:
  - health_check
  pipelines:
    traces/internal:
      receivers:
      - otlp/internal
      processors:
      - k8sattributes
      - batch
      - resource
      exporters:
      - kafka

Log output

2024-02-21T18:08:40.331Z        info    zapgrpc/zapgrpc.go:178  [core] [Server #1 ListenSocket #2] ListenSocket created {"grpc_log": true}
2024-02-21T18:08:42.143Z        info    zapgrpc/zapgrpc.go:178  [transport] [server-transport 0xc002fff1e0] Closing: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"\\x16\\x03\\x01\\x00\\xee\\x01\\x00\\x00\\xea\\x03\\x03\\xd6\\xe7\\xd3\\x1a\\xa8\\xa1\\x9eC|\\xae\\xf6\\xdd\\xc5\""  {"grpc_log": true}
2024-02-21T18:08:42.143Z        info    zapgrpc/zapgrpc.go:178  [core] [Server #1] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"\\x16\\x03\\x01\\x00\\xee\\x01\\x00\\x00\\xea\\x03\\x03\\xd6\\xe7\\xd3\\x1a\\xa8\\xa1\\x9eC|\\xae\\xf6\\xdd\\xc5\""       {"grpc_log": true}

Additional context

No response

[github-actions]

Pinging code owners:

• processor/k8sattributes: @dmitryax @rmfitzpatrick @fatsheep9146 @TylerHelmuth

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Hello @gebn, can you share the full configuration you're using? (Including the receiver and exporter configurations)

[gebn]

Hi @crobert-1, will share as much as I'm able to, I'll get back to you towards the end of next week.

[gebn]

Frustratingly, I can no longer reproduce this. I will reopen if it happens again.