Why is beyla included in the instance label of the traces_service_graph metric?

[dscho99]

Hi Community
I'm a beginner with OpenTelemetry (otel) and Beyla.
I'm asking this question after installing the provided Helm chart.

Looking at the following metric:

traces_service_graph_request_total{client="tsdb-timescaledb", client_service_namespace="dscho-system", instance="beyla-mxk66:5289", job="dscho-system/python3.9", server="kubernetes", server_service_namespace="default", source="beyla"}

Q1. The instance label is set to beyla. What is the reason for this?

Q2. If this is the expected behavior, what is the reason for the Beyla instance being collected instead of the TSDB instance?

Q3. I couldn't find documentation related to this metric. Is there any documentation available that could provide more information?

[grcevski]

Hi @dscho99,

Can you give us a bit more details on how you deployed the monitoring side of this setup? Are you using Prometheus scraping for the metrics or OpenTelemetry push. One reason for the instance to be set to Beyla is related to how Prometheus scrapes the metrics. If honor_labels is not set to the Prometheus configuration for Beyla, it will overwrite the instance value we set (which in your case should be for timescaledb) and set the value to what it sourced the information from.

[dscho99]

Thank you for your response

I installed Beyla (as DaemonSets), Tempo, and OpenTelemetry-Demo (including OpenTelemetry and Prometheus) using Helm.

The architecture is as follows:
Beyla -> OpenTelemetry -> Prometheus, using a push model instead of a scrape model to send data to OpenTelemetry.

Configurations:
Beyla Environment Variables:

- name: BEYLA_CONFIG_PATH
  value: /etc/beyla/config/beyla-config.yml
- name: OTEL_EXPORTER_OTLP_ENDPOINT
  value: http://my-otel-demo-otelcol.demo:4318
- name: BEYLA_OTEL_METRICS_FEATURES
  value: application,application_span,application_service_graph,network,application_process
- name: BEYLA_OTEL_ALLOW_SERVICE_GRAPH_SELF_REFERENCES
  value: "false"
- name: BEYLA_NETWORK_METRICS
  value: "true"
- name: BEYLA_KUBE_METADATA_ENABLE
  value: "true"
- name: BEYLA_LOG_LEVEL
  value: DEBUG
- name: BEYLA_BPF_DEBUG
  value: "1"
- name: BEYLA_PRINT_TRACES
  value: "true"

Beyla Configuration (beyla-config.yml):

network:
  enable: true
discovery:
  services:
    - k8s_namespace: .
  exclude_services:
    - exe_path: ".*alloy.*|.*otelcol.*|.*beyla.*"
    - k8s_pod_labels:
        app: console
attributes:
  kubernetes:
    enable: true
  select:
    beyla_network_flow_bytes:
      include:
        - k8s.src.owner.type
        - k8s.dst.owner.type
        - k8s.dst.owner.name
        - k8s.src.owner.name
        - direction
filter:
  network:
    k8s_dst_owner_name:
      not_match: '{*jaeger-agent*,*prometheus*,*promtail*,*grafana-agent*}'
    k8s_src_owner_name:
      not_match: '{*jaeger-agent*,*prometheus*,*promtail*,*grafana-agent*}'
      

OpenTelemetry Configuration:

data:
  relay: |
    exporters:
      opensearch:
        http:
          endpoint: http://otel-demo-opensearch:9200
          tls:
            insecure: true
        logs_index: otel
      otlp:
        endpoint: 'my-release-tempo:4317'
        tls:
          insecure: true
      otlphttp/prometheus:
        endpoint: http://my-otel-demo-prometheus-server:9090/api/v1/otlp
        tls:
          insecure: true
    extensions:
      health_check:
        endpoint: ${env:MY_POD_IP}:13133
    processors:
      batch: {}
      k8sattributes:
        extract:
          metadata:
          - k8s.namespace.name
          - k8s.deployment.name
          - k8s.statefulset.name
          - k8s.daemonset.name
          - k8s.cronjob.name
          - k8s.job.name
          - k8s.node.name
          - k8s.pod.name
          - k8s.pod.uid
          - k8s.pod.start_time
        passthrough: false
        pod_association:
        - sources:
          - from: resource_attribute
            name: k8s.pod.ip
        - sources:
          - from: resource_attribute
            name: k8s.pod.uid
        - sources:
          - from: connection
      resource:
        attributes:
        - action: insert
          from_attribute: k8s.pod.uid
          key: service.instance.id
    receivers:
      httpcheck/frontendproxy:
        targets:
        - endpoint: http://my-otel-demo-frontendproxy:8080
      otlp:
        protocols:
          grpc:
            endpoint: ${env:MY_POD_IP}:4317
          http:
            cors:
              allowed_origins:
              - http://*
              - https://*
            endpoint: ${env:MY_POD_IP}:4318
      prometheus:
        config:
          scrape_configs:
          - job_name: opentelemetry-collector
            scrape_interval: 10s
            static_configs:
            - targets:
              - ${env:MY_POD_IP}:8888
    service:
      extensions:
      - health_check
      pipelines:
        logs:
          exporters:
          - opensearch
          processors:
          - k8sattributes
          - resource
          - batch
          receivers:
          - otlp
        metrics:
          exporters:
          - otlphttp/prometheus
          processors:
          - batch
          - resource
          receivers:
          - otlp
        traces:
          exporters:
          - otlp
          processors:
          - k8sattributes
          - resource
          - batch
          receivers:
          - otlp
      telemetry:
        metrics:
          address: ${env:MY_POD_IP}:8888

After applying server label filtering, the output is as follows:

query:
rate(traces_service_graph_request_total{server="tsdb-timescaledb"}[5m])


result:
{client="dscho-host-master", client_service_namespace="dscho-system", instance="tsdb-timescaledb-0:timescaledb", job="dscho-system/tsdb-timescaledb", server="tsdb-timescaledb", server_service_namespace="dscho-system", source="beyla"}	0
{client="dscho-host-master", instance="beyla-ztdnf:5289", job="python3.9", server="tsdb-timescaledb", server_service_namespace="dscho-system", source="beyla"}	0.020338983050847456

[grcevski]

Hm, that's surprising. Is it possible for you to see where if anywhere is the port 5289 set in your configuration? Maybe by listing the pods metadata? It's very peculiar that we have that port, it looks like purposefully set.

[gouthamve]

I can replicate this and I think this is a bug, so I moved it here: #1499

[dscho99]

@grcevski

there is nowhere that sets the 5289 port.
i restarted the TSDB pod and checked again.

{client="dscho-host-master", instance="beyla-tgqnv:989865", job="python3.9", server="tsdb-timescaledb", server_service_namespace="dscho-system", source="beyla"}	0.003389830508474576
{client="kubernetes", client_service_namespace="default", instance="beyla-tgqnv:989865", job="default/python3.9", server="tsdb-timescaledb", server_service_namespace="dscho-system", source="beyla"}	0

i thought this was a port, but it seems to be a PID.

root@dscho-host-master:~# ps -ef | grep 989865
tpl       989865  989844  0 13:36 ?        00:00:03 /usr/bin/python3 /usr/bin/patroni /etc/timescaledb/patroni.yaml
root     1050962  988579  0 15:27 pts/1    00:00:00 grep --color=auto 989865

{client="dscho-host-master", instance="beyla-tgqnv:5982", job="calico-node", server="kubernetes", server_service_namespace="default", source="beyla"}
root@dscho-host-master:~# ps -ef | grep 5982
root        5982    5963  0  2024 ?        00:05:41 calico-node -monitor-token

[mariomac]

The instance is set this way:

1. By default, populate the instance with the Beyla host name + the PID of the instrumented application. This covers the case when Beyla runs as an OS Process.
2. Then, if running on kubernetes, it overrides the instance with the instrumented Pod + Container names.

It seems that the second step is not being performed.

Might happen that this "calico-node" is not a Pod but a process that runs in the Node? In that case it would be expected because Beyla does not have any other metadata to identify it.

In any case, @dscho99 can you share some debug logs of the Beyla start process? I'd need to see if there is any issue with the kubernetes metadata connection

[grcevski]

To follow up on this, @dscho99, you can enable beyla debug logs with BEYLA_LOG_LEVEL=debug set to the Beyla deployment as environment variable.

[dscho99]

calico-node is pod

beyla log
log43.log

it seems to occur only in specific cases.
in general, the instance appears correctly as shown below:

{client="dscho-host-master", client_service_namespace="dscho-system", instance="tsdb-timescaledb-0:timescaledb", job="acc-system/tsdb-timescaledb", server="tsdb-timescaledb", server_service_namespace="dscho-system", source="beyla"}	15.324119975988614
{client="dscho-host-master", instance="beyla-qwhg4:989865", job="python3.9", server="tsdb-timescaledb", server_service_namespace="dscho-system", source="beyla"}	0.01694915254237288

[mariomac]

In the above example, can you check in your metrics explorer when was the undecorated metric updated for the last time?

I think the case might be:

1. Beyla starts and discovers the timescaledb process before it has internally stored the Pod metadata, so it decorates it as a node-level regular process.
2. Beyla receives the Kubernetes Pod metadata, then starts decorating the metric with the Pod information.

If this is the case, you would see that the undecorated service graph metrics are only reported during beyla startup and then they stop getting updated.

This is still an undesired behavior and we need to fix it.

[dscho99]

The logs passed by too quickly, so I couldn’t check the updated timestamp.
Thank you for your answer.