Grafana Agent Operator PodLogs CR pipelineStages

[tmarshall98]

Hello, I am attempting to setup a custom pipeline in the Grafana Agent PodLogs pipelineStages.

---
apiVersion: monitoring.grafana.com/v1alpha1
kind: PodLogs
metadata:
  labels:
    instance: primary
  name: kubernetes-pods-production-apps
spec:
  pipelineStages:
    - docker: {}
    - match:
        selector: '{namespace="my-namespace"}'
        stages: |
          - static_labels:
              pipeline_stage: my-stage-name
          - logfmt:
              mapping:
                msg: msg
                level: level
                time: time
                error: error
          - labels:
              level: level
              time: time
              error: error
              msg: msg 
  namespaceSelector:
    any: true
  selector:
    matchLabels: {}

Here is my current CR definition. I am getting logs into Loki from containers on my cluster without issue, but I noticed that the
sub-stages inside matches is actually getting lost inside the grafana-agent config in the pods

/var/lib/grafana-agent/config # cat agent.yml 
logs:
    configs:
        - clients:
            - external_labels:
                cluster: grafana-agent-operator/standard-grafana-agent
              url: <redacted>
          name: grafana-agent-operator/primary
          scrape_configs:
            - job_name: podLogs/grafana-agent-operator/kubernetes-pods-production-apps
              kubernetes_sd_configs:
                - role: pod
              pipeline_stages:
                - docker: {}
                - match:
                    selector: '{namespace="my-namespace"}'
                    stages:
                        - {}  # <--- notice how the static_labels
                        - {} # <-- and the logfmt became empty dictionaryies
                        - labels:
                            error: error
                            level: level
                            msg: msg
                            time: time

Any idea what's happening here?

[tmarshall98]

I started investigating how the config secret gets generated for the operator and found myself in the jsonnet files
the Match.stages key, which is a string, is getting parsed by marshal.intoStages(spec.Match.Stages) here

This is then calling the native function by the same name intoStages here

This function is assuming that stages is a PipelineStageSpec, which appears to be lacking the additional stage types, such as logfmt and static_labels

The loki version of this is a not even a static struct: https://github.com/grafana/loki/blob/c39786bd516c1839c4e698c7bed29573eeedca2b/clients/pkg/logentry/stages/stage.go#L137