Grafana Agent Operator PodLogs CR to config generation improperly marshaling sub-stages

[tmarshall98]

Discussed in #6551

^{Originally posted by tmarshall98 February 28, 2024}
Hello, I am attempting to setup a custom pipeline in the Grafana Agent PodLogs pipelineStages.

---
apiVersion: monitoring.grafana.com/v1alpha1
kind: PodLogs
metadata:
  labels:
    instance: primary
  name: kubernetes-pods-production-apps
spec:
  pipelineStages:
    - docker: {}
    - match:
        selector: '{namespace="my-namespace"}'
        stages: |
          - static_labels:
              pipeline_stage: my-stage-name
          - logfmt:
              mapping:
                msg: msg
                level: level
                time: time
                error: error
          - labels:
              level: level
              time: time
              error: error
              msg: msg 
  namespaceSelector:
    any: true
  selector:
    matchLabels: {}

Here is my current CR definition. I am getting logs into Loki from containers on my cluster without issue, but I noticed that the
sub-stages inside matches is actually getting lost inside the grafana-agent config in the pods

/var/lib/grafana-agent/config # cat agent.yml 
logs:
    configs:
        - clients:
            - external_labels:
                cluster: grafana-agent-operator/standard-grafana-agent
              url: <redacted>
          name: grafana-agent-operator/primary
          scrape_configs:
            - job_name: podLogs/grafana-agent-operator/kubernetes-pods-production-apps
              kubernetes_sd_configs:
                - role: pod
              pipeline_stages:
                - docker: {}
                - match:
                    selector: '{namespace="my-namespace"}'
                    stages:
                        - {}  # <--- notice how the static_labels
                        - {} # <-- and the logfmt became empty dictionaryies
                        - labels:
                            error: error
                            level: level
                            msg: msg
                            time: time

I started investigating how the config secret gets generated for the operator and found myself in the jsonnet files.
the Match.stages key, which is a string, is getting parsed by marshal.intoStages(spec.Match.Stages) here

This is then calling the native function by the same name intoStages here

This function is assuming that stages is a PipelineStageSpec, which appears to be lacking the additional stage types, such as logfmt and static_labels

The loki version of this is a not even a static struct: https://github.com/grafana/loki/blob/c39786bd516c1839c4e698c7bed29573eeedca2b/clients/pkg/logentry/stages/stage.go#L137

[github-actions]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!