Label filters ignore parsed label if it exists in structured metadata

[hvenev]

Describe the bug
When a line has a label in structured metadata, and the label is then overwritten by a label format expression, the label filter uses the structured metadata instead of the updated value.

To Reproduce
Steps to reproduce the behavior:

1. Started Loki (SHA or version): v3.1.1
2. Sent record to /loki/api/v1/push:

{
  "streams": [
    {
      "stream": {"service_name":"test"},
      "values": [["1724169776000000000", "hello", {"x": "foo"}]]
    }
  ]
}

Command:

curl -H 'Content-Type: application/json' -d '{"streams": [{"stream": {"service_name":"test"},"values":[["1724169776000000000", "hello", {"x": "foo"}]]}]}' -v 'http://127.0.0.1:3100/loki/api/v1/push'

4. Sent queries:

• {service_name="test"} | label_format x="bar" | x="foo"
• {service_name="test"} | label_format x="bar" | x="bar"

Commands:

curl -s -G http://127.0.0.1:3100/loki/api/v1/query_range --data-urlencode start=1724169776000000000 --data-urlencode query='{service_name="test"} | label_format x="bar" | x="foo"' | jq '.data.result[]'
curl -s -G http://127.0.0.1:3100/loki/api/v1/query_range --data-urlencode start=1724169776000000000 --data-urlencode query='{service_name="test"} | label_format x="bar" | x="bar"' | jq '.data.result[]'

5. Observe that the first query (the one with the x="foo" filter) produces a result with x="bar", and the second query (the one with the x="bar" filter) produces no result.

Expected behavior
Given that label_format is supposed to always set x to "bar", only the second query (... | label_format x="bar" | x="bar") should ever produce a result. The first query (... | label_format x="bar" | x="foo") should never produce anything, as "bar" != "foo".

Environment:

• Infrastructure: bare-metal
• Deployment tool: /bin/bash

Screenshots, Promtail config, or terminal output

localhost ~ # curl -H 'Content-Type: application/json' -d '{"streams": [{"stream": {"service_name":"test"},"values":[["1724169776000000000", "hello", {"x": "foo"}]]}]}' 'http://127.0.0.1:3100/loki/api/v1/push'
localhost ~ # curl -s -G http://127.0.0.1:3100/loki/api/v1/query_range --data-urlencode start=1724169776000000000 --data-urlencode query='{service_name="test"} | label_format x="bar" | x="foo"' | jq '.data.result'
[
  {
    "stream": {
      "service_name": "test",
      "x": "bar"
    },
    "values": [
      [
        "1724169776000000000",
        "hello"
      ]
    ]
  }
]
localhost ~ # curl -s -G http://127.0.0.1:3100/loki/api/v1/query_range --data-urlencode start=1724169776000000000 --data-urlencode query='{service_name="test"} | label_format x="bar" | x="bar"' | jq '.data.result'
[]

[chaudum]

@hvenev Thanks for the bug report its detailed instructions to reproduce the issue!