loki.process --> stage.json, stage.logfmt - extract all label:value pairs without explicit expression

[Nachtfalkeaw]

Request

At the moment in Grafana alloy 1.5.0 and loki.process -> stage.json needs an "expression" and stage.logfmt needs a "mapping".
all these stages make it mandatory to always know all of the label:value pairs which may appear in a log line.

On the other hand in Grafana Explore we have line filter "JSON" and "LOGFMT" and with one click all json formatted parts are correctly listed in label:value.

I would like to see a similar option in loki.process stages:

1.) I would like to be able to configures a specific set of labels I want to extract (as it is now)
2.) I would like to be able to extract all possible labels except the list I define (opposite of now.)
3.) I would like to be able to extract all possible labels.

I would like to see this for JSON and LOGFMT - maybe others are possible, too.

For following stages like "stage.labels" or "stage.struchtured_metadata" I would like to be able to do similar thing:

1.) create struchtured_metadata for all previous extracted labels
2.) create struchtured_metadata for all previous extracted labels except the list I configure
3.) create structured_metadata for a set of previous extracted labels

Use case

This would simplify and shorten the alloy config files. If I want to extract 10 labels of a json logline I have to add 10 stage.json label:value pairs.
Next step is to add 8 out of 10 of these values to stage.structured_metadata.

If I have a stage.match and I want to match the log lines based on the process/service, then I have to do these stages and labels for all services.

In my example I have the following applications:
alloy, mimir, loki, tempo, grafana, prometheus, blackbox_exporter, smokeping_prober, snmp_exporter
All these applications log to /var/log/messages,.

I create a stage.regex and parse the process and the log message which is logfmt.

I create several stage.match for
process=alloy
process=mimir
process=loki
process=tempo
...

I create stage.logfmt for every process to extract always the same parts (ts, msg, level, component, component_id, trace_id, ..) and there are several more fields I do not know because they may only appear in e.g. debug or info level mode.

after I extracted all these 10.20 fields I create stage.structured_metadata for 17 out of 20 labels.

Every stage.match gets it's own service_name label.

If I could use:

stage.json {
    expressions = { ".*" }
}

I would extract everything which is formatted as json in the logline. if new parts appear or new software version adds new field I do not have to modify it.

// This would use als stage.json label_value pairs except "ts" and "msg"
stage.structured_metadata {
    values_exclude = {
      ts  = "",
      msg = "",
    }
}

// This would only use the selected labels "ts" and "msg"
stage.structured_metadata {
    values = {
      ts  = "",
      msg = "",
    }
}

// this will include all stage.json labels in struchtured_metadata
stage.structured_metadata {
    values = {
      .*  = ".*",
    }
}

I don't know if this is usefull for "stage.labels" as we should avoid high labels count - however this may be up to the user.

[Nachtfalkeaw]

maybe somehow related to this or can be included:
grafana/loki#15188

[vladst3f]

related: #926