[loki-canary] fully configurable query

[pgassmann]

Is your feature request related to a problem? Please describe.

loki-canary needs a single label which uniquely identify its logs. This is not always the case if loki-canary is deployed on multiple hosts or namespaces.

Also, the message cannot be packed inside a json (e.g. from Pipeline pack or from vector.dev json encoding.

Describe the solution you'd like

it would be nice if we could configure a custom LogQL query instead of the fixed selectors. (label, stream).

• use combination of labels to uniquely identify the relevant logs.
• allow extracting/unpacking of actual message

Log QL Query {category="dockerlogs", host="logserver1", container_name="loki-canary", source="stdout"} | json | line_format "{{.message}}"

Describe alternatives you've considered

• change setup to not use packed messages with the log line as part of a json structure
• configure unique container_name

Additional context

Example log message format when using vector.dev Loki sink with encoding=json

{
  "container_created_at": "2022-03-29T14:39:48.925912154Z",
  "container_id": "237a4235d3c97da8c2faeec8ea318b6689d363240f368bb9a1cee0d2e4e82beb",
  "container_name": "loki-canary_loki-canary_1",
  "host": "logserver1",
  "image": "grafana/loki-canary:2.4.2",
  "label": {
    "com.docker.compose.config-hash": "c32cc5fcd69a0dc9c632510fdb36096fbf7366df9e09f0d68c466decf5f91608",
    "com.docker.compose.container-number": "1",
    "com.docker.compose.oneoff": "False",
    "com.docker.compose.project": "loki-canary",
    "com.docker.compose.project.config_files": "docker-compose.yml",
    "com.docker.compose.project.working_dir": "/container/loki-canary",
    "com.docker.compose.service": "loki-canary",
    "com.docker.compose.version": "1.29.2",
    "org.opencontainers.image.created": "2022-01-12T16:55:53Z",
    "org.opencontainers.image.revision": "525040a3265715ca8cfb52cd39108ccfac9f8af6",
    "org.opencontainers.image.source": "https://github.com/grafana/loki.git",
    "org.opencontainers.image.url": "https://github.com/grafana/loki"
  },
  "message": "1648564959737014786 ppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp",
  "source_type": "docker",
  "stream": "stdout"
}

Related: #1435 #2259

[stale]

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely
  to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

[pgassmann]

@dannykopping @slim-bean can you review this issue and tell me if it is feasible to implement?

[dannykopping]

@pgassmann apologies for the poor comms here!
Yes, this seems like a worthwhile feature.

It seems to me that we could replace the existing 4 arguments (-label{name,value}, -stream{name,value}) with 3:

• -instant-query (to be used in the call to /loki/api/v1/query)
• -range-query (to be used in the call to /loki/api/v1/query_range)
• -tail-query (to be used in the call to /loki/api/v1/tail)

I can't see much value in keeping each of the existing 4 arguments as individual fields since we hardly use them separately (expect for https://github.com/grafana/loki/blob/main/pkg/canary/reader/reader.go#L419 AFAICS)

What do you think @slim-bean?