Feature Request: Customize swarmStackLabelName and swarmServiceLabelName values

[kinghuang]

Is your feature request related to a problem? Please describe.

The Loki docker-driver has hard-coded labels for Docker Swarm stack and services as swarm_stack and swarm_service, respectively (config.go#L47). I would like to be able to customize these labels to namespace and service, to match corresponding metrics in Prometheus.

Describe the solution you'd like

Provide a way to customize the label name values for swarmServiceLabelName and swarmStackLabelName.

Describe alternatives you've considered

Not sure if there's an equivalent to Prometheus' metric_relabel_configs in Loki.

Additional context

Similar hard-coded labels for Docker Compose projects were added in #970.

[cyriltovena]

Definitively a fair request. I'm thinking about this. I'm considering relabelling but the config is complex to go into log option --log-opts ???.

Using a file has been a nightmare for pipeline stages, so I'm reluctant to that.

Would base 64 encode the config ?

--log-opts relabel_config=ICAtIHNvdXJjZV9sYWJlbHM6IFsnam9iJ10KICAgIHJlZ2V4OiBeKHByb21ldGhldXMtbm9kZS1leHBvcnRlcikuKgogICAgYWN0aW9uOiBrZWVwCiAgLSBzb3VyY2VfbGFiZWxzOiBbX19hZGRyZXNzX19dCiAgICByZWdleDogXiguKik6KC4qKSQKICAgIHRhcmdldF9sYWJlbDogaW5zdGFuY2UKICAgIHJlcGxhY2VtZW50OiAkezF9CiAgLSBzb3VyY2VfbGFiZWxzOiBbX19hZGRyZXNzX19dCiAgICByZWdleDogXiguKik6KC4qKSQKICAgIHRhcmdldF9sYWJlbDogX19hZGRyZXNzX18KICAgIHJlcGxhY2VtZW50OiAkezF9OjkyNzQ=

??
/cc @slim-bean

[kinghuang]

I like the use of the relabel_configs options in your base64 encoded example! Perhaps this should be configured via environment variables during plugin installation. The docker plugin install command takes in [KEY=VALUE...] options for configuring plugins.

$ docker plugin install --help

Usage:	docker plugin install [OPTIONS] PLUGIN [KEY=VALUE...]

$ cat << EOF > relabel_config.yml
- source_labels: ['job']
  regex: ^(prometheus-node-exporter).*
  action: keep
- source_labels: [__address__]
  regex: ^(.*):(.*)$
  target_label: instance
  replacement: ${1}
- source_labels: [__address__]
  regex: ^(.*):(.*)$
  target_label: __address__
  replacement: ${1}:9274
EOF
$ docker plugin install grafana/loki LOKI_RELABEL_CONFIG="$(cat relabel_config.yml)"

[cyriltovena]

I don't know if that example would not explode because of yaml inside quote, this is why I was suggesting base 64.

[kinghuang]

I'm pretty sure that works, because I have Docker services configured that way with big blocks of TOML or YAML passed in via environment variables. If I have some time later this week, I'll try passing something like that into the loki-docker-driver plugin and see what happens.

[cyriltovena]

Alright let's build a POC and you let me know. I'll build an image for you.

[cyriltovena]

Here is the commit: cyriltovena@2de9596

You can try using this plugin: grafana/loki-docker-driver:docker-driver-relabling-2de9596

the log option name is loki-relabel-config and it needs to be as we said yaml list of relabeling config.

Here is an example rewriting swarm_stack to job: cyriltovena@2de9596#diff-cbce68a72af88601836a4b17c3df17afR11

Let me know if it works and I'll add the doc and send a PR if this solves your problem. Or you can even take over my branch if you want to send the PR with the doc update :)

[cyriltovena]

How was it ? @kinghuang any feedback ? If that works for you I'll send a PR.

[kinghuang]

@CyrilPeponnet Sorry, my bad on the huge delay. I'll schedule some time to test this out next week.

[kinghuang]

That works perfectly! I created a new swarm cluster with grafana/loki-docker-driver:docker-driver-relabling-2de9596 installed, and specified a log-opts named loki-relabel-config like so.

As YAML:

log-driver: loki
log-opts:
  loki-url: https://loki.transzap.com/loki/api/v1/push
  loki-external-labels: container_name={{ .Name }},cluster=${name}
  loki-relabel-config: |
    - action: labelmap
      regex: swarm_stack
      replacement: namespace
    - action: labelmap
      regex: swarm_(service)
  max-size: "20m"
  max-file: "5"

Rendered config in daemon.json:

{
  "log-driver": "loki",
  "log-opts": {
    "loki-external-labels": "container_name={{ .Name }},cluster=cloud-test",
    "loki-relabel-config": "- action: labelmap\n  regex: swarm_stack\n  replacement: namespace\n- action: labelmap\n  regex: swarm_(service)\n",
    "loki-url": "https://loki.transzap.com/loki/api/v1/push",
    "max-file": "5",
    "max-size": "20m"
  }
}

Logs arrived in Loki with namespace and service instead of swarm_stack and swarm_service.

▸ ~ http https://loki.transzap.com/loki/api/v1/labels
HTTP/1.1 200 OK
Content-Length: 147
Content-Type: text/plain; charset=utf-8
Date: Fri, 29 May 2020 16:24:46 GMT

{
    "data": [
        "__name__",
        "cluster",
        "container_name",
        "filename",
        "host",
        "namespace",
        "service",
        "source"
    ],
    "status": "success"
}

Sorry again for the huge delay in testing this!

[cyriltovena]

Nice I'll finish this PR soon enough so that it hits master !