Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

promtail high CPU usage on default helm install #1102

Closed
shuttie opened this issue Oct 2, 2019 · 6 comments · Fixed by #1118
Closed

promtail high CPU usage on default helm install #1102

shuttie opened this issue Oct 2, 2019 · 6 comments · Fixed by #1118
Labels
help wanted We would love help on these issues. Please come help us! keepalive An issue or PR that will be kept alive and never marked as stale. type/enhancement Something existing could be improved

Comments

@shuttie
Copy link
Contributor

shuttie commented Oct 2, 2019

Describe the bug
I'm currently playing with promtail 0.3.0 on a medium-sized 50-node kubernetes cluster. And I'm observing a really high CPU usage of promtail pods even on a nodes with low amount of logs generated.

For example, there is a node with 3 pods running, emitting about 10kb of logs per second. Promtail has a constant high cpu usage pattern staying somewhere between 70-150% of CPU. Starting the promtail on this node with server.log_level=debug gives no hints about what this thing is doing: only lines regarding new targets registered once in a while.

Dumping goroutines multiple times shows that most of the running goroutines are spent somewhere in stages.Pipeline.Process. We also have a yet another promtail installation (non-k8s on AWS ElasticBeanstalk, so no helm) with much higher logging throughtput, maybe ~0.5Mb/s, and promtail is almost idle there. This hints me on some sort of performance issue with pipeline and relabeling, as helm chart's promtail config is quite heavy on label transformations.

To Reproduce
Steps to reproduce the behavior:

  1. Started loki 0.3.0
  2. Started promtail 0.3.0 to tail all running pods on the cluster

Expected behavior
Promtail should not constantly consume 100% of single CPU core.

Environment:

Screenshots, promtail config, or terminal output
Promtail logs example (nothing interesting here), with server.log_level=debug:

level=debug ts=2019-10-02T10:04:22.361120061Z caller=filetargetmanager.go:221 msg="dropping target, no labels"
level=debug ts=2019-10-02T10:04:22.361138462Z caller=filetargetmanager.go:203 msg="new target" labels="{__address__=\"172.16.217.55\", __meta_kubernetes_pod_container_init=\"false\", __meta_kubernetes_pod_container_name=\"fp\"}"
level=debug ts=2019-10-02T10:04:22.36143678Z caller=filetargetmanager.go:221 msg="dropping target, no labels"
level=debug ts=2019-10-02T10:04:22.361457241Z caller=filetargetmanager.go:203 msg="new target" labels="{__address__=\"172.16.137.63\", __meta_kubernetes_pod_container_init=\"false\", __meta_kubernetes_pod_container_name=\"fp\"}"
level=debug ts=2019-10-02T10:04:22.361668397Z caller=filetargetmanager.go:221 msg="dropping target, no labels"
level=debug ts=2019-10-02T10:04:22.361689391Z caller=filetargetmanager.go:203 msg="new target" labels="{__address__=\"172.16.106.72\", __meta_kubernetes_pod_container_init=\"false\", __meta_kubernetes_pod_container_name=\"fp\"}"
level=debug ts=2019-10-02T10:04:22.361894125Z caller=filetargetmanager.go:221 msg="dropping target, no labels"
level=debug ts=2019-10-02T10:04:22.362032904Z caller=filetargetmanager.go:203 msg="new target" labels="{__address__=\"172.16.88.212\", __meta_kubernetes_pod_container_init=\"false\", __meta_kubernetes_pod_container_name=\"fp\"}"
level=debug ts=2019-10-02T10:04:22.36224878Z caller=filetargetmanager.go:221 msg="dropping target, no labels"
level=debug ts=2019-10-02T10:04:22.362270489Z caller=filetargetmanager.go:203 msg="new target" labels="{__address__=\"172.16.209.175\", __meta_kubernetes_pod_container_init=\"false\", __meta_kubernetes_pod_container_name=\"fp\"}"
level=debug ts=2019-10-02T10:04:22.362482575Z caller=filetargetmanager.go:221 msg="dropping target, no labels"
level=debug ts=2019-10-02T10:04:22.362502755Z caller=filetargetmanager.go:203 msg="new target" labels="{__address__=\"172.16.82.158:61678\", __meta_kubernetes_pod_container_init=\"false\", __meta_kubernetes_pod_container_name=\"aws-node\", __meta_kubernetes_pod_container_port_name=\"metrics\", __meta_kubernetes_pod_container_port_number=\"61678\", __meta_kubernetes_pod_container_port_protocol=\"TCP\"}"
level=debug ts=2019-10-02T10:04:22.362829621Z caller=filetargetmanager.go:221 msg="dropping target, no labels"
level=debug ts=2019-10-02T10:04:22.362860575Z caller=filetargetmanager.go:203 msg="new target" labels="{__address__=\"172.16.163.154\", __meta_kubernetes_pod_container_init=\"false\", __meta_kubernetes_pod_container_name=\"fp\"}"
level=debug ts=2019-10-02T10:04:22.363072773Z caller=filetargetmanager.go:221 msg="dropping target, no labels"
level=debug ts=2019-10-02T10:04:22.363094928Z caller=filetargetmanager.go:203 msg="new target" labels="{__address__=\"172.16.244.175\", __meta_kubernetes_pod_container_init=\"false\", __meta_kubernetes_pod_container_name=\"fp\"}"
level=debug ts=2019-10-02T10:04:22.363319234Z caller=filetargetmanager.go:221 msg="dropping target, no labels"
level=debug ts=2019-10-02T10:04:22.363464435Z caller=filetargetmanager.go:203 msg="new target" labels="{__address__=\"172.16.207.78\", __meta_kubernetes_pod_container_init=\"false\", __meta_kubernetes_pod_container_name=\"fp\"}"
level=debug ts=2019-10-02T10:04:22.363674391Z caller=filetargetmanager.go:221 msg="dropping target, no labels"
level=debug ts=2019-10-02T10:04:22.363698319Z caller=filetargetmanager.go:203 msg="new target" labels="{__address__=\"172.16.246.17\", __meta_kubernetes_pod_container_init=\"false\", __meta_kubernetes_pod_container_name=\"fp\"}"
level=debug ts=2019-10-02T10:04:22.363923151Z caller=filetargetmanager.go:221 msg="dropping target, no labels"
level=debug ts=2019-10-02T10:04:22.363944101Z caller=filetargetmanager.go:203 msg="new target" labels="{__address__=\"172.16.100.57:8080\", __meta_kubernetes_pod_container_init=\"false\", __meta_kubernetes_pod_container_name=\"lucy\", __meta_kubernetes_pod_container_port_name=\"api\", __meta_kubernetes_pod_container_port_number=\"8080\", __meta_kubernetes_pod_container_port_protocol=\"TCP\"}"
level=debug ts=2019-10-02T10:04:22.364331786Z caller=filetargetmanager.go:221 msg="dropping target, no labels"
level=debug ts=2019-10-02T10:04:22.364504333Z caller=filetargetmanager.go:203 msg="new target" labels="{__address__=\"172.16.100.57:10001\", __meta_kubernetes_pod_container_init=\"false\", __meta_kubernetes_pod_container_name=\"lucy\", __meta_kubernetes_pod_container_port_name=\"javadbg\", __meta_kubernetes_pod_container_port_number=\"10001\", __meta_kubernetes_pod_container_port_protocol=\"TCP\"}"
level=debug ts=2019-10-02T10:04:22.364863355Z caller=pipeline.go:81 component=file_pipeline component=pipeline msg="processing pipeline" stage=0 name=pipeline labels="{app=\"promtail\", container_name=\"promtail\", controller_revision_hash=\"5f9d7bdc6d\", filename=\"/var/log/pods/f7cabb3b-e4f4-11e9-877f-12a04c6dabdc/promtail/7.log\", instance=\"loki-promtail-q8nth\", job=\"logs/promtail\", namespace=\"logs\", pod_template_generation=\"4\", release=\"loki\"}" time=2019-10-02T10:04:22.364834655Z entry="{\"log\":\"level=debug ts=2019-10-02T10:04:20.092914145Z caller=filetargetmanager.go:203 msg=\\\"new target\\\" labels=\\\"{__address__=\\\\\\\"172.16.149.165:8080\\\\\\\", __meta_kubernetes_pod_container_init=\\\\\\\"false\\\\\\\", __meta_kubernetes_pod_container_name=\\\\\\\"lucy\\\\\\\", __meta_kubernetes_pod_container_port_name=\\\\\\\"api\\\\\\\", __meta_kubernetes_pod_container_port_number=\\\\\\\"8080\\\\\\\", __meta_kubernetes_pod_container_port_protocol=\\\\\\\"TCP\\\\\\\"}\\\"\\n\",\"stream\":\"stderr\",\"time\":\"2019-10-02T10:04:20.092963048Z\"}"
level=debug ts=2019-10-02T10:04:22.364915046Z caller=pipeline.go:81 component=file_pipeline component=pipeline msg="processing pipeline" stage=0 name=json labels="{app=\"promtail\", container_name=\"promtail\", controller_revision_hash=\"5f9d7bdc6d\", filename=\"/var/log/pods/f7cabb3b-e4f4-11e9-877f-12a04c6dabdc/promtail/7.log\", instance=\"loki-promtail-q8nth\", job=\"logs/promtail\", namespace=\"logs\", pod_template_generation=\"4\", release=\"loki\"}" time=2019-10-02T10:04:22.364834655Z entry="{\"log\":\"level=debug ts=2019-10-02T10:04:20.092914145Z caller=filetargetmanager.go:203 msg=\\\"new target\\\" labels=\\\"{__address__=\\\\\\\"172.16.149.165:8080\\\\\\\", __meta_kubernetes_pod_container_init=\\\\\\\"false\\\\\\\", __meta_kubernetes_pod_container_name=\\\\\\\"lucy\\\\\\\", __meta_kubernetes_pod_container_port_name=\\\\\\\"api\\\\\\\", __meta_kubernetes_pod_container_port_number=\\\\\\\"8080\\\\\\\", __meta_kubernetes_pod_container_port_protocol=\\\\\\\"TCP\\\\\\\"}\\\"\\n\",\"stream\":\"stderr\",\"time\":\"2019-10-02T10:04:20.092963048Z\"}"
level=debug ts=2019-10-02T10:04:22.365124989Z caller=pipeline.go:81 component=file_pipeline component=pipeline msg="processing pipeline" stage=1 name=labels labels="{app=\"promtail\", container_name=\"promtail\", controller_revision_hash=\"5f9d7bdc6d\", filename=\"/var/log/pods/f7cabb3b-e4f4-11e9-877f-12a04c6dabdc/promtail/7.log\", instance=\"loki-promtail-q8nth\", job=\"logs/promtail\", namespace=\"logs\", pod_template_generation=\"4\", release=\"loki\"}" time=2019-10-02T10:04:22.364834655Z entry="{\"log\":\"level=debug ts=2019-10-02T10:04:20.092914145Z caller=filetargetmanager.go:203 msg=\\\"new target\\\" labels=\\\"{__address__=\\\\\\\"172.16.149.165:8080\\\\\\\", __meta_kubernetes_pod_container_init=\\\\\\\"false\\\\\\\", __meta_kubernetes_pod_container_name=\\\\\\\"lucy\\\\\\\", __meta_kubernetes_pod_container_port_name=\\\\\\\"api\\\\\\\", __meta_kubernetes_pod_container_port_number=\\\\\\\"8080\\\\\\\", __meta_kubernetes_pod_container_port_protocol=\\\\\\\"TCP\\\\\\\"}\\\"\\n\",\"stream\":\"stderr\",\"time\":\"2019-10-02T10:04:20.092963048Z\"}"
level=debug ts=2019-10-02T10:04:22.36516738Z caller=pipeline.go:81 component=file_pipeline component=pipeline msg="processing pipeline" stage=2 name=timestamp labels="{app=\"promtail\", container_name=\"promtail\", controller_revision_hash=\"5f9d7bdc6d\", filename=\"/var/log/pods/f7cabb3b-e4f4-11e9-877f-12a04c6dabdc/promtail/7.log\", instance=\"loki-promtail-q8nth\", job=\"logs/promtail\", namespace=\"logs\", pod_template_generation=\"4\", release=\"loki\", stream=\"stderr\"}" time=2019-10-02T10:04:22.364834655Z entry="{\"log\":\"level=debug ts=2019-10-02T10:04:20.092914145Z caller=filetargetmanager.go:203 msg=\\\"new target\\\" labels=\\\"{__address__=\\\\\\\"172.16.149.165:8080\\\\\\\", __meta_kubernetes_pod_container_init=\\\\\\\"false\\\\\\\", __meta_kubernetes_pod_container_name=\\\\\\\"lucy\\\\\\\", __meta_kubernetes_pod_container_port_name=\\\\\\\"api\\\\\\\", __meta_kubernetes_pod_container_port_number=\\\\\\\"8080\\\\\\\", __meta_kubernetes_pod_container_port_protocol=\\\\\\\"TCP\\\\\\\"}\\\"\\n\",\"stream\":\"stderr\",\"time\":\"2019-10-02T10:04:20.092963048Z\"}"
level=debug ts=2019-10-02T10:04:22.365817007Z caller=pipeline.go:81 component=file_pipeline component=pipeline msg="processing pipeline" stage=3 name=output labels="{app=\"promtail\", container_name=\"promtail\", controller_revision_hash=\"5f9d7bdc6d\", filename=\"/var/log/pods/f7cabb3b-e4f4-11e9-877f-12a04c6dabdc/promtail/7.log\", instance=\"loki-promtail-q8nth\", job=\"logs/promtail\", namespace=\"logs\", pod_template_generation=\"4\", release=\"loki\", stream=\"stderr\"}" time=2019-10-02T10:04:20.092963048Z entry="{\"log\":\"level=debug ts=2019-10-02T10:04:20.092914145Z caller=filetargetmanager.go:203 msg=\\\"new target\\\" labels=\\\"{__address__=\\\\\\\"172.16.149.165:8080\\\\\\\", __meta_kubernetes_pod_container_init=\\\\\\\"false\\\\\\\", __meta_kubernetes_pod_container_name=\\\\\\\"lucy\\\\\\\", __meta_kubernetes_pod_container_port_name=\\\\\\\"api\\\\\\\", __meta_kubernetes_pod_container_port_number=\\\\\\\"8080\\\\\\\", __meta_kubernetes_pod_container_port_protocol=\\\\\\\"TCP\\\\\\\"}\\\"\\n\",\"stream\":\"stderr\",\"time\":\"2019-10-02T10:04:20.092963048Z\"}"
level=debug ts=2019-10-02T10:04:22.365890841Z caller=pipeline.go:85 component=file_pipeline component=pipeline msg="finished processing log line" labels="{app=\"promtail\", container_name=\"promtail\", controller_revision_hash=\"5f9d7bdc6d\", filename=\"/var/log/pods/f7cabb3b-e4f4-11e9-877f-12a04c6dabdc/promtail/7.log\", instance=\"loki-promtail-q8nth\", job=\"logs/promtail\", namespace=\"logs\", pod_template_generation=\"4\", release=\"loki\", stream=\"stderr\"}" time=2019-10-02T10:04:20.092963048Z entry="level=debug ts=2019-10-02T10:04:20.092914145Z caller=filetargetmanager.go:203 msg=\"new target\" labels=\"{__address__=\\\"172.16.149.165:8080\\\", __meta_kubernetes_pod_container_init=\\\"false\\\", __meta_kubernetes_pod_container_name=\\\"lucy\\\", __meta_kubernetes_pod_container_port_name=\\\"api\\\", __meta_kubernetes_pod_container_port_number=\\\"8080\\\", __meta_kubernetes_pod_container_port_protocol=\\\"TCP\\\"}\"\n" duration_s=0.000974756
level=debug ts=2019-10-02T10:04:22.365930177Z caller=pipeline.go:85 component=file_pipeline component=pipeline msg="finished processing log line" labels="{app=\"promtail\", container_name=\"promtail\", controller_revision_hash=\"5f9d7bdc6d\", filename=\"/var/log/pods/f7cabb3b-e4f4-11e9-877f-12a04c6dabdc/promtail/7.log\", instance=\"loki-promtail-q8nth\", job=\"logs/promtail\", namespace=\"logs\", pod_template_generation=\"4\", release=\"loki\", stream=\"stderr\"}" time=2019-10-02T10:04:20.092963048Z entry="level=debug ts=2019-10-02T10:04:20.092914145Z caller=filetargetmanager.go:203 msg=\"new target\" labels=\"{__address__=\\\"172.16.149.165:8080\\\", __meta_kubernetes_pod_container_init=\\\"false\\\", __meta_kubernetes_pod_container_name=\\\"lucy\\\", __meta_kubernetes_pod_container_port_name=\\\"api\\\", __meta_kubernetes_pod_container_port_number=\\\"8080\\\", __meta_kubernetes_pod_container_port_protocol=\\\"TCP\\\"}\"\n" duration_s=0.001075288
level=debug ts=2019-10-02T10:04:22.366106854Z caller=pipeline.go:81 component=file_pipeline component=pipeline msg="processing pipeline" stage=0 name=pipeline labels="{app=\"promtail\", container_name=\"promtail\", controller_revision_hash=\"5f9d7bdc6d\", filename=\"/var/log/pods/f7cabb3b-e4f4-11e9-877f-12a04c6dabdc/promtail/7.log\", instance=\"loki-promtail-q8nth\", job=\"logs/promtail\", namespace=\"logs\", pod_template_generation=\"4\", release=\"loki\"}" time=2019-10-02T10:04:22.364839841Z entry="{\"log\":\"level=debug ts=2019-10-02T10:04:20.092962592Z caller=filetargetmanager.go:221 msg=\\\"dropping target, no labels\\\"\\n\",\"stream\":\"stderr\",\"time\":\"2019-10-02T10:04:20.092991634Z\"}"
level=debug ts=2019-10-02T10:04:22.366159489Z caller=pipeline.go:81 component=file_pipeline component=pipeline msg="processing pipeline" stage=0 name=json labels="{app=\"promtail\", container_name=\"promtail\", controller_revision_hash=\"5f9d7bdc6d\", filename=\"/var/log/pods/f7cabb3b-e4f4-11e9-877f-12a04c6dabdc/promtail/7.log\", instance=\"loki-promtail-q8nth\", job=\"logs/promtail\", namespace=\"logs\", pod_template_generation=\"4\", release=\"loki\"}" time=2019-10-02T10:04:22.364839841Z entry="{\"log\":\"level=debug ts=2019-10-02T10:04:20.092962592Z caller=filetargetmanager.go:221 msg=\\\"dropping target, no labels\\\"\\n\",\"stream\":\"stderr\",\"time\":\"2019-10-02T10:04:20.092991634Z\"}"
level=debug ts=2019-10-02T10:04:22.366242379Z caller=pipeline.go:81 component=file_pipeline component=pipeline msg="processing pipeline" stage=1 name=labels labels="{app=\"promtail\", container_name=\"promtail\", controller_revision_hash=\"5f9d7bdc6d\", filename=\"/var/log/pods/f7cabb3b-e4f4-11e9-877f-12a04c6dabdc/promtail/7.log\", instance=\"loki-promtail-q8nth\", job=\"logs/promtail\", namespace=\"logs\", pod_template_generation=\"4\", release=\"loki\"}" time=2019-10-02T10:04:22.364839841Z entry="{\"log\":\"level=debug ts=2019-10-02T10:04:20.092962592Z caller=filetargetmanager.go:221 msg=\\\"dropping target, no labels\\\"\\n\",\"stream\":\"stderr\",\"time\":\"2019-10-02T10:04:20.092991634Z\"}"
level=debug ts=2019-10-02T10:04:22.366415105Z caller=pipeline.go:81 component=file_pipeline component=pipeline msg="processing pipeline" stage=2 name=timestamp labels="{app=\"promtail\", container_name=\"promtail\", controller_revision_hash=\"5f9d7bdc6d\", filename=\"/var/log/pods/f7cabb3b-e4f4-11e9-877f-12a04c6dabdc/promtail/7.log\", instance=\"loki-promtail-q8nth\", job=\"logs/promtail\", namespace=\"logs\", pod_template_generation=\"4\", release=\"loki\", stream=\"stderr\"}" time=2019-10-02T10:04:22.364839841Z entry="{\"log\":\"level=debug ts=2019-10-02T10:04:20.092962592Z caller=filetargetmanager.go:221 msg=\\\"dropping target, no labels\\\"\\n\",\"stream\":\"stderr\",\"time\":\"2019-10-02T10:04:20.092991634Z\"}"
level=debug ts=2019-10-02T10:04:22.366453Z caller=pipeline.go:81 component=file_pipeline component=pipeline msg="processing pipeline" stage=3 name=output labels="{app=\"promtail\", container_name=\"promtail\", controller_revision_hash=\"5f9d7bdc6d\", filename=\"/var/log/pods/f7cabb3b-e4f4-11e9-877f-12a04c6dabdc/promtail/7.log\", instance=\"loki-promtail-q8nth\", job=\"logs/promtail\", namespace=\"logs\", pod_template_generation=\"4\", release=\"loki\", stream=\"stderr\"}" time=2019-10-02T10:04:20.092991634Z entry="{\"log\":\"level=debug ts=2019-10-02T10:04:20.092962592Z caller=filetargetmanager.go:221 msg=\\\"dropping target, no labels\\\"\\n\",\"stream\":\"stderr\",\"time\":\"2019-10-02T10:04:20.092991634Z\"}"
level=debug ts=2019-10-02T10:04:22.366491562Z caller=pipeline.go:85 component=file_pipeline component=pipeline msg="finished processing log line" labels="{app=\"promtail\", container_name=\"promtail\", controller_revision_hash=\"5f9d7bdc6d\", filename=\"/var/log/pods/f7cabb3b-e4f4-11e9-877f-12a04c6dabdc/promtail/7.log\", instance=\"loki-promtail-q8nth\", job=\"logs/promtail\", namespace=\"logs\", pod_template_generation=\"4\", release=\"loki\", stream=\"stderr\"}" time=2019-10-02T10:04:20.092991634Z entry="level=debug ts=2019-10-02T10:04:20.092962592Z caller=filetargetmanager.go:221 msg=\"dropping target, no labels\"\n" duration_s=0.000331517
level=debug ts=2019-10-02T10:04:22.366646347Z caller=pipeline.go:85 component=file_pipeline component=pipeline msg="finished processing log line" labels="{app=\"promtail\", container_name=\"promtail\", controller_revision_hash=\"5f9d7bdc6d\", filename=\"/var/log/pods/f7cabb3b-e4f4-11e9-877f-12a04c6dabdc/promtail/7.log\", instance=\"loki-promtail-q8nth\", job=\"logs/promtail\", namespace=\"logs\", pod_template_generation=\"4\", release=\"loki\", stream=\"stderr\"}" time=2019-10-02T10:04:20.092991634Z entry="level=debug ts=2019-10-02T10:04:20.092962592Z caller=filetargetmanager.go:221 msg=\"dropping target, no labels\"\n" duration_s=0.000539449
level=debug ts=2019-10-02T10:04:22.368392489Z caller=filetargetmanager.go:221 msg="dropping target, no labels"
level=debug ts=2019-10-02T10:04:22.368503662Z caller=filetargetmanager.go:203 msg="new target" labels="{__address__=\"172.16.100.57:50000\", __meta_kubernetes_pod_container_init=\"false\", __meta_kubernetes_pod_container_name=\"lucy\", __meta_kubernetes_pod_container_port_name=\"metrics\", __meta_kubernetes_pod_container_port_number=\"50000\", __meta_kubernetes_pod_container_port_protocol=\"TCP\"}"

goroutine dump 1:

{"log":"goroutine 210 [runnable]:\n","stream":"stderr","time":"2019-10-02T09:33:17.064543703Z"}
{"log":"github.com/grafana/loki/vendor/github.com/hpcloud/tail.(*Tail).sendLine(0xc0007562c0, 0xc015319a40, 0xb6, 0x0)\n","stream":"stderr","time":"2019-10-02T09:33:17.064546907Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/hpcloud/tail/tail.go:474 +0x179\n","stream":"stderr","time":"2019-10-02T09:33:17.064548811Z"}
{"log":"github.com/grafana/loki/vendor/github.com/hpcloud/tail.(*Tail).tailFileSync(0xc0007562c0)\n","stream":"stderr","time":"2019-10-02T09:33:17.06455058Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/hpcloud/tail/tail.go:309 +0x12a\n","stream":"stderr","time":"2019-10-02T09:33:17.064552344Z"}
{"log":"created by github.com/grafana/loki/vendor/github.com/hpcloud/tail.TailFile\n","stream":"stderr","time":"2019-10-02T09:33:17.0645541Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/hpcloud/tail/tail.go:133 +0x15e\n","stream":"stderr","time":"2019-10-02T09:33:17.064555879Z"}
{"log":"\n","stream":"stderr","time":"2019-10-02T09:33:17.064557623Z"}
{"log":"goroutine 211 [runnable]:\n","stream":"stderr","time":"2019-10-02T09:33:17.064559424Z"}
{"log":"bytes.(*Buffer).WriteString(0xc0047b2930, 0xc01791395d, 0x41, 0x0, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:33:17.064562239Z"}
{"log":"\u0009/usr/local/go/src/bytes/buffer.go:182 +0x115\n","stream":"stderr","time":"2019-10-02T09:33:17.064564979Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt.writeQuotedString(0x2648320, 0xc00c243610, 0xc017913900, 0x125, 0xc013d097e0, 0xc013d097d0, 0xc013d097e0)\n","stream":"stderr","time":"2019-10-02T09:33:17.064567676Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt/jsonstring.go:48 +0xd1\n","stream":"stderr","time":"2019-10-02T09:33:17.064570429Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt.writeStringValue(0x2648320, 0xc00c243610, 0xc017913900, 0x125, 0xc01ff7e901, 0x1, 0x0)\n","stream":"stderr","time":"2019-10-02T09:33:17.064572247Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt/encode.go:244 +0x9c\n","stream":"stderr","time":"2019-10-02T09:33:17.064574069Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt.writeValue(0x2648320, 0xc00c243610, 0x20a69e0, 0xc01ff7e930, 0x1, 0x0)\n","stream":"stderr","time":"2019-10-02T09:33:17.064575903Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt/encode.go:219 +0x531\n","stream":"stderr","time":"2019-10-02T09:33:17.064577719Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt.(*Encoder).EncodeKeyval(0xc00c243600, 0x1d199c0, 0x2639990, 0x20a69e0, 0xc01ff7e930, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:33:17.064579478Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt/encode.go:61 +0x101\n","stream":"stderr","time":"2019-10-02T09:33:17.064581302Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt.(*Encoder).EncodeKeyvals(0xc00c243600, 0xc02577edc0, 0x16, 0x16, 0xc00474c070, 0x449a67)\n","stream":"stderr","time":"2019-10-02T09:33:17.064583101Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt/encode.go:84 +0x12c\n","stream":"stderr","time":"2019-10-02T09:33:17.064584923Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-kit/kit/log.logfmtLogger.Log(0x2648b80, 0xc000749980, 0xc02577edc0, 0x16, 0x16, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:33:17.064586907Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-kit/kit/log/logfmt_logger.go:46 +0xde\n","stream":"stderr","time":"2019-10-02T09:33:17.064588751Z"}
{"log":"github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util.(*PrometheusLogger).Log(0xc000754b00, 0xc02577edc0, 0x16, 0x16, 0xc0233c4180, 0xc)\n","stream":"stderr","time":"2019-10-02T09:33:17.064595695Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util/log.go:84 +0x55\n","stream":"stderr","time":"2019-10-02T09:33:17.064597635Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-kit/kit/log.(*context).Log(0xc01ff7ec30, 0xc0233c4180, 0xc, 0xc, 0xc027b29f60, 0x0)\n","stream":"stderr","time":"2019-10-02T09:33:17.064599464Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-kit/kit/log/log.go:124 +0x1c0\n","stream":"stderr","time":"2019-10-02T09:33:17.064601262Z"}
{"log":"github.com/grafana/loki/pkg/logentry/stages.(*Pipeline).Process(0xc000596940, 0xc01ff7e930, 0xc01ff7e960, 0xc017d89e00, 0xc027b29f60)\n","stream":"stderr","time":"2019-10-02T09:33:17.064603029Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/logentry/stages/pipeline.go:81 +0x32b\n","stream":"stderr","time":"2019-10-02T09:33:17.064604845Z"}
{"log":"github.com/grafana/loki/pkg/logentry/stages.(*Pipeline).Process(0xc000596980, 0xc01ff7e930, 0xc01ff7e960, 0xc017d89e00, 0xc027b29f60)\n","stream":"stderr","time":"2019-10-02T09:33:17.064606596Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/logentry/stages/pipeline.go:82 +0x373\n","stream":"stderr","time":"2019-10-02T09:33:17.064608423Z"}
{"log":"github.com/grafana/loki/pkg/logentry/stages.(*Pipeline).Wrap.func1(0xc01ff7e930, 0xbf5d399743592b94, 0x14ab3c2556, 0x3ecf380, 0xc015319a40, 0xb6, 0x3, 0xffffffffffffffff)\n","stream":"stderr","time":"2019-10-02T09:33:17.064610221Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/logentry/stages/pipeline.go:100 +0x101\n","stream":"stderr","time":"2019-10-02T09:33:17.064612073Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.EntryHandlerFunc.Handle(0xc00041dbc0, 0xc01ff7e930, 0xbf5d399743592b94, 0x14ab3c2556, 0x3ecf380, 0xc015319a40, 0xb6, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:33:17.064613833Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:19 +0x65\n","stream":"stderr","time":"2019-10-02T09:33:17.064615686Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.AddLabelsMiddleware.func1.1(0xc01ff7e900, 0xbf5d399743592b94, 0x14ab3c2556, 0x3ecf380, 0xc015319a40, 0xb6, 0x0, 0xffffffffffffffff)\n","stream":"stderr","time":"2019-10-02T09:33:17.064617471Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:40 +0xa3\n","stream":"stderr","time":"2019-10-02T09:33:17.064619312Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.EntryHandlerFunc.Handle(0xc001883f20, 0xc01ff7e900, 0xbf5d399743592b94, 0x14ab3c2556, 0x3ecf380, 0xc015319a40, 0xb6, 0xc013d09e08, 0x4)\n","stream":"stderr","time":"2019-10-02T09:33:17.06462125Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:19 +0x65\n","stream":"stderr","time":"2019-10-02T09:33:17.064623168Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.AddLabelsMiddleware.func1.1(0xc01ff7e8d0, 0xbf5d399743592b94, 0x14ab3c2556, 0x3ecf380, 0xc015319a40, 0xb6, 0x2179200, 0xc01ff7e8d0)\n","stream":"stderr","time":"2019-10-02T09:33:17.064624932Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:40 +0xa3\n","stream":"stderr","time":"2019-10-02T09:33:17.064626805Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.EntryHandlerFunc.Handle(0xc0018a0240, 0xc01ff7e8d0, 0xbf5d399743592b94, 0x14ab3c2556, 0x3ecf380, 0xc015319a40, 0xb6, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:33:17.064628581Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:19 +0x65\n","stream":"stderr","time":"2019-10-02T09:33:17.064630433Z"}
{"log":"github.com/grafana/loki/pkg/promtail/targets.(*tailer).run(0xc007eece60)\n","stream":"stderr","time":"2019-10-02T09:33:17.064632161Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/targets/tailer.go:106 +0x417\n","stream":"stderr","time":"2019-10-02T09:33:17.064633879Z"}
{"log":"created by github.com/grafana/loki/pkg/promtail/targets.newTailer\n","stream":"stderr","time":"2019-10-02T09:33:17.064638102Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/targets/tailer.go:71 +0x496\n","stream":"stderr","time":"2019-10-02T09:33:17.064640681Z"}
{"log":"\n","stream":"stderr","time":"2019-10-02T09:33:17.064643197Z"}

goroutine dump 2:

{"log":"goroutine 122 [runnable]:\n","stream":"stderr","time":"2019-10-02T09:34:26.783842868Z"}
{"log":"github.com/grafana/loki/pkg/promtail/client.(*client).run(0xc0002e4d80)\n","stream":"stderr","time":"2019-10-02T09:34:26.78384449Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:121 +0x1f4\n","stream":"stderr","time":"2019-10-02T09:34:26.783846236Z"}
{"log":"created by github.com/grafana/loki/pkg/promtail/client.New\n","stream":"stderr","time":"2019-10-02T09:34:26.78384796Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:105 +0x33b\n","stream":"stderr","time":"2019-10-02T09:34:26.783849668Z"}
{"log":"\n","stream":"stderr","time":"2019-10-02T09:34:26.78385142Z"}
{"log":"goroutine 133 [runnable]:\n","stream":"stderr","time":"2019-10-02T09:34:26.783853068Z"}
{"log":"github.com/grafana/loki/vendor/github.com/prometheus/common/model.LabelSet.Merge(0xc00bd4a960, 0xc00bd4a9c0, 0x4)\n","stream":"stderr","time":"2019-10-02T09:34:26.7838551Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/prometheus/common/model/labelset.go:126 +0x1f2\n","stream":"stderr","time":"2019-10-02T09:34:26.783856954Z"}
{"log":"github.com/grafana/loki/pkg/promtail/targets.(*targetSyncer).sync(0xc00072d880, 0xc02672c000, 0x2086, 0x2400)\n","stream":"stderr","time":"2019-10-02T09:34:26.783858698Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/targets/filetargetmanager.go:205 +0x97f\n","stream":"stderr","time":"2019-10-02T09:34:26.783862385Z"}
{"log":"github.com/grafana/loki/pkg/promtail/targets.(*FileTargetManager).run(0xc000711c50)\n","stream":"stderr","time":"2019-10-02T09:34:26.783864158Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/targets/filetargetmanager.go:134 +0xd5\n","stream":"stderr","time":"2019-10-02T09:34:26.783865905Z"}
{"log":"created by github.com/grafana/loki/pkg/promtail/targets.NewFileTargetManager\n","stream":"stderr","time":"2019-10-02T09:34:26.783867671Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/targets/filetargetmanager.go:125 +0x36e\n","stream":"stderr","time":"2019-10-02T09:34:26.783869372Z"}
{"log":"\n","stream":"stderr","time":"2019-10-02T09:34:26.783871116Z"}
{"log":"goroutine 208 [runnable]:\n","stream":"stderr","time":"2019-10-02T09:34:26.784784425Z"}
{"log":"github.com/grafana/loki/vendor/github.com/hpcloud/tail.(*Tail).sendLine(0xc000403ef0, 0xc012741c00, 0x1f3, 0x0)\n","stream":"stderr","time":"2019-10-02T09:34:26.784786687Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/hpcloud/tail/tail.go:474 +0x179\n","stream":"stderr","time":"2019-10-02T09:34:26.784789157Z"}
{"log":"github.com/grafana/loki/vendor/github.com/hpcloud/tail.(*Tail).tailFileSync(0xc000403ef0)\n","stream":"stderr","time":"2019-10-02T09:34:26.784791558Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/hpcloud/tail/tail.go:309 +0x12a\n","stream":"stderr","time":"2019-10-02T09:34:26.78479575Z"}
{"log":"created by github.com/grafana/loki/vendor/github.com/hpcloud/tail.TailFile\n","stream":"stderr","time":"2019-10-02T09:34:26.784798189Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/hpcloud/tail/tail.go:133 +0x15e\n","stream":"stderr","time":"2019-10-02T09:34:26.784800605Z"}
{"log":"\n","stream":"stderr","time":"2019-10-02T09:34:26.784803096Z"}
{"log":"goroutine 209 [runnable]:\n","stream":"stderr","time":"2019-10-02T09:34:26.784805272Z"}
{"log":"bytes.(*Buffer).WriteByte(0xc00ad32a10, 0xc012741c5c, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:34:26.784807455Z"}
{"log":"\u0009/usr/local/go/src/bytes/buffer.go:265 +0xb6\n","stream":"stderr","time":"2019-10-02T09:34:26.784809913Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt.writeQuotedString(0x2648320, 0xc003565f10, 0xc012741c00, 0x1f3, 0x4bfa21, 0x1d199c0, 0xc0032bf820)\n","stream":"stderr","time":"2019-10-02T09:34:26.784812229Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt/jsonstring.go:53 +0x221\n","stream":"stderr","time":"2019-10-02T09:34:26.784814797Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt.writeStringValue(0x2648320, 0xc003565f10, 0xc012741c00, 0x1f3, 0xc009b47701, 0x4bf914, 0x1d199c0)\n","stream":"stderr","time":"2019-10-02T09:34:26.784817273Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt/encode.go:244 +0x9c\n","stream":"stderr","time":"2019-10-02T09:34:26.784819757Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt.writeValue(0x2648320, 0xc003565f10, 0x1d199c0, 0xc0032bfe80, 0xc0032bfe80, 0x198)\n","stream":"stderr","time":"2019-10-02T09:34:26.78482224Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt/encode.go:202 +0x5c6\n","stream":"stderr","time":"2019-10-02T09:34:26.784824867Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt.writeValue(0x2648320, 0xc003565f10, 0x1ca7b00, 0xc0032bf820, 0x1, 0x0)\n","stream":"stderr","time":"2019-10-02T09:34:26.784827504Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt/encode.go:229 +0x3c5\n","stream":"stderr","time":"2019-10-02T09:34:26.784830052Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt.(*Encoder).EncodeKeyval(0xc003565f00, 0x1d199c0, 0x26399b0, 0x1ca7b00, 0xc0032bf820, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:34:26.784832462Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt/encode.go:61 +0x101\n","stream":"stderr","time":"2019-10-02T09:34:26.784834982Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt.(*Encoder).EncodeKeyvals(0xc003565f00, 0xc00de7d8c0, 0x16, 0x16, 0xc00ada20e0, 0x449a67)\n","stream":"stderr","time":"2019-10-02T09:34:26.784837433Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt/encode.go:84 +0x12c\n","stream":"stderr","time":"2019-10-02T09:34:26.784839962Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-kit/kit/log.logfmtLogger.Log(0x2648b80, 0xc000738560, 0xc00de7d8c0, 0x16, 0x16, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:34:26.784842395Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-kit/kit/log/logfmt_logger.go:46 +0xde\n","stream":"stderr","time":"2019-10-02T09:34:26.784844959Z"}
{"log":"github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util.(*PrometheusLogger).Log(0xc00073a4a0, 0xc00de7d8c0, 0x16, 0x16, 0xc00e784fc0, 0xc)\n","stream":"stderr","time":"2019-10-02T09:34:26.784852514Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util/log.go:84 +0x55\n","stream":"stderr","time":"2019-10-02T09:34:26.784856894Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-kit/kit/log.(*context).Log(0xc0150022d0, 0xc00e784fc0, 0xc, 0xc, 0xc0032bf820, 0x0)\n","stream":"stderr","time":"2019-10-02T09:34:26.784859326Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-kit/kit/log/log.go:124 +0x1c0\n","stream":"stderr","time":"2019-10-02T09:34:26.784862149Z"}
{"log":"github.com/grafana/loki/pkg/logentry/stages.(*Pipeline).Process(0xc00076ef80, 0xc015002030, 0xc015002090, 0xc00ac420a0, 0xc0032bf820)\n","stream":"stderr","time":"2019-10-02T09:34:26.784864561Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/logentry/stages/pipeline.go:81 +0x32b\n","stream":"stderr","time":"2019-10-02T09:34:26.784867041Z"}
{"log":"github.com/grafana/loki/pkg/logentry/stages.(*Pipeline).Process(0xc00076efc0, 0xc015002030, 0xc015002090, 0xc00ac420a0, 0xc0032bf820)\n","stream":"stderr","time":"2019-10-02T09:34:26.784869392Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/logentry/stages/pipeline.go:82 +0x373\n","stream":"stderr","time":"2019-10-02T09:34:26.784871932Z"}
{"log":"github.com/grafana/loki/pkg/logentry/stages.(*Pipeline).Wrap.func1(0xc015002030, 0xbf5d39a8ae05e460, 0x9677e1021, 0x3ecf380, 0xc012741c00, 0x1f3, 0x7, 0xffffffffffffffff)\n","stream":"stderr","time":"2019-10-02T09:34:26.784874692Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/logentry/stages/pipeline.go:100 +0x101\n","stream":"stderr","time":"2019-10-02T09:34:26.784877306Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.EntryHandlerFunc.Handle(0xc000739da0, 0xc015002030, 0xbf5d39a8ae05e460, 0x9677e1021, 0x3ecf380, 0xc012741c00, 0x1f3, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:34:26.784879676Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:19 +0x65\n","stream":"stderr","time":"2019-10-02T09:34:26.784882224Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.AddLabelsMiddleware.func1.1(0xc015125fb0, 0xbf5d39a8ae05e460, 0x9677e1021, 0x3ecf380, 0xc012741c00, 0x1f3, 0x0, 0xffffffffffffffff)\n","stream":"stderr","time":"2019-10-02T09:34:26.784884604Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:40 +0xa3\n","stream":"stderr","time":"2019-10-02T09:34:26.784887201Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.EntryHandlerFunc.Handle(0xc0046c0e00, 0xc015125fb0, 0xbf5d39a8ae05e460, 0x9677e1021, 0x3ecf380, 0xc012741c00, 0x1f3, 0xc009b47e08, 0x5)\n","stream":"stderr","time":"2019-10-02T09:34:26.784889558Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:19 +0x65\n","stream":"stderr","time":"2019-10-02T09:34:26.784892181Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.AddLabelsMiddleware.func1.1(0xc015125f20, 0xbf5d39a8ae05e460, 0x9677e1021, 0x3ecf380, 0xc012741c00, 0x1f3, 0x2179200, 0xc015125f20)\n","stream":"stderr","time":"2019-10-02T09:34:26.784894537Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:40 +0xa3\n","stream":"stderr","time":"2019-10-02T09:34:26.784897078Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.EntryHandlerFunc.Handle(0xc0046c1080, 0xc015125f20, 0xbf5d39a8ae05e460, 0x9677e1021, 0x3ecf380, 0xc012741c00, 0x1f3, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:34:26.784899423Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:19 +0x65\n","stream":"stderr","time":"2019-10-02T09:34:26.784901986Z"}
{"log":"github.com/grafana/loki/pkg/promtail/targets.(*tailer).run(0xc0143cf400)\n","stream":"stderr","time":"2019-10-02T09:34:26.784904346Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/targets/tailer.go:106 +0x417\n","stream":"stderr","time":"2019-10-02T09:34:26.784906678Z"}
{"log":"created by github.com/grafana/loki/pkg/promtail/targets.newTailer\n","stream":"stderr","time":"2019-10-02T09:34:26.784908991Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/targets/tailer.go:71 +0x496\n","stream":"stderr","time":"2019-10-02T09:34:26.784913101Z"}
{"log":"\n","stream":"stderr","time":"2019-10-02T09:34:26.784915444Z"}

dump 3:

{"log":"goroutine 134 [runnable]:\n","stream":"stderr","time":"2019-10-02T09:45:00.620471693Z"}
{"log":"github.com/grafana/loki/vendor/github.com/prometheus/prometheus/pkg/labels.(*Labels).Less(0xc00ec007e0, 0xf, 0x17, 0x0)\n","stream":"stderr","time":"2019-10-02T09:45:00.62047466Z"}
{"log":"\u0009\u003cautogenerated\u003e:1 +0x81\n","stream":"stderr","time":"2019-10-02T09:45:00.620477246Z"}
{"log":"sort.doPivot(0x266dd60, 0xc00ec007e0, 0xf, 0x1d, 0xf, 0xf)\n","stream":"stderr","time":"2019-10-02T09:45:00.620479817Z"}
{"log":"\u0009/usr/local/go/src/sort/sort.go:121 +0x149\n","stream":"stderr","time":"2019-10-02T09:45:00.620482152Z"}
{"log":"sort.quickSort(0x266dd60, 0xc00ec007e0, 0xf, 0x1d, 0x9)\n","stream":"stderr","time":"2019-10-02T09:45:00.620484519Z"}
{"log":"\u0009/usr/local/go/src/sort/sort.go:190 +0x9a\n","stream":"stderr","time":"2019-10-02T09:45:00.620487012Z"}
{"log":"sort.quickSort(0x266dd60, 0xc00ec007e0, 0x0, 0x1d, 0x9)\n","stream":"stderr","time":"2019-10-02T09:45:00.620489299Z"}
{"log":"\u0009/usr/local/go/src/sort/sort.go:197 +0x133\n","stream":"stderr","time":"2019-10-02T09:45:00.620491613Z"}
{"log":"sort.Sort(0x266dd60, 0xc00ec007e0)\n","stream":"stderr","time":"2019-10-02T09:45:00.620496596Z"}
{"log":"\u0009/usr/local/go/src/sort/sort.go:218 +0x79\n","stream":"stderr","time":"2019-10-02T09:45:00.620498933Z"}
{"log":"github.com/grafana/loki/vendor/github.com/prometheus/prometheus/pkg/labels.(*Builder).Labels(0xc00ae19848, 0xc00c41ea40, 0xb, 0xc00c41ea58)\n","stream":"stderr","time":"2019-10-02T09:45:00.620501411Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/prometheus/prometheus/pkg/labels/labels.go:346 +0x2ac\n","stream":"stderr","time":"2019-10-02T09:45:00.620504029Z"}
{"log":"github.com/grafana/loki/vendor/github.com/prometheus/prometheus/pkg/relabel.relabel(0xc01ff49500, 0x1c, 0x1c, 0xc0000b3100, 0xc01ff49500, 0x1c, 0x1c)\n","stream":"stderr","time":"2019-10-02T09:45:00.620506494Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/prometheus/prometheus/pkg/relabel/relabel.go:254 +0x402\n","stream":"stderr","time":"2019-10-02T09:45:00.620509176Z"}
{"log":"github.com/grafana/loki/vendor/github.com/prometheus/prometheus/pkg/relabel.Process(0xc01ff49500, 0x1c, 0x1c, 0xc0000a8f00, 0xa, 0xa, 0x749, 0x28, 0x54)\n","stream":"stderr","time":"2019-10-02T09:45:00.620511676Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/prometheus/prometheus/pkg/relabel/relabel.go:185 +0x78\n","stream":"stderr","time":"2019-10-02T09:45:00.620514248Z"}
{"log":"github.com/grafana/loki/pkg/promtail/targets.(*targetSyncer).sync(0xc0003e8f00, 0xc0210ac000, 0x2106, 0x2400)\n","stream":"stderr","time":"2019-10-02T09:45:00.620516776Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/targets/filetargetmanager.go:211 +0x241\n","stream":"stderr","time":"2019-10-02T09:45:00.620519311Z"}
{"log":"github.com/grafana/loki/pkg/promtail/targets.(*FileTargetManager).run(0xc000745140)\n","stream":"stderr","time":"2019-10-02T09:45:00.620529966Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/targets/filetargetmanager.go:134 +0xd5\n","stream":"stderr","time":"2019-10-02T09:45:00.620532519Z"}
{"log":"created by github.com/grafana/loki/pkg/promtail/targets.NewFileTargetManager\n","stream":"stderr","time":"2019-10-02T09:45:00.620535046Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/targets/filetargetmanager.go:125 +0x36e\n","stream":"stderr","time":"2019-10-02T09:45:00.62053744Z"}
{"log":"\n","stream":"stderr","time":"2019-10-02T09:45:00.620539916Z"}
{"log":"goroutine 170 [runnable]:\n","stream":"stderr","time":"2019-10-02T09:45:00.621530948Z"}
{"log":"github.com/grafana/loki/vendor/github.com/hpcloud/tail.(*Tail).sendLine(0xc0001c86e0, 0xc00b1e4cc0, 0xb6, 0x0)\n","stream":"stderr","time":"2019-10-02T09:45:00.621534516Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/hpcloud/tail/tail.go:474 +0x179\n","stream":"stderr","time":"2019-10-02T09:45:00.62153729Z"}
{"log":"github.com/grafana/loki/vendor/github.com/hpcloud/tail.(*Tail).tailFileSync(0xc0001c86e0)\n","stream":"stderr","time":"2019-10-02T09:45:00.621539861Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/hpcloud/tail/tail.go:309 +0x12a\n","stream":"stderr","time":"2019-10-02T09:45:00.621542282Z"}
{"log":"created by github.com/grafana/loki/vendor/github.com/hpcloud/tail.TailFile\n","stream":"stderr","time":"2019-10-02T09:45:00.621545026Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/hpcloud/tail/tail.go:133 +0x15e\n","stream":"stderr","time":"2019-10-02T09:45:00.621547626Z"}
{"log":"\n","stream":"stderr","time":"2019-10-02T09:45:00.621550156Z"}
{"log":"goroutine 171 [runnable]:\n","stream":"stderr","time":"2019-10-02T09:45:00.621552458Z"}
{"log":"syscall.Syscall(0x1, 0x2, 0xc0007a1000, 0x2d3, 0x2d3, 0x2d3, 0x0)\n","stream":"stderr","time":"2019-10-02T09:45:00.621569746Z"}
{"log":"\u0009/usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5\n","stream":"stderr","time":"2019-10-02T09:45:00.621572117Z"}
{"log":"syscall.write(0x2, 0xc0007a1000, 0x2d3, 0x2d89, 0xc007fb7f01, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:45:00.621574599Z"}
{"log":"\u0009/usr/local/go/src/syscall/zsyscall_linux_amd64.go:1005 +0x5a\n","stream":"stderr","time":"2019-10-02T09:45:00.621576923Z"}
{"log":"syscall.Write(0x2, 0xc0007a1000, 0x2d3, 0x2d89, 0x0, 0x0, 0x1d199c0)\n","stream":"stderr","time":"2019-10-02T09:45:00.621579289Z"}
{"log":"\u0009/usr/local/go/src/syscall/syscall_unix.go:191 +0x49\n","stream":"stderr","time":"2019-10-02T09:45:00.621581522Z"}
{"log":"internal/poll.(*FD).Write(0xc000084120, 0xc0007a1000, 0x2d3, 0x2d89, 0x0, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:45:00.621583951Z"}
{"log":"\u0009/usr/local/go/src/internal/poll/fd_unix.go:268 +0x165\n","stream":"stderr","time":"2019-10-02T09:45:00.621586325Z"}
{"log":"os.(*File).write(0xc00000e020, 0xc0007a1000, 0x2d3, 0x2d89, 0xd0, 0x532b, 0xd0)\n","stream":"stderr","time":"2019-10-02T09:45:00.621588579Z"}
{"log":"\u0009/usr/local/go/src/os/file_unix.go:266 +0x4e\n","stream":"stderr","time":"2019-10-02T09:45:00.621591112Z"}
{"log":"os.(*File).Write(0xc00000e020, 0xc0007a1000, 0x2d3, 0x2d89, 0x3e6aa37, 0x1, 0x1)\n","stream":"stderr","time":"2019-10-02T09:45:00.621593617Z"}
{"log":"\u0009/usr/local/go/src/os/file.go:145 +0x6f\n","stream":"stderr","time":"2019-10-02T09:45:00.621596222Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-kit/kit/log.(*fdSyncWriter).Write(0xc00025ab40, 0xc0007a1000, 0x2d3, 0x2d89, 0x0, 0x0, 0xc00509b960)\n","stream":"stderr","time":"2019-10-02T09:45:00.621598726Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-kit/kit/log/sync.go:90 +0x66\n","stream":"stderr","time":"2019-10-02T09:45:00.62160365Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-kit/kit/log.logfmtLogger.Log(0x2648b80, 0xc00025ab40, 0xc01a46cc60, 0x16, 0x16, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:45:00.621606532Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-kit/kit/log/logfmt_logger.go:58 +0x16c\n","stream":"stderr","time":"2019-10-02T09:45:00.621609418Z"}
{"log":"github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util.(*PrometheusLogger).Log(0xc000047130, 0xc01a46cc60, 0x16, 0x16, 0xc01cd5f200, 0xc)\n","stream":"stderr","time":"2019-10-02T09:45:00.621618162Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util/log.go:84 +0x55\n","stream":"stderr","time":"2019-10-02T09:45:00.62162105Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-kit/kit/log.(*context).Log(0xc015209890, 0xc01cd5f200, 0xc, 0xc, 0xc01dfe0800, 0x0)\n","stream":"stderr","time":"2019-10-02T09:45:00.621623844Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-kit/kit/log/log.go:124 +0x1c0\n","stream":"stderr","time":"2019-10-02T09:45:00.621626349Z"}
{"log":"github.com/grafana/loki/pkg/logentry/stages.(*Pipeline).Process(0xc000081a00, 0xc015209560, 0xc015209590, 0xc00096dd60, 0xc01dfe0800)\n","stream":"stderr","time":"2019-10-02T09:45:00.621628629Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/logentry/stages/pipeline.go:81 +0x32b\n","stream":"stderr","time":"2019-10-02T09:45:00.621641787Z"}
{"log":"github.com/grafana/loki/pkg/logentry/stages.(*Pipeline).Process(0xc000081ac0, 0xc015209560, 0xc015209590, 0xc00096dd60, 0xc01dfe0800)\n","stream":"stderr","time":"2019-10-02T09:45:00.621644739Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/logentry/stages/pipeline.go:82 +0x373\n","stream":"stderr","time":"2019-10-02T09:45:00.621647428Z"}
{"log":"github.com/grafana/loki/pkg/logentry/stages.(*Pipeline).Wrap.func1(0xc015209560, 0xbf5d3a4724835671, 0x897211df76, 0x3ecf380, 0xc00b1e4cc0, 0xb6, 0x4, 0xffffffffffffffff)\n","stream":"stderr","time":"2019-10-02T09:45:00.621649924Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/logentry/stages/pipeline.go:100 +0x101\n","stream":"stderr","time":"2019-10-02T09:45:00.62165252Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.EntryHandlerFunc.Handle(0xc0003f6380, 0xc015209560, 0xbf5d3a4724835671, 0x897211df76, 0x3ecf380, 0xc00b1e4cc0, 0xb6, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:45:00.621655025Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:19 +0x65\n","stream":"stderr","time":"2019-10-02T09:45:00.621657663Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.AddLabelsMiddleware.func1.1(0xc015209500, 0xbf5d3a4724835671, 0x897211df76, 0x3ecf380, 0xc00b1e4cc0, 0xb6, 0x0, 0xffffffffffffffff)\n","stream":"stderr","time":"2019-10-02T09:45:00.621660028Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:40 +0xa3\n","stream":"stderr","time":"2019-10-02T09:45:00.621662662Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.EntryHandlerFunc.Handle(0xc002b9bce0, 0xc015209500, 0xbf5d3a4724835671, 0x897211df76, 0x3ecf380, 0xc00b1e4cc0, 0xb6, 0xc00509be08, 0x4)\n","stream":"stderr","time":"2019-10-02T09:45:00.621665216Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:19 +0x65\n","stream":"stderr","time":"2019-10-02T09:45:00.621667766Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.AddLabelsMiddleware.func1.1(0xc0152094a0, 0xbf5d3a4724835671, 0x897211df76, 0x3ecf380, 0xc00b1e4cc0, 0xb6, 0x2179200, 0xc0152094a0)\n","stream":"stderr","time":"2019-10-02T09:45:00.621670169Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:40 +0xa3\n","stream":"stderr","time":"2019-10-02T09:45:00.621674892Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.EntryHandlerFunc.Handle(0xc002b9be20, 0xc0152094a0, 0xbf5d3a4724835671, 0x897211df76, 0x3ecf380, 0xc00b1e4cc0, 0xb6, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:45:00.621677752Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:19 +0x65\n","stream":"stderr","time":"2019-10-02T09:45:00.621680442Z"}
{"log":"github.com/grafana/loki/pkg/promtail/targets.(*tailer).run(0xc005facc30)\n","stream":"stderr","time":"2019-10-02T09:45:00.621683136Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/targets/tailer.go:106 +0x417\n","stream":"stderr","time":"2019-10-02T09:45:00.621685941Z"}
{"log":"created by github.com/grafana/loki/pkg/promtail/targets.newTailer\n","stream":"stderr","time":"2019-10-02T09:45:00.621688615Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/targets/tailer.go:71 +0x496\n","stream":"stderr","time":"2019-10-02T09:45:00.621691041Z"}
{"log":"\n","stream":"stderr","time":"2019-10-02T09:45:00.621693429Z"}

dump 4:

{"log":"goroutine 215 [runnable]:\n","stream":"stderr","time":"2019-10-02T09:46:45.84543918Z"}
{"log":"runtime.Caller(0x3, 0xc00bb6d8f0, 0xc012b959e8, 0x548e8d, 0x1fe7900, 0xc012b959c0)\n","stream":"stderr","time":"2019-10-02T09:46:45.845441365Z"}
{"log":"\u0009/usr/local/go/src/runtime/extern.go:191 +0x171\n","stream":"stderr","time":"2019-10-02T09:46:45.845443725Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-kit/kit/log.Caller.func1(0x1fe7900, 0xc00bb6d8f0)\n","stream":"stderr","time":"2019-10-02T09:46:45.845445995Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-kit/kit/log/value.go:86 +0x31\n","stream":"stderr","time":"2019-10-02T09:46:45.845448304Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-kit/kit/log.bindValues(0xc00483e780, 0xa, 0x14)\n","stream":"stderr","time":"2019-10-02T09:46:45.84545241Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-kit/kit/log/value.go:20 +0x7f\n","stream":"stderr","time":"2019-10-02T09:46:45.845454807Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-kit/kit/log.(*context).Log(0xc00bb6d8c0, 0xc001a172c0, 0xa, 0xa, 0xc017654db0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:46:45.845457559Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-kit/kit/log/log.go:122 +0x189\n","stream":"stderr","time":"2019-10-02T09:46:45.845460026Z"}
{"log":"github.com/grafana/loki/pkg/logentry/stages.(*Pipeline).Process(0xc000080d80, 0xc00bb6d530, 0xc00bb6d590, 0xc02195fa20, 0xc017654db0)\n","stream":"stderr","time":"2019-10-02T09:46:45.84546241Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/logentry/stages/pipeline.go:85 +0x5ea\n","stream":"stderr","time":"2019-10-02T09:46:45.845473932Z"}
{"log":"github.com/grafana/loki/pkg/logentry/stages.(*Pipeline).Process(0xc000080e00, 0xc00bb6d530, 0xc00bb6d590, 0xc02195fa20, 0xc017654db0)\n","stream":"stderr","time":"2019-10-02T09:46:45.845476375Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/logentry/stages/pipeline.go:82 +0x373\n","stream":"stderr","time":"2019-10-02T09:46:45.845478857Z"}
{"log":"github.com/grafana/loki/pkg/logentry/stages.(*Pipeline).Wrap.func1(0xc00bb6d530, 0xbf5d3a6171b1f871, 0x158886b37b, 0x3ecf380, 0xc0000c1340, 0x143, 0x4, 0xffffffffffffffff)\n","stream":"stderr","time":"2019-10-02T09:46:45.845481192Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/logentry/stages/pipeline.go:100 +0x101\n","stream":"stderr","time":"2019-10-02T09:46:45.845483736Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.EntryHandlerFunc.Handle(0xc0002763a0, 0xc00bb6d530, 0xbf5d3a6171b1f871, 0x158886b37b, 0x3ecf380, 0xc0000c1340, 0x143, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:46:45.845486071Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:19 +0x65\n","stream":"stderr","time":"2019-10-02T09:46:45.84548869Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.AddLabelsMiddleware.func1.1(0xc00bb6d4a0, 0xbf5d3a6171b1f871, 0x158886b37b, 0x3ecf380, 0xc0000c1340, 0x143, 0x0, 0xffffffffffffffff)\n","stream":"stderr","time":"2019-10-02T09:46:45.845491096Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:40 +0xa3\n","stream":"stderr","time":"2019-10-02T09:46:45.845493846Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.EntryHandlerFunc.Handle(0xc00259b260, 0xc00bb6d4a0, 0xbf5d3a6171b1f871, 0x158886b37b, 0x3ecf380, 0xc0000c1340, 0x143, 0xc012b95e08, 0x5)\n","stream":"stderr","time":"2019-10-02T09:46:45.845496214Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:19 +0x65\n","stream":"stderr","time":"2019-10-02T09:46:45.845498798Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.AddLabelsMiddleware.func1.1(0xc00bb6d470, 0xbf5d3a6171b1f871, 0x158886b37b, 0x3ecf380, 0xc0000c1340, 0x143, 0x2179200, 0xc00bb6d470)\n","stream":"stderr","time":"2019-10-02T09:46:45.845501175Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:40 +0xa3\n","stream":"stderr","time":"2019-10-02T09:46:45.845503745Z"}
{"log":"github.com/grafana/loki/pkg/promtail/api.EntryHandlerFunc.Handle(0xc00259b440, 0xc00bb6d470, 0xbf5d3a6171b1f871, 0x158886b37b, 0x3ecf380, 0xc0000c1340, 0x143, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:46:45.845506056Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/api/types.go:19 +0x65\n","stream":"stderr","time":"2019-10-02T09:46:45.845508688Z"}
{"log":"github.com/grafana/loki/pkg/promtail/targets.(*tailer).run(0xc003f13450)\n","stream":"stderr","time":"2019-10-02T09:46:45.845511607Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/targets/tailer.go:106 +0x417\n","stream":"stderr","time":"2019-10-02T09:46:45.845514033Z"}
{"log":"created by github.com/grafana/loki/pkg/promtail/targets.newTailer\n","stream":"stderr","time":"2019-10-02T09:46:45.845516362Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/targets/tailer.go:71 +0x496\n","stream":"stderr","time":"2019-10-02T09:46:45.845520495Z"}
{"log":"\n","stream":"stderr","time":"2019-10-02T09:46:45.845522856Z"}

dump 5:

{"log":"goroutine 143 [runnable]:\n","stream":"stderr","time":"2019-10-02T09:51:11.317438602Z"}
{"log":"reflect.Value.String(0x1e29cc0, 0xc0125f6350, 0x98, 0x71, 0x5)\n","stream":"stderr","time":"2019-10-02T09:51:11.31744024Z"}
{"log":"\u0009/usr/local/go/src/reflect/value.go:1706 +0x120\n","stream":"stderr","time":"2019-10-02T09:51:11.317441953Z"}
{"log":"fmt.(*pp).printValue(0xc018d98f00, 0x1e29cc0, 0xc0125f6350, 0x98, 0x73, 0x0)\n","stream":"stderr","time":"2019-10-02T09:51:11.317443739Z"}
{"log":"\u0009/usr/local/go/src/fmt/print.go:734 +0x23e7\n","stream":"stderr","time":"2019-10-02T09:51:11.317445501Z"}
{"log":"fmt.(*pp).printArg(0xc018d98f00, 0x1e29cc0, 0xc0125f6350, 0x73)\n","stream":"stderr","time":"2019-10-02T09:51:11.317447203Z"}
{"log":"\u0009/usr/local/go/src/fmt/print.go:689 +0x2b7\n","stream":"stderr","time":"2019-10-02T09:51:11.317448922Z"}
{"log":"fmt.(*pp).doPrintf(0xc018d98f00, 0x223f382, 0x5, 0xc024a3d528, 0x2, 0x2)\n","stream":"stderr","time":"2019-10-02T09:51:11.317450612Z"}
{"log":"\u0009/usr/local/go/src/fmt/print.go:1003 +0x166\n","stream":"stderr","time":"2019-10-02T09:51:11.317452339Z"}
{"log":"fmt.Sprintf(0x223f382, 0x5, 0xc024a3d528, 0x2, 0x2, 0x3, 0xc02bf8f400)\n","stream":"stderr","time":"2019-10-02T09:51:11.317454388Z"}
{"log":"\u0009/usr/local/go/src/fmt/print.go:203 +0x66\n","stream":"stderr","time":"2019-10-02T09:51:11.317456189Z"}
{"log":"github.com/grafana/loki/vendor/github.com/prometheus/common/model.LabelSet.String(0xc00ccee9c0, 0x249b258, 0x7f08f866b458)\n","stream":"stderr","time":"2019-10-02T09:51:11.317457912Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/prometheus/common/model/labelset.go:135 +0x205\n","stream":"stderr","time":"2019-10-02T09:51:11.317459704Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt.safeString(0x7f08f866b458, 0xc00ccee9c0, 0x0, 0x0, 0xc00ccee900)\n","stream":"stderr","time":"2019-10-02T09:51:11.317461446Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt/encode.go:300 +0x86\n","stream":"stderr","time":"2019-10-02T09:51:11.317463236Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt.writeValue(0x2648320, 0xc00a1cb310, 0x20a69e0, 0xc00ccee9c0, 0x1, 0x0)\n","stream":"stderr","time":"2019-10-02T09:51:11.317465042Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt/encode.go:218 +0x519\n","stream":"stderr","time":"2019-10-02T09:51:11.317466851Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt.(*Encoder).EncodeKeyval(0xc00a1cb300, 0x1d199c0, 0x263a190, 0x20a69e0, 0xc00ccee9c0, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:51:11.317470475Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt/encode.go:61 +0x101\n","stream":"stderr","time":"2019-10-02T09:51:11.317472469Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt.(*Encoder).EncodeKeyvals(0xc00a1cb300, 0xc022dee540, 0xa, 0xc, 0xc00387a2a0, 0x449a67)\n","stream":"stderr","time":"2019-10-02T09:51:11.31747456Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-logfmt/logfmt/encode.go:84 +0x12c\n","stream":"stderr","time":"2019-10-02T09:51:11.317477468Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-kit/kit/log.logfmtLogger.Log(0x2648b80, 0xc00000c980, 0xc022dee540, 0xa, 0xc, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:51:11.317480202Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-kit/kit/log/logfmt_logger.go:46 +0xde\n","stream":"stderr","time":"2019-10-02T09:51:11.317482969Z"}
{"log":"github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util.(*PrometheusLogger).Log(0xc000419010, 0xc022dee540, 0xa, 0xc, 0xc00b7839c0, 0x4)\n","stream":"stderr","time":"2019-10-02T09:51:11.317489479Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util/log.go:84 +0x55\n","stream":"stderr","time":"2019-10-02T09:51:11.317491347Z"}
{"log":"github.com/grafana/loki/vendor/github.com/go-kit/kit/log.(*context).Log(0xc012a53500, 0xc00b7839c0, 0x4, 0x4, 0x267a820, 0xc00a467a80)\n","stream":"stderr","time":"2019-10-02T09:51:11.317493525Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/go-kit/kit/log/log.go:124 +0x1c0\n","stream":"stderr","time":"2019-10-02T09:51:11.31749544Z"}
{"log":"github.com/grafana/loki/pkg/promtail/targets.(*targetSyncer).sync(0xc0000cd300, 0xc02bb70000, 0x20a1, 0x2400)\n","stream":"stderr","time":"2019-10-02T09:51:11.317497247Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/targets/filetargetmanager.go:203 +0x95d\n","stream":"stderr","time":"2019-10-02T09:51:11.317499016Z"}
{"log":"github.com/grafana/loki/pkg/promtail/targets.(*FileTargetManager).run(0xc000082270)\n","stream":"stderr","time":"2019-10-02T09:51:11.317500773Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/targets/filetargetmanager.go:134 +0xd5\n","stream":"stderr","time":"2019-10-02T09:51:11.317502523Z"}
{"log":"created by github.com/grafana/loki/pkg/promtail/targets.NewFileTargetManager\n","stream":"stderr","time":"2019-10-02T09:51:11.317504282Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/pkg/promtail/targets/filetargetmanager.go:125 +0x36e\n","stream":"stderr","time":"2019-10-02T09:51:11.317506025Z"}
{"log":"\n","stream":"stderr","time":"2019-10-02T09:51:11.317507785Z"}
{"log":"goroutine 189 [runnable]:\n","stream":"stderr","time":"2019-10-02T09:51:11.318292929Z"}
{"log":"syscall.Seek(0x11, 0x0, 0x1, 0xc0007ad1b8, 0xc00e2a8ad8, 0xc0053badd8)\n","stream":"stderr","time":"2019-10-02T09:51:11.318294597Z"}
{"log":"\u0009/usr/local/go/src/syscall/zsyscall_linux_amd64.go:1338 +0xf0\n","stream":"stderr","time":"2019-10-02T09:51:11.318296331Z"}
{"log":"internal/poll.(*FD).Seek(0xc00a946720, 0x0, 0x1, 0x0, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:51:11.318298049Z"}
{"log":"\u0009/usr/local/go/src/internal/poll/fd_unix.go:404 +0x9f\n","stream":"stderr","time":"2019-10-02T09:51:11.318299735Z"}
{"log":"os.(*File).seek(0xc0049b0258, 0x0, 0x1, 0x3ecf380, 0xc02a8df800, 0x1f4)\n","stream":"stderr","time":"2019-10-02T09:51:11.318301425Z"}
{"log":"\u0009/usr/local/go/src/os/file_unix.go:284 +0x44\n","stream":"stderr","time":"2019-10-02T09:51:11.318303167Z"}
{"log":"os.(*File).Seek(0xc0049b0258, 0x0, 0x1, 0xc0053baf60, 0x0, 0x1b41b00)\n","stream":"stderr","time":"2019-10-02T09:51:11.318304872Z"}
{"log":"\u0009/usr/local/go/src/os/file.go:196 +0x59\n","stream":"stderr","time":"2019-10-02T09:51:11.318306563Z"}
{"log":"github.com/grafana/loki/vendor/github.com/hpcloud/tail.(*Tail).Tell(0xc0001bec60, 0x0, 0x0, 0x0)\n","stream":"stderr","time":"2019-10-02T09:51:11.318308253Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/hpcloud/tail/tail.go:146 +0x5f\n","stream":"stderr","time":"2019-10-02T09:51:11.318310016Z"}
{"log":"github.com/grafana/loki/vendor/github.com/hpcloud/tail.(*Tail).tailFileSync(0xc0001bec60)\n","stream":"stderr","time":"2019-10-02T09:51:11.318312437Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/hpcloud/tail/tail.go:298 +0x7d1\n","stream":"stderr","time":"2019-10-02T09:51:11.318314877Z"}
{"log":"created by github.com/grafana/loki/vendor/github.com/hpcloud/tail.TailFile\n","stream":"stderr","time":"2019-10-02T09:51:11.318317403Z"}
{"log":"\u0009/go/src/github.com/grafana/loki/vendor/github.com/hpcloud/tail/tail.go:133 +0x15e\n","stream":"stderr","time":"2019-10-02T09:51:11.318319915Z"}
{"log":"\n","stream":"stderr","time":"2019-10-02T09:51:11.318322487Z"}
@shuttie
Copy link
Contributor Author

shuttie commented Oct 2, 2019 •
edited
Loading

Generated profile and trace files for promtail:master-c4913bf
Profile: pprof002.svg.gz
Trace: trace.gz

So these CPU spikes are actually GC activity. I've tried to correlate these GC spikes with logs and looks like it's happening exactly when kubernetes discovery kicks in. We have quite a lot of pods in the cluster (total amount is about 8000, but only 2000 is running, others are in a completed state after a CronJob), so the discovery pulls all the information about all the pods in the cluster, and then discards almost all of it.

I have validated this idea by limiting the namespaces to monitor. Removing the one with most of the pods solved the issue (but left me without logs for the crowded namespace).

@shuttie
Copy link
Contributor Author

shuttie commented Oct 2, 2019 •
edited
Loading

As a suggestion about how it can be fixed, I have following ideas:

  • have an option about frequency of kubernetes discovery kicking in. According to trace, currently it's happening every 5 seconds. If it will happen only once in a minute, it won't be that painful.
  • while pulling pod information from kubernetes, either filter pods belonging to current node on a k8s api level (most probably it's a bit complicated, as kubernetes discovery is reused from prometheus, so it will require an upstream fix), or discard it as early as possible, before trying to pass it through the whole processing pipeline.

@cyriltovena
Copy link
Contributor

@shuttie Thank you so much for all of this !

@cyriltovena cyriltovena added keepalive An issue or PR that will be kept alive and never marked as stale. type/enhancement Something existing could be improved labels Oct 2, 2019
@chancez
Copy link
Contributor

chancez commented Oct 2, 2019

I was following along on slack and had some thoughts on this.

Kubelet has many of these same problems, and there's been a lot of work to optimize it if I recall, but I'd suggest looking at how kubelet handles this.

I believe field selectors do support node name when listing pods, but I believe the filtering on that is still happening in-memory on the API server, so I'm not sure how good that is, as it only reduces data transmitted, but could still be expensive for the API server, but it's also probably worth investigating.

Since the metadata necessarily is is pretty limited, it might be available by querying the container runtime or parsing files for necessary info (namespace/pod-name) then only querying the single objects necessary (rather than doing a list/watch + filtering). This would probably mean more queries, but less overall information being fetched and then filtered.

@chancez
Copy link
Contributor

chancez commented Oct 2, 2019

I back tracked the kubelet code to see where it sets up it's watchers and finally found it at: https://github.com/kubernetes/kubernetes/blob/67d928acdc351d3d4f9e20a92cee4452a26ed0c4////pkg/kubelet/config/apiserver.go#L33

It seems like it just uses a filtered watch using a field selector on the pod's spec.nodeName field matching the kubelet's nodeName. Seems like a relatively straight-forward change. In Upstream Prometheus that seems to be something that would be done here: https://github.com/prometheus/prometheus/blob/0fbc8c13a71d18c0d6c50b11d33f339954451d46/discovery/kubernetes/kubernetes.go#L271-L276 though I don't know what the best method of handling this change would be since it needs to be dealt with in upstream Prometheus.

@cyriltovena
Copy link
Contributor

I think the best option is to open an issue on the Prometheus repo referencing this issue just to see if it would be acceptable.

This was referenced Oct 3, 2019
Closed
Merged
@Fabian-K Fabian-K mentioned this issue Nov 25, 2019
Closed
@slim-bean slim-bean added the help wanted We would love help on these issues. Please come help us! label Jun 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted We would love help on these issues. Please come help us! keepalive An issue or PR that will be kept alive and never marked as stale. type/enhancement Something existing could be improved
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

promtail: fix high CPU usage on large kubernetes clusters. shuttie/loki
4 participants
@shuttie @cyriltovena @chancez @slim-bean