Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

panic: rare nil metric influx output #5514

Closed
goller opened this issue Mar 1, 2019 · 0 comments · Fixed by #5524
Closed

panic: rare nil metric influx output #5514

goller opened this issue Mar 1, 2019 · 0 comments · Fixed by #5524
Assignees
Labels
bug unexpected problem or unintended behavior panic issue that results in panics from Telegraf regression something that used to work, but is now broken
Milestone

Comments

@goller
Copy link
Contributor

goller commented Mar 1, 2019

Relevant telegraf.conf:

    [[outputs.influxdb]]
      urls = ["$MONITOR_HOST"]
      database = "$MONITOR_DATABASE"
      retention_policy = "$MONITOR_RP"
      write_consistency = "any"
      timeout = "5s"
      username = "$MONITOR_USERNAME"
      password = "$MONITOR_PASSWORD"
    [[outputs.influxdb_v2]]
      urls = ["$MONITOR_HOST_V2"]
      token = "$MONITOR_TOKEN"
      organization = "$MONITOR_DATABASE"
      bucket = "$MONITOR_RP"
      timeout = "5s"
    [global_tags]
      env = "$ENV"
      hostname = "$HOSTNAME"
      nodename = "$NODENAME"
      role = "gateway"
    [[inputs.prometheus]]
      urls = ["http://127.0.0.1:9999/metrics"]

System info:

Telegraf Version: 1.9.5
OS: Linux

Steps to reproduce:

Extremely rare. Monitored processes are under heavy load with many metrics, yet, this seems to have happened only once.

Actual behavior:

Panic:

2019-03-01T20:01:45Z E! [agent] Error writing to output [influxdb_v2]: could not write any address
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x90 pc=0x86219a]
goroutine 4923 [running]:
github.com/influxdata/telegraf/plugins/serializers/influx.(*Serializer).buildHeader(0xc0008f2000, 0x0, 0x0, 0xc000a9e180, 0x300000002)
    /go/src/github.com/influxdata/telegraf/plugins/serializers/influx/influx.go:143 +0x4a
github.com/influxdata/telegraf/plugins/serializers/influx.(*Serializer).writeMetric(0xc0008f2000, 0x2500e80, 0xc002cf33b0, 0x0, 0x0, 0x7f360c6d9da8, 0x1)
    /go/src/github.com/influxdata/telegraf/plugins/serializers/influx/influx.go:200 +0x5a
github.com/influxdata/telegraf/plugins/serializers/influx.(*Serializer).Write(0xc0008f2000, 0x2500e80, 0xc002cf33b0, 0x0, 0x0, 0x7f3623295d80, 0x0, 0x4301a3)
    /go/src/github.com/influxdata/telegraf/plugins/serializers/influx/influx.go:124 +0x53
github.com/influxdata/telegraf/plugins/serializers/influx.(*reader).Read(0xc000766090, 0xc00116a000, 0x8000, 0x8000, 0x8000, 0x8000, 0xc001e423b8)
    /go/src/github.com/influxdata/telegraf/plugins/serializers/influx/reader.go:52 +0xea
io.copyBuffer(0x2500f40, 0xc0002bf760, 0x2501ba0, 0xc000766090, 0xc00116a000, 0x8000, 0x8000, 0x0, 0xc000900000, 0x4)
    /usr/local/go/src/io/io.go:402 +0x125
io.Copy(0x2500f40, 0xc0002bf760, 0x2501ba0, 0xc000766090, 0x45e836, 0xc001ac5080, 0xc0008f8120)
    /usr/local/go/src/io/io.go:364 +0x5a
github.com/influxdata/telegraf/internal.CompressWithGzip.func1(0xc0002bf760, 0x2501ba0, 0xc000766090, 0xc001428090, 0xc002e17fe8)
    /go/src/github.com/influxdata/telegraf/internal/internal.go:317 +0x4f
created by github.com/influxdata/telegraf/internal.CompressWithGzip
    /go/src/github.com/influxdata/telegraf/internal/internal.go:316 +0x1b1

Additional info:

I'm not totally certain if this happened on upgrade from 1.9.2 to 1.9.5

@danielnelson danielnelson self-assigned this Mar 1, 2019
@danielnelson danielnelson added this to the 1.10.0 milestone Mar 1, 2019
@danielnelson danielnelson added bug unexpected problem or unintended behavior panic issue that results in panics from Telegraf regression something that used to work, but is now broken labels Mar 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug unexpected problem or unintended behavior panic issue that results in panics from Telegraf regression something that used to work, but is now broken
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants