1.17 on freebsd slow death?

[bunnyevans]

System info:

Freebsd 12.2 with telegraph from pkg, 1.17
No bells or whistles, just some SNMP.

Actual behavior:

After some unknown amount of uptime, things break.
It does not however die, or even stop suddenly.

e.g. snmp traffic from telegraf dropped suddenly about 2230, then stopped altogether around 0200

Additional info:

2021-03-08T16:27:49Z I! Starting Telegraf 1.17.0
...
wirep: p->m=0(0) p->status=2
fatal error: wirep: invalid p state

runtime stack:
runtime.throw(0xb86614, 0x16)
/usr/local/go/src/runtime/panic.go:1116 +0x72
runtime.wirep(0xc00005d000)
/usr/local/go/src/runtime/proc.go:4492 +0x128
runtime.acquirep(0xc00005d000)
/usr/local/go/src/runtime/proc.go:4461 +0x2b
runtime.stopm()
/usr/local/go/src/runtime/proc.go:1926 +0xed
runtime.findrunnable(0xc000067000, 0x0)
/usr/local/go/src/runtime/proc.go:2485 +0xa7f
runtime.schedule()
/usr/local/go/src/runtime/proc.go:2683 +0x2d7
runtime.park_m(0xc000bc7380)
/usr/local/go/src/runtime/proc.go:2851 +0x9d
runtime.mcall(0x200000)
/usr/local/go/src/runtime/asm_amd64.s:318 +0x5b

goroutine 1 [semacquire, 8652 minutes]:
sync.runtime_Semacquire(0xc0006bda38)
/usr/local/go/src/runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc0006bda30)
/usr/local/go/src/sync/waitgroup.go:130 +0x65
wirep: p->m=0(0) p->status=3
fatal error: wirep: invalid p state

Second crash, same thing:

2021-03-14T16:40:11Z I! Starting Telegraf 1.17.0
wirep: p->m=0(0) p->status=2
fatal error: wirep: invalid p state

runtime stack:
runtime.throw(0xb86614, 0x16)
/usr/local/go/src/runtime/panic.go:1116 +0x72
runtime.wirep(0xc000069800)
/usr/local/go/src/runtime/proc.go:4492 +0x128
runtime.acquirep(0xc000069800)
/usr/local/go/src/runtime/proc.go:4461 +0x2b
runtime.findrunnable(0xc00005a800, 0x0)
/usr/local/go/src/runtime/proc.go:2463 +0x837
runtime.schedule()
/usr/local/go/src/runtime/proc.go:2683 +0x2d7
runtime.park_m(0xc0007faa80)
/usr/local/go/src/runtime/proc.go:2851 +0x9d
runtime.mcall(0x200000)
/usr/local/go/src/runtime/asm_amd64.s:318 +0x5b

goroutine 1 [semacquire, 218 minutes]:
sync.runtime_Semacquire(0xc0003f2178)
/usr/local/go/src/runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc0003f2170)

[bunnyevans]

Some crash logs:

crash4.txt
crash1.txt
crash2.txt
crash3.txt

[ivorybilled]

Saw the same error mentioned in this issue about freeBSD, I wonder if it's related at all. golang/go#43873

[bunnyevans]

No one cares about old versions. Only new hotness with broken SNMP.

[powersj]

@bunnyevans,

Thank you for uploading a number of crash logs. Those logs do not point to something in Telegraf, but instead in Go itself. If you will note the comment made by Jagularr right after you filed this, he pointed to an upstream Go issue with another user of FreeBSD with a similar crash log.

This lead me to golang/go#43873, which lead to this FreeBSD kernel patch. All of these reports are with other Go programs and pointed to an issue with the *BSD kernel. See a similar report for NetBSD. All of these issues are due to AMD Zen-based CPUs. Are you running one of those?

Based on all the logs and the above issues I do not think there is anything we can do from Telegraf's perspective here, but I do encourage you to check out those bug reports or see when the kernel will get that fix.