after enabling supervisord collector node_exporter fails after a while

[leyraroro]

LOGS

Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: fatal error: unexpected signal during runtime execution
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x63 pc=0x7f340d21660d]
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: runtime stack:
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: runtime.throw(0xae1a2b, 0x2a)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/runtime/panic.go:605 +0x95
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: runtime.sigpanic()
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/runtime/signal_unix.go:351 +0x2b8
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: goroutine 1098214 [syscall, locked to thread]:
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: runtime.cgocall(0x8e8e90, 0xc45b750de8, 0xae0be1)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/runtime/cgocall.go:132 +0xe4 fp=0xc45b750da8 sp=0xc45b750d68 pc=0x402514
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: net._C2func_getaddrinfo(0x7f33f80008c0, 0x0, 0xc4233191a0, 0xc42000e510, 0x0, 0x0, 0x0)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: net/_obj/_cgo_gotypes.go:86 +0x5f fp=0xc45b750de8 sp=0xc45b750da8 pc=0x58e1df
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: net.cgoLookupIPCNAME.func2(0x7f33f80008c0, 0x0, 0xc4233191a0, 0xc42000e510, 0xc43b547bc0, 0xc42034
7b30, 0x9)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/net/cgo_unix.go:151 +0x13f fp=0xc45b750e40 sp=0xc45b750de8 pc=0x59577f
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: net.cgoLookupIPCNAME(0xc420347b30, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/net/cgo_unix.go:151 +0x175 fp=0xc45b750f38 sp=0xc45b750e40 pc=0x58fa35
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: net.cgoIPLookup(0xc43b547ec0, 0xc420347b30, 0x9)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/net/cgo_unix.go:203 +0x4d fp=0xc45b750fc8 sp=0xc45b750f38 pc=0x59017d
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: runtime.goexit()
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc45b750fd0 sp=0xc45b750fc8 pc=0x45d331
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: created by net.cgoLookupIP
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/net/cgo_unix.go:213 +0xaf
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: goroutine 1 [IO wait]:
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: internal/poll.runtime_pollWait(0x7f340e5d2f70, 0x72, 0xffffffffffffffff)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/runtime/netpoll.go:173 +0x57
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: internal/poll.(*pollDesc).wait(0xc420150a98, 0x72, 0xc420043a00, 0x0, 0x0)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: internal/poll.(*pollDesc).waitRead(0xc420150a98, 0xffffffffffffff00, 0x0, 0x0)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: internal/poll.(*FD).Accept(0xc420150a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/internal/poll/fd_unix.go:335 +0x1e2
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: net.(*netFD).accept(0xc420150a80, 0xaf3b80, 0xc420043b98, 0x40204b)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/net/fd_unix.go:238 +0x42
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: net.(*TCPListener).accept(0xc42014a888, 0xa0d4c0, 0xc420043bc8, 0x400de7)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: net.(*TCPListener).AcceptTCP(0xc42014a888, 0xc420043c10, 0xc420043c18, 0xc420043c08)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/net/tcpsock.go:234 +0x49
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: net/http.tcpKeepAliveListener.Accept(0xc42014a888, 0xaf3438, 0xc420166000, 0x10220e0, 0xc420215bc0
)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/net/http/server.go:3120 +0x2f
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: net/http.(*Server).Serve(0xc4201bcea0, 0x1021a20, 0xc42014a888, 0x0, 0x0)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/net/http/server.go:2695 +0x1b2
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: net/http.(*Server).ListenAndServe(0xc4201bcea0, 0xc4201bcea0, 0x2)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/net/http/server.go:2636 +0xa9
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: net/http.ListenAndServe(0xac6958, 0x5, 0x0, 0x0, 0x1, 0xc4201ac600)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/net/http/server.go:2882 +0x7f
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: main.main()
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /go/src/github.com/prometheus/node_exporter/node_exporter.go:112 +0xa7a
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: goroutine 21 [chan receive, 4132 minutes]:
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: github.com/prometheus/node_exporter/vendor/github.com/kolo/xmlrpc.(*clientCodec).ReadResponseHeade
r(0xc42014dec0, 0xc420215bf0, 0xc42005e170, 0xc42005e170)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /go/src/github.com/prometheus/node_exporter/vendor/github.com/kolo/xmlrpc/client.go:68 +0x57
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: net/rpc.(*Client).input(0xc420146c00)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/net/rpc/client.go:109 +0xb5
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: created by net/rpc.NewClientWithCodec
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/net/rpc/client.go:201 +0xc5
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: goroutine 1042 [chan receive, 4129 minutes]:
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: github.com/prometheus/node_exporter/vendor/github.com/kolo/xmlrpc.(*clientCodec).ReadResponseHeade
r(0xc420080740, 0xc42013e180, 0xc420546620, 0x0)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /go/src/github.com/prometheus/node_exporter/vendor/github.com/kolo/xmlrpc/client.go:68 +0x57
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: net/rpc.(*Client).input(0xc4202ede60)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/net/rpc/client.go:109 +0xb5
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: created by net/rpc.NewClientWithCodec
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/net/rpc/client.go:201 +0xc5
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: goroutine 10 [chan receive, 4132 minutes]:
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: github.com/prometheus/node_exporter/vendor/github.com/kolo/xmlrpc.(*clientCodec).ReadResponseHeade
r(0xc420052a00, 0xc420215c20, 0xc4202dba20, 0x0)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /go/src/github.com/prometheus/node_exporter/vendor/github.com/kolo/xmlrpc/client.go:68 +0x57
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: net/rpc.(*Client).input(0xc4200108a0)
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/net/rpc/client.go:109 +0xb5
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: created by net/rpc.NewClientWithCodec
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: /usr/local/go/src/net/rpc/client.go:201 +0xc5
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: goroutine 5471 [chan receive, 4112 minutes]:
Jun 11 12:27:14 ip-172-31-12-17 node_exporter[9758]: github.com/prometheus/node_exporter/vendor/github.com/kolo/xmlrpc.(*clientCodec).ReadResponseHeade
r(0xc420342b40, 0xc4203ba060, 0xc4205bdb80, 0x0)

Host operating system: output of uname -a

[centos@x ~]$ uname -a
Linux ip-x.eu-west-1.compute.internal 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 0.16.0 (branch: HEAD, revision: d42bd70)
build user: root@a67a9bc13a69
build date: 20180515-15:52:42
go version: go1.9.6

node_exporter command line flags

node_exporter --collector.supervisord

Are you running node_exporter in Docker?

No

What did you do that produced an error?

we were using node_exporter for a long time without any issues, we just enabled the supervisord collector and it has been failing since then. It crashes between the 1st day and 3rd day running. We restart the service until it crashes again.

What did you expect to see?

What did you see instead?

[SuperQ]

We've replaced the library used to parse supervisord data.

See: #978

[FaHeymann]

Hey @SuperQ thank you for the work! Are you planning to create a new release with the changes soon?

[SuperQ]

Yes, hopefully soon. I have a number of changes that need to be released.

[discordianfish]

Should be fixed in 0.17.0, so going to close this. If not, please re-open.