panic: runtime error: index out of range

[chexxor]

I'm unsure what I did to break the logger, but it's broken and in a "CrashLoopBackOff" and I don't know how to recover it.

This occurred while I was in a long session of deis pulling my app as a Docker image with small changes to the code each time.

I haven't looked into this Go stacktrace yet. Any interpretations of this?

[chexxor@fedora myproject]$ kubectl --namespace=deis logs deis-logger-9343270-5l8d8
2017/01/31 04:17:15 INF    1 [logs/consume] (10.71.240.167:4150) connecting to nsqd
2017/01/31 04:17:15 Log aggregator running
2017/01/31 04:17:15 Weblog server serving at http://[::]:8088
panic: runtime error: index out of range

goroutine 49 [running]:
panic(0x7afc00, 0xc42000c0a0)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/deis/logger/log.buildApplicationLogMessage(0xc420083540, 0xc420315c50, 0x816c4a)
	/go/src/github.com/deis/logger/log/message_handler.go:61 +0x3b8
github.com/deis/logger/log.handle(0xc42013001e, 0x1d0, 0x1d0, 0x9f20a0, 0xc42010c780, 0xc420013130, 0xc4201184d8)
	/go/src/github.com/deis/logger/log/message_handler.go:33 +0xf3
github.com/deis/logger/log.newNSQAggregator.func1(0xc420119c20, 0xc4202d0c01, 0x101)
	/go/src/github.com/deis/logger/log/nsq_aggregator.go:23 +0x57
github.com/deis/logger/vendor/github.com/nsqio/go-nsq.HandlerFunc.HandleMessage(0xc4201166c0, 0xc420119c20, 0x7b43a0, 0xc4201166c0)
	/go/src/github.com/deis/logger/vendor/github.com/nsqio/go-nsq/consumer.go:42 +0x30
github.com/deis/logger/vendor/github.com/nsqio/go-nsq.(*Consumer).handlerLoop(0xc420144000, 0x9ecac0, 0xc4201166c0)
	/go/src/github.com/deis/logger/vendor/github.com/nsqio/go-nsq/consumer.go:1105 +0x14a
created by github.com/deis/logger/vendor/github.com/nsqio/go-nsq.(*Consumer).AddConcurrentHandlers
	/go/src/github.com/deis/logger/vendor/github.com/nsqio/go-nsq/consumer.go:1087 +0xaa

[rimusz]

yes, I'm seeing the same issue in on of our clusters too

[n0n0x]

had the same issue, fixed this by deleting nsqd pod and then the logger pod. Both will be automatically recreated.

Don't know if this would be the best solution, but it worked for me :)

[bacongobbler]

@mboersma have you been able to reproduce this issue or know of a fix?

[bacongobbler]

The error is pointing directly at https://github.com/deis/logger/blob/master/log/message_handler.go#L61 which means that the pod name that the logger is attempting to use does not pass the regex. We should handle that better.