Packetbeat panic: Too many events acked

[jamiesmith]

A google search found a similar issue on discuss https://discuss.elastic.co/t/packetbeat-panic-too-many-events-acked/102055/2

• Version: packetbeat-6.0.0-rc2-darwin-x86_64
• Operating System: osx 10.13.1
• Steps to Reproduce: does it pretty much every time, some times right away, while other times a few minutes. I have tried both logstash and elasticsearch outputs.

sudo ./packetbeat -e -c packetbeat.yml

Logs:

2017/11/06 19:14:32.960838 beat.go:426: INFO Home path: [/Users/jamie/ELK/packetbeat-6.0.0-rc2-darwin-x86_64] Config path: [/Users/jamie/ELK/packetbeat-6.0.0-rc2-darwin-x86_64] Data path: [/Users/jamie/ELK/packetbeat-6.0.0-rc2-darwin-x86_64/data] Logs path: [/Users/jamie/ELK/packetbeat-6.0.0-rc2-darwin-x86_64/logs]
2017/11/06 19:14:32.960849 metrics.go:23: INFO Metrics logging every 30s
2017/11/06 19:14:32.961000 beat.go:433: INFO Beat UUID: ffd57af7-5cdb-499d-8e29-819beb3fc8b4
2017/11/06 19:14:32.961015 beat.go:192: INFO Setup Beat: packetbeat; Version: 6.0.0-rc2
2017/11/06 19:14:32.964648 client.go:123: INFO Elasticsearch url: http://orion:9200
2017/11/06 19:14:32.965746 module.go:80: INFO Beat name: orion.jamieandamy.com
2017/11/06 19:14:32.965887 procs.go:78: INFO Process matching disabled
2017/11/06 19:14:32.966985 beat.go:260: INFO packetbeat start running.


2017/11/06 19:14:34.520111 client.go:651: INFO Connected to Elasticsearch version 6.0.0-rc2
2017/11/06 19:14:34.522433 load.go:73: INFO Template already exists and will not be overwritten.
2017/11/06 19:15:02.961310 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=42328608 beat.memstats.memory_alloc=37841056 beat.memstats.memory_total=94225656 libbeat.output.read.bytes=60061 libbeat.output.type=elasticsearch libbeat.output.write.bytes=3717008 libbeat.pipeline.clients=13 libbeat.pipeline.events.active=34 libbeat.pipeline.events.published=4235 libbeat.pipeline.events.retry=50 libbeat.pipeline.events.total=4235 libbeat.pipeline.queue.acked=4201
2017/11/06 19:15:32.961258 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=1646464 beat.memstats.memory_alloc=-9652832 beat.memstats.memory_total=110347280 libbeat.output.read.bytes=112273 libbeat.output.write.bytes=6476849 libbeat.pipeline.events.active=26 libbeat.pipeline.events.published=8415 libbeat.pipeline.events.total=8415 libbeat.pipeline.queue.acked=8389
2017/11/06 19:16:02.961262 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=11024512 beat.memstats.memory_alloc=11270176 beat.memstats.memory_total=108921680 libbeat.output.read.bytes=132602 libbeat.output.write.bytes=7469656 libbeat.pipeline.events.active=-45 libbeat.pipeline.events.published=9897 libbeat.pipeline.events.total=9897 libbeat.pipeline.queue.acked=9942
2017/11/06 19:16:32.961929 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=-14667728 beat.memstats.memory_alloc=-16130072 beat.memstats.memory_total=117723768 libbeat.output.read.bytes=128616 libbeat.output.write.bytes=7252169 libbeat.pipeline.events.active=51 libbeat.pipeline.events.published=9661 libbeat.pipeline.events.total=9661 libbeat.pipeline.queue.acked=9610
panic: Too many events acked (expected=50, total=54)

goroutine 59 [running]:
github.com/elastic/beats/libbeat/publisher/queue/memqueue.(*bufferingEventLoop).processACK(0xc4213d5680, 0x0, 0x0, 0x32)
	/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/eventloop.go:528 +0x468
go.(interface { memqueue.processACK(github.com/elastic/beats/libbeat/publisher/queue/memqueue.chanList, int); memqueue.run() }).(github.com/elastic/beats/libbeat/publisher/queue/memqueue.processACK)-fm(0xc421a2c1c0, 0xc421a2c1c0, 0x32)
	/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:134 +0x4d
github.com/elastic/beats/libbeat/publisher/queue/memqueue.(*ackLoop).handleBatchSig(0xc42142a550, 0xc4212c4120)
	/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/ackloop.go:95 +0xae
github.com/elastic/beats/libbeat/publisher/queue/memqueue.(*ackLoop).run(0xc42142a550)
	/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/ackloop.go:61 +0x377
github.com/elastic/beats/libbeat/publisher/queue/memqueue.NewBroker.func2(0xc421397c00, 0xc42142a550)
	/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:143 +0x57
created by github.com/elastic/beats/libbeat/publisher/queue/memqueue.NewBroker
	/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:144 +0x423

While running, I was also looping through a set of dig commands (script attached)

hot-dig-ity.sh.txt

EDIT- note - the script just digs a subset of wikipedia's top 100 sites

[jamiesmith]

I was trying to re-create this with the debug on (-d '*') but haven't been able to. The frequency of it had died down also.

I ran two different ways - one was sending to logstash, the other to elasticsearch. I saw a similar stack each time.

packetbeat-es.yml.txt