After a while, etcd consumes most memory and CPU due to constant re-elections

[TJC]

Seen on "CoreOS 367.1.0", running inside KVM-based virtual machines under Linux.

After running for "some time", etcd appears to get into some kind of tight loop, consuming 100% of a CPU all the time, and gradually increasing in memory usage. Eventually it will consume enough to start breaking other services on the machine.

Looking at the logs for etcd, it seems to be constantly electing new leaders due to heartbeat timeouts -- but I suspect this is a symptom of the memory exhaustion rather than the original cause.

[jonboulle]

Seems like this is probably etcd-io/etcd#868 and/or etcd-io/etcd#915

[yichengq]

This seems to be etcd-io/etcd#868, and it has fixed in 0.4.6

[TJC]

Ah, cool. Sorry, I tried searching for similar issues first, but missed that ticket.
I'll look forward to the 0.4.6 etcd hitting beta streams.

[TJC]

I hope that fixes the memory leak issue as well? etcd was using a couple of gigabytes of memory after a couple of weeks of runtime.

[yichengq]

@TJC Yes, it fixes the memory leak and the memory usage should be reasonable now.

[TJC]

OK, cool, I guess I should close this ticket now then.

[pmatety]

I still see the issue in latest kubernetes

[euank]

@Rev1408 Please file a new issue with the details of the issue you're encountering (etcd version, memory usage, preferably heap snapshot after turning on --enable-pprof and dumping it).