Update Serf and memberlist

[banks]

This includes fixes that improve gossip scalability on very large (> 10k node) clusters.

The Serf changes:

• take snapshot disk IO out of the critical path for handling messages Snapshot flushes to disk async from event channel serf#524
• make snapshot compaction much less aggressive - the old fixed threshold caused snapshots to be constantly compacted (synchronously with request handling) on clusters larger than about 2000 nodes! Snapshot compaction threshold is relative to node count serf#525

Memberlist changes:

• prioritize handling alive messages over suspect/dead to improve stability, and handle queue in LIFO order to avoid acting on info that 's already stale in the queue by the time we handle it. Prioritize alive message over other messages memberlist#159
• limit the number of concurrent pushPull requests being handled at once to 128. In one test scenario with 10s of thousands of servers we saw channel and lock blocking cause over 3000 pushPulls at once which ballooned the memory of the server because each push pull contained a de-serialised list of all known 10k+ nodes and their tags for a total of about 60 million objects and 7GB of memory stuck. While the rest of the fixes here should prevent the same root cause from blocking in the same way, this prevents any other bug or source of contention from allowing pushPull messages to stack up and eat resources. Limit concurrent push/pull connections memberlist#158

[mkeeler]

LGTM

[bcmills]

In the course of investigating golang/go#27457, I noticed that the latest tagged version of consul requires a version of serf from after its latest release tag, and I suspect it's due to this change.

Was that property intentional? (If not, you might want to add a step to your release-tagging checklist to make sure all dependencies are also tagged.)

[banks]

@bcmills Yes sadly it is intentional. Serf as a product has not a lot of active development currently so there have been several bug fixes to it that are needed in upstreams that consume it as a library but no official release of the Serf binary yet.

We intend release Serf soon though as several worthwhile improvements and fixes have accumulated now. If it's causing any issues for you let us know.