Epic: safekeeper coordination

[kelvich]

Summary & Motivation

In a few scenarios safekeeper (SK) needs coordination with other SK’s that serve the same tenant:

1. WAL deletion. SK needs to know what WAL was safely replicated to delete it. Now we keep WAL indefinitely.
2. Deciding on who is sending WAL to the pageserver. Now sending safekeeper crash/ may lead to a livelock where nobo
3. To enable SK to SK direct recovery without involving compute (that will be implemented in Epic: recovery of lagging safekeepers #1012)

DoD

• safekeepers can re-elect a new a new safekeper to push WAL to the pageserver
• no livelocks that we currently have

A quick follow through for this epic is #1403

Implementation plan:

Per neondatabase/rfcs#16 we decided to have centralized broker (currently etcd) to which every storage node can push / subscribe to. The rough steps are

• Assign storage node ids Generate safekeeper unique IDs. #1228
• Introduce node id to identify pageservers (other services) #1105
• zenithdb/console#746
• Pull & Push safekeeper data from the broker for active timelines #1401
• set up etcd on staging #1462
• enable etcd in staging safekeepers #1540
• etcd support in pageserver and Replicate callmemaybe functionality via etcd #1396
• set up etcd on production
• trim WAL on safekeepers #1403
• improvements for etcd integration for safekeepers #1970
• enable etcd in production safekeepers (or pageservers?) once the prod release is out

[stepashka]

@arssher , please update the implementation thoughts here based on the recent decision to go forward with the etcd instead of gossip for now and #1180