Fix candidate set address state handling

[teor2345]

Motivation

The CandidateSet address books: disconnected, gossiped, and failed, are not disjoint. But the module documentation says they should be.

The overall peer state handling is also a bit broken and inconsistent - it looks like peers can get stuck in particular states (perhaps related to #1633 or #1435).

Solution

Design:

• Redesign AddressBook by deleting duplicate indexes, rather than maintaining complex runtime invariants
• Add a PeerConnectionState to each MetaAddr
• Use a single peer set for all peers, regardless of state
• Implement time-based liveness as an AddressBook method, rather than a PeerConnectionState variant

Implementation:

• Simplify the AddressBook iterator implementation, replacing it with methods that are more obviously correct
• Simplify AddressBook changes using update and take modifier methods
• Consistently collect peer metrics

Documentation:

• Expand and update the peer set documentation

We can optimise later, but for now we want simple code that is more obviously correct.

The code in this pull request has:

• Documentation Comments
• Integration Tests

Review

I don't know if @yaahc or @dconnolly is more familiar with this code, but it's a big enough change that everyone should probably take a look.

This rewrite might solve a whole bunch of our network problems, so it's a high priority. But the review is not urgent.

Related Issues

Closes #1707.
Closes #1435, according to our testing - Zebra hasn't hung over several weeks of cumulative node runtime.

Follow Up Work

Fix the Client state handling bug (#1599)

Performance/Security:

• limit the total address book size to maintain performance, discarding the oldest addresses once the limit is reached
  • 2x the peer set target size should be more than sufficient for our needs - this is close to the current number of active mainnet peers, so the code actually gets exercised on a regular basis

Security:

• handle future peer-provided last_seen times by subtracting the local/remote time delta from all the times in that peer's response
  • use relative skew to avoid assuming anything about the correctness of the local or remote clocks (the security analysis and testing done absolute times can get quite complex)
  • check for anywhere else in the protocol that uses peer-provided times or other untrusted/unverified data
• filter out known addresses, then limit the number of new addresses used from each request to target peer size/3
  • with a minimum of 3, so we almost always get addresses from multiple nodes, because some nodes are dual stack
  • this avoids having a single peer provide our entire set of peer connections (or our entire address book)

Network health:

• in response to peer requests, only provide peers we've actually connected to, to avoid gossiping unverified peer addresses, last_seen times, or services
  • decide if we want to send live or Responded peers
  • think about what happens for IPv4-only or IPv6-only peers
• don't gossip peers that are older than a day or so - some NAT / dynamic DNS networks change addresses every day, or even multiple times a day (laptops/mobiles)

Correctness:

• reject updates that move existing peers into the AssumedDead and NeverAttempted states. (Updates can only be AttemptPending, AssumedAlive, or Failed.)
• ensure that all inserts of new peers are in the NeverAttempted state.

Refactors:

• Implement time-based liveness as an AssumedDead state, via update and take inner type methods
  • make the MetaAddr state into a template parameter, which defaults to the outer state enum
• make future peer-provided last_seen times into a parse error, if possible, so API users have to handle them

Features:

• Open a follow-up ticket to set up a persistent peer address book cache using RocksDB?

[teor2345]

@dconnolly @yaahc I've made the required design changes from the first review.

I'd like to get this reviewed and merged soon, so we can get it into the next alpha.

There's a bunch of other refactors, performance improvements, and security fixes we could do. But they're not required to fix the hangs. So I think we should open separate tickets for them, and schedule them in future sprints.

[teor2345]

I've rebased on #1750 to silence a bunch of clippy lints.

[teor2345]

The CI failure is #1730.

[teor2345]

Testnet is unstable, we'll fix that in #1222

[teor2345]

Merging because @dconnolly approved and asked for a rebase