[EPIC] HeaderEx protocol extensions and hardening

[Wondertan]

Mainly we should remove hard dependency on trusted peers, so any node is able to sync headers from any immediate peers and not only trusted and/or bootstrappers. The trusted peers would still remain for automatic subjective initialization. Also, there are multiple networking optimization can be applied to speed up significantly speedup synchronization time.

TODO

• header: RW deadlines on stream #635
  • Fixes potential vulnerability
• header: Retry requesting headers several times on network error #636
• header: Error codes to respond with meaningful information if something wrong happens on the responding side, so clients can act accordingly #906
  • ErrNotFound on the wire should be implemented, so client header ex once requested could go further and ask other peers for headers. This is needed to avoid the general request retrying logic and simply go further to another peer.
• service/header: Implement throttling on how many ExtendedHeaders can be requested/sent at once. #191
  • Currently, we only limit the requestor, but an attacker can currently request millions of headers in a range and OOM the responder
  • An error code response saying the request can't be processed as it's bigger than the limit
• header/p2p: Request Head from multiple trusted peers instead of just one and compare them #1019
  • Request trusted header from all and compared, choosing the one that the majority chooses
• header/p2p: Parallelize requesting of headers from multiple peers. #1020
  • Measure response latency and prefer peers with the shortest
  • The simplest working approach would be searching for more peers in header-sub HeaderSub topic(or custom header discovery topic as they are pros and cons of a unified topic that needs to be justified) with transient connections and asking them for a header until we find the required one.
  • exchange.Head should only rely on trustedPeer as subjective initialization
• service/header: Use oneof for ExtendedHeaderRequest so we can define the request by hashOrNumber #197
• header: P2PExchange should respect context on requests. #366
• header/p2p: Ensure Head method sanity checks headers array returned #980
• header/p2p: remove ExtendedHeaderRequest structure #1140
• header: extend Getter interface with GetVerifiedRangeByHeight #1201 // should be done in scope on one PR
• header/p2p | header/core: Sanity check batched header requests for adjacency (make sure they are contiguous) #1078 // should be done in scope on one PR
• improvement(header/p2p): calculate best head on available peers in case of timeout. #1309
• header/core: Investigate if tendermint implements server-side header range limit and implement on CoreExchange #1282
• header/p2p: Implement server-side sanity check on header range request to-1 param such that it does not significantly exceed store.Head() #1316
• improvement(header/p2p): implement retry mechanism for p2p.session #1406
• header/p2p: implement gc for peerTracker #1411
• improvement(header/p2p): remove limit status code #1529
• feat(header|p2p): add blacklisting to peerTracker #1531

[renaynay]

Request size configuration for clients and servers. Servers should tell the number of headers they can serve with the hello msg.

I think this should be an optimisation for much later as I'm sure this will be useful for light nodes. I think for now, we can set a comfortable default for all node types that serve headers and then introduce this customisation later. It will be annoying to update the Hello packet in that case though.

We should think about what the Hello packet should contain. We could probably even include MaxHeadersPerRequest (or whatever we wanna call it) field and just set it to a default for now so we don't have to update the packet later

[Wondertan]

@renaynay, agree with you. Also, note that there won't be any single hello message and each subprotocol may have its own hello msg. Adding more fields to hello msg shouldn't be problematic in this case together with versioning for subprotocol/wire messages

[renaynay]

We still need to turn all TODOs in the comment into issues to make this a tracking issue

[vgonkivs]

• feat!(header/p2p): add deadlines on stream and rework proto #1038
• header/p2p: request head from multiple peers #1046
• feat!(header/p2p): extend header response with status code #1135
• feat(header/p2p): implement ErrHeadersLimitExceeded error #1153

[vgonkivs]

• improvement(header/server): implement timeout for GetRangeByHeight request #1318
• feat(header/p2p): implement GetVerifiedRangeByHeight #1305
• feat(header/p2p): implement gcing for peerTracker #1298
• improvement(header/p2p): parallelize GetRangeByHeight request #1206
• improvement(header/p2p): get bestHead if timeout is reached #1319
• feat(header/p2p): implement retry mechanism for header/p2p session #1477