Fix duplicate headers download in initial sync

[codablock]

Now that initial block download is delayed until the headers sync is done,
it was noticed that the initial headers sync may happen multiple times in
parallel in the case new blocks are announced. This happens because for
every block in INV that is received, a getheaders message is immediately
sent out resulting in a full download of the headers chain starting from
the point of where the initial headers sync is currently at. This happens
once for each peer that announces the new block. This slows down the
initial headers sync and increases the chance of another block being
announced before it is finished, probably leading to the same behavior
as already described, slowing down the sync even more...and so on.

This commit delays sending of GETHEADERS to later in case the chain is too
far behind while a new block gets announced. Header chains will still be
downloaded multiple times, but the downloading will start much closer
to the tip of the chain, so the damage is not that bad anymore.

This ensures that we get all headers from all peers, even if any of them
is on another chain. This should avoid what happened in
bitcoin#8054
which needed to be reverted later.

This fixes the Bitcoin issue bitcoin#6755

[codablock]

I added a new commit on top that should fix the failing tests.

There is probably still an issue in case someone got unlucky when the initial headers sync peer is chosen and that one is really much behind. I'm going to test this on testnet as there seem to be many peers lurking around which stopped catching up. I've got an idea how to fix this but I need to verify this first.

[codablock]

I added another commit on top. What I assumed about not catched up peers was correct and could easily be reproduced on testnet.

If your fresh node got unlucky and chose a not catched up peer for the initial headers download, the initial headers download would stall forever (because the headers tip never comes near the current time). As I added delaying of GETHEADERS messages with my initial commit, there would never be a new attempt an downloading a longer chain from another peer. This in turn resulted in the delayed sending of GETHEADERS to be never triggered and thus stall everything.

The solution is to check for finished HEADER chains (if nCount < MAX_HEADERS_RESULTS, including 0) and then disconnect that peer if we did not come close to the tip. This triggers another attempt for the initial headers download, which will start from where the previous one finished.

I was not sure about the disconnecting of the peer, as that is quite hard. But at the same time I think that such a peer should never have tried to send us any headers, that's why GETHEADERS is actually ignored while a node is in IBD.

I'm not sure why there are so many nodes on testnet which are so far behind. In a discussion with @Alex-Werner he mentioned that many 0.12.1 nodes might suffer from an old sync problem with super blocks and thus be stalled forever. Good these nodes were actually there, as otherwise it would have been hard to figure out this new sync problem :)

Some of these nodes might actually also have forked away, because I see some headers syncing action with these after my node has finished initial headers download. This looks like they have foked shorty before they stopped catching up. Not sure if someone should look into excessive forking in testnet.

EDIT: I feel the PR is ready to be merged after some review and testing.

[UdjinM6]

👍

Slightly tested, seems to work as expected - picked two peers who were stuck at different height and they were indeed disconnected so sync continued from a better one (3rd peer).

ACK