Fix premature disconnections from seeds

[chimp1984]

Fixes issues with disconnections while peer is still in the initital data request phase.

We have several Request/Response cycles at startup:

1. PreliminaryGetDataRequest -> GetDataResponse
2. GetUpdatedDataRequest -> GetDataResponse
3. GetBlocksRequest ->GetBlocksResponse
4. GetDaoStateHashesRequest -> GetDaoStateHashesResponse
5. GetBlindVoteStateHashesRequest -> GetBlindVoteStateHashesResponse
6. GetProposalStateHashesRequest -> GetProposalStateHashesResponse

We set the PeerType to INITIAL_DATA_REQUEST but we had reset the PeerType to PEER once seed sent the GetDataResponse for the GetUpdatedDataRequest. This was too early as the PeerManager at connection management disconnects the low-prio connection (PEER) if there are too many connections.
This has led to the problem that the seed closed the connection before the peer can start the GetBlocksRequest.
Also the last GetDataResponse was at risk to not arrive as the connection might got closed too fast.

We changed that model now so that we count the requests and responses of this bootstrap phasse and if we reach the expected number we assume we are done and reset the PeerType to PEER. As we cannot rely on that we also start a timer to reset after 4 minutes.
When we exceed 20 connections at a normal peer or 34 at seed nodes (with maxConnections 20) we start to disconnect also peers with type INITIAL_DATA_EXCHANGE. We sort by oldest date when we sent or received a msg, so increase the chance that this peer is already done with bootstrapping.

We also removed the removeSuperfluousSeedNodes method which was a bit dangerous from the app side as it was protected only by the allowDisconnectSeedNodes flag which was handled by the GetBlockHandlers, but there was risk that we disconnect before we can request the StateHashes.

We have to take care that seed nodes do not get too weakly connected to healty nodes as they would miss then broadcast messages if all their connections would be occupied by bootstrapping nodes. I doubt that the changes here would have any impact on that behaviour but as its pretty complex we have to be careful with deployment.

I will test it on a small group of seed nodes first and add some more statistics to see which types of connections the seed has.

[cbeams]

Concept ACK. Described changes make sense. I've only scanned the code changes themselves, though, so not a real review. Thanks for turning this around so quickly, @chimp1984!

[chimp1984]

I deployed it to 3 seed nodes. Observing how they behave before extending to other seeds.

I added new statistics logging per connection. That helps to analyse the peer management and see where are bottlenecks.

Jan-06 21:49:33.738 [SeedNodeMain] INFO  b.n.p.p.PeerManager: Connection statistics: 

Connection 1
Age: 1 minute, 45.838 seconds
Peer: [Seed node] sn2bisqad7ncazupgbd3dcedqh5ptirgwofw63djwpdtftwhddo75oid.onion:8000 
Type: PEER 
Direction: Outbound
UID: e2e209db-b50e-4caa-972d-cced1afb90f8
Time since last message: 0.800 seconds
Time for response: [PreliminaryGetDataRequest/Response: 18.532 seconds, GetDaoStateHashesRequest/Response: 0.550 seconds, GetProposalStateHashesRequest/Response: 0.684 seconds, GetBlindVoteStateHashesRequest/Response: 0.621 seconds, GetUpdatedDataRequest/Response: 1.549 seconds]
Sent data: 665.76 kB; {PreliminaryGetDataRequest=1, AddDataMessage=6, GetDaoStateHashesRequest=1, GetProposalStateHashesRequest=1, GetBlindVoteStateHashesRequest=1, RefreshOfferMessage=13, GetUpdatedDataRequest=1, GetPeersRequest=1, BundleOfEnvelopes=7}
Received data: 5.068 MB; {AddDataMessage=2, GetDaoStateHashesResponse=1, GetPeersResponse=1, GetDataResponse=2, RefreshOfferMessage=11, GetProposalStateHashesResponse=1, RemoveDataMessage=1, GetBlindVoteStateHashesResponse=1}
CPU time spent on sending messages: 0.179 seconds
CPU time spent on receiving messages: 3.171 seconds

Connection 2
Age: 1 minute, 45.427 seconds
Peer: [Seed node] wizseedscybbttk4bmb2lzvbuk2jtect37lcpva4l3twktmkzemwbead.onion:8000 
Type: INITIAL_DATA_EXCHANGE 
Direction: Outbound
UID: f65a92c4-45b7-497f-a79c-107393b1287d
Time since last message: 0.701 seconds
Time for response: [PreliminaryGetDataRequest/Response: 19.199 seconds, GetDaoStateHashesRequest/Response: 0.704 seconds, GetProposalStateHashesRequest/Response: 0.834 seconds, GetBlindVoteStateHashesRequest/Response: 0.690 seconds]
Sent data: 337.351 kB; {PreliminaryGetDataRequest=1, AddDataMessage=11, GetDaoStateHashesRequest=1, GetProposalStateHashesRequest=1, GetBlindVoteStateHashesRequest=1, RefreshOfferMessage=17, RemoveDataMessage=1, RemoveMailboxDataMessage=4, BundleOfEnvelopes=10}
Received data: 5.052 MB; {GetDaoStateHashesResponse=1, GetDataResponse=1, GetProposalStateHashesResponse=1, GetBlindVoteStateHashesResponse=1}
CPU time spent on sending messages: 0.090 seconds
CPU time spent on receiving messages: 0.457 seconds

Connection 3
Age: 1 minute, 38.621 seconds
Peer: devinsn2teu33efff62bnvwbxmfgbfjlgqsu3ad4b4fudx3a725eqnyd.onion:8000 
Type: INITIAL_DATA_EXCHANGE 
Direction: Inbound
UID: 81eb9315-5186-4fd6-a153-f9597883e2e6
Time since last message: 2.445 seconds
Time for response: [GetDaoStateHashesRequest awaiting response... , GetProposalStateHashesRequest awaiting response... , GetBlindVoteStateHashesRequest awaiting response... , GetUpdatedDataRequest/Response: 0.008 seconds]
Sent data: 1.989 kB; {GetDataResponse=1, RefreshOfferMessage=1, RemoveDataMessage=1}
Received data: 369.567 kB; {AddDataMessage=8, GetDaoStateHashesRequest=1, GetProposalStateHashesRequest=1, GetBlindVoteStateHashesRequest=1, RefreshOfferMessage=25, GetUpdatedDataRequest=1}
CPU time spent on sending messages: 0.011 seconds
CPU time spent on receiving messages: 0.147 seconds

[sqrrm]

utACK

Will hold off with merging until test nodes have run for a bit

[ripcurlx]

@chimp1984 Regarding the Codacy complaint. Do you want to keep the extra if statement to improve readability?

[chimp1984]

@chimp1984 Regarding the Codacy complaint. Do you want to keep the extra if statement to improve readability?

Yes, I prefer to keep it as it is.

[sqrrm]

utACK