Improve block propagation

[Isabello]

Belongs to #449

Currently the network queries peers using /peer/height. This function returns the height of the peers. It could be improved to push blocks to peers that are not at the same height as the querying node.

Option A

When a peer is behind a set number of blocks, it could be leveraged to automatically post the missing blocks to that peer. This will create more fluidity in block replication for nodes that are only mildly behind.

Other functions that might work for this:

[log] 2016-12-03 07:46:57 | GET /peer/transactions from 45.76.21.88
[log] 2016-12-03 07:46:57 | GET /peer/signatures from 5.249.159.157

Option B

Create a broadcast queue that uses the gathered peers data the pushes the string of blocks nodes are missing to those nodes. This could be put on the same cycle as our transaction release mechanism. This would need to have a limitation (5-10 blocks) to not cause overload for nodes pushing 100s of blocks to behind peers.

[Isabello]

Option A would tie in very well with #332 and the reduction of overall peers queried when the node is not forging a block. While fewer nodes are queried, we make up for this with having my throughput to send blocks.

One thing I am unsure of is if nodes update peers data when responding to an api request that requires system headers. If this is the case we can allow peers data to refresh its self through network activity. This would be an improvement to the efficiency and help mitigate problems that may come about from reducing the total number of peers queried.

[karmacoma]

@Isabello Do you see a problem with simply syncing / pulling more aggressively? I think current timeout for resorting to syncing is maybe too long. Timeout currently is 120 seconds.

One other option to improve reduce the weight of block replication, and thinking of long term when we increase block size, is to push only block headers (minus transactions), with transactions pulled separately. This would restrict immediate payload sent over network for each block. This is the approach currently taken by Ark whom have increased block size.

[Isabello]

Syncing more aggressively opens us up to other issues, for example, we don't receive blocks while undergoing a sync action. Moving the timer downwards would be a definite benefit but theres more we can do on the front side of block replication to be proactive, rather than reactive

Pulling transactions separately to reduce block size, how would that work here? Sending just the block header doesn't give us the whole picture of the block, the nodes would still have to search other nodes for these transactions. As we know checking other nodes for tx/multisig data can lead to complications. Moving away from this should be the path we take, especially when it comes to doing DB commits atomically.

[karmacoma]

Syncing more aggressively opens us up to other issues, for example, we don't receive blocks while undergoing a sync action. Moving the timer downwards would be a definite benefit but theres more we can do on the front side of block replication to be proactive, rather than reactive

Surely we are either doing one operation or the other. At least this is the case currently.

Pulling transactions separately to reduce block size, how would that work here? Sending just the block header doesn't give us the whole picture of the block, the nodes would still have to search other nodes for these transactions. As we know checking other nodes for tx/multisig data can lead to complications. Moving away from this should be the path we take, especially when it comes to doing DB commits atomically.

There is a numberOfTransactions property included in each block which tells us how many transactions there are. We can conditionally pull the transactions from a peer before processing, therefore before any db writes.

[Isabello]

From our discussion earlier we should:

Broadcast to upto 100 matched peers
Broadcast to upto 100 unmatched/fallback peers, prioritizing nodes that are behind by less than x blocks (I prefer 6)

By adding more peers to our broadcast we can avoid peers getting "lost" because their peers data is stale.

[karmacoma]

Decoupling @4miners ideas from #406:

• After block is forged - full block should be broadcasted to limited number of peers (max 100), we should broadcast to peers that send us full blocks before in first order. We can also send some other peers block header. That approach will ensure that every forging delegate receive last network block first, no matter how big is network, should also prevent non-active delegates to determine who is forging.
• When full block is received - we should relay immediatelly if passed schema validation.
• Block header should be relayed (with txs ids) instead of full blocks. If node already have those ids - can reconstruct block on his own, if not - should ask other peer (who have that block) for transactions. That approach will decrease amount of data floating between peers significantly when there is lot of transactions in block.
• When node receive block header - should broadcast it immediatelly to other nodes if it pass schema validation, no matter what. Also we can check that block if is forked one before we ask for full transactions.
• If last block receipt is stale - forging delegates should figure out why and ask network for blocks more aggresively. Every node should take care on his own, if I missing some blocks I should ask for them and get it instead of waiting until somebody send me block stream that I missing.

[karmacoma]

Closing in favour of: #541