Enable snapshot and finalizer commits integration

[frolosofsky]

This PR adds integration code for snapshot and finalizer commits that effectively enables fast-sync with minor restrictions (#852 (review)). On the first step of fast-sync, node restores finalization state by downloading finalizer commits from peers. On the second step, node downloads UTXO snapshot from peers and then downloads last missing blocks.

TODO:

• Check that all the votes are recorded during commits exchange when fast-sync enabled.

Chapter in the #836 story.

[frolosofsky]

Rebased with master, updated test to epoch=1 feature. Picked updated #856.

[kostyantyn]

ConceptACK 473e9ee

We need to extend LookupFinalizedBlockIndex https://github.com/dtr-org/unit-e/blob/master/src/snapshot/p2p_processing.cpp#L19-L28 and check that the snapshot hash is part of the finalized epoch.

We should also extend feature_snapshot.py if we see it's not enough. The added feature_snapshot_finalization.py seems to duplicate the same functionality.

In p2p_snapshot.py we check snapshot sending on a P2P level and trying to catch edge-cases. Would be good to extend test_invalid_snapshot https://github.com/dtr-org/unit-e/blob/master/test/functional/p2p_snapshot.py#L452 and provide the snapshot which is not from the finalized epoch.

[frolosofsky]

@kostyantyn

We should also extend feature_snapshot.py if we see it's not enough. The added feature_snapshot_finalization.py seems to duplicate the same functionality.

It does not duplicate. It tests snapshot and commits integration, i.e., checks that finalization state transfer complete and node can operate after fast sync, e.g., accept votes and process finalization state further.

[kostyantyn]

@frolosofsky we don't have 2 ways to sync snapshot. We used headers first, now we switched to commits first, so the initial test needs to be adjusted. From the description of that test we cover:

This test covers the following scenarios:
1. node can discover the peer that has snapshot
2. sync using snapshot
3. after the sync, the node can accept/propose new blocks
4. the node can switch to the fork which is created right after the snapshot

At the time of writing that test, we didn't have commits so we didn't assert finalizationstate but now we do have so let's add assert_finalizationstate there. This is the only thing I see is not covered in feature_snapshot.py.

[frolosofsky]

@kostyantyn It's not enough to add assert_finalizationstate, we also need to connect validator and setup some deposit. That test already runs 6 nodes that makes logs messy.

I'm not a big fan of having all-in-one tests. Let's split that test on three files and then merge test_fast_sync scenario with feature_snapshot_finalization.

[frolosofsky]

Extended feature_snapshot_finalization with a checks that test double votes conditions during fast-sync.

[kostyantyn]

curious, why the current changes impacted on this test?

[frolosofsky]

There was a race condition detected in this unit test after adding https://github.com/dtr-org/unit-e/pull/852/files#diff-eff7adeaec73a769788bb78858815c91R3273

[kostyantyn]

Would suggest keeping this function in the current test feature_snapshot_finalization.py because it's not generic enough to use in multiple scenarios and then work on it separately.

1. it won't work correctly if we are at height=0 as we will generate 4 blocks (we will be at the checkpoint) and node can't vote at a checkpoint
2. It can be useful to provide the list of finalizers and then everyone must vote. I see a few examples then when this function can be used.
3. I'd like to control where the votes are included. In most of tests, we vote after the 1st block, here we vote at checkpoint-1.
4. Would expect to return votes of my finalizer(s) but not all.

[frolosofsky]

It seems to be used in #883. I used similar algorithms in couple of tests already.

it won't work correctly if we are at height=0 as we will generate 4 blocks (we will be at the checkpoint) and node can't vote at a checkpoint
It can be useful to provide the list of finalizers and then everyone must vote. I see a few examples then when this function can be used.

Thanks, I will fix it when move this function al long as wait_for_vote_and_disconnect to the util.py. Let's focus on this in different PR.

I'd like to control where the votes are included. In most of tests, we vote after the 1st block, here we vote at checkpoint-1.

If you like to control, you probably write your code explicitly. This function is a variation of generatetoaddress(X * EPOCH_LENGTH) which ensures finalizer is in time to vote for every epoch.

[thothd]

utACK d90c996

[kostyantyn]

utACK 120b1e2

[castarco]

ConceptACK 120b1e2