[5 / 5] Introduce approval-voting-parallel

[alexggh]

This is the implementation of the approach described here: #1617 (comment) & #1617 (comment) & #1617 (comment).

Description of changes

The end goal is to have an architecture where we have single subsystem(approval-voting-parallel) and multiple worker types that would full-fill the work that currently is fulfilled by the approval-distribution and approval-voting subsystems. The main loop of the new subsystem would do just the distribution of work to the workers.

The new subsystem will have:

• N approval-distribution workers: This would do the work that is currently being done by the approval-distribution subsystem and in addition to that will also perform the crypto-checks that an assignment is valid and that a vote is correctly signed. Work is assigned via the following formula: worker_index = msg.validator % WORKER_COUNT, this guarantees that all assignments and approvals from the same validator reach the same worker.
• 1 approval-voting worker: This would receive an already valid message and do everything the approval-voting currently does, except the crypto-checking that has been moved already to the approval-distribution worker.

On the hot path of processing messages no synchronisation and waiting is needed between approval-distribution and approval-voting workers.

Guidelines for reading

The full implementation is broken in 5 PRs and all of them are self-contained and improve things incrementally even without the parallelisation being implemented/enabled, the reason this approach was taken instead of a big-bang PR, is to make things easier to review and reduced the risk of breaking this critical subsystems.

After reading the full description of this PR, the changes should be read in the following order:

1. [1 / 5] Optimize logic for gossiping assignments #4848, some other micro-optimizations for networks with a high number of validators. This change gives us a speed up by itself without any other changes.
2. [2 / 5] Make approval-distribution logic runnable on a separate thread #4845 , this contains only interface changes to decouple the subsystem from the Context and be able to run multiple instances of the subsystem on different threads. No functional changes
3. [3 / 5] Move crypto checks in the approval-distribution #4928, moving of the crypto checks from approval-voting in approval-distribution, so that the approval-distribution has no reason to wait after approval-voting anymore. This change gives us a speed up by itself without any other changes.
4. [4 / 5] Make approval-voting runnable on a worker thread #4846, interface changes to make approval-voting runnable on a separate thread. No functional changes
5. This PR, where we instantiate an approval-voting-parallel subsystem that runs on different workers the logic currently in approval-distribution and approval-voting.
6. The next step after this changes get merged and deploy would be to bring all the files from approval-distribution, approval-voting, approval-voting-parallel into a single rust crate, to make it easier to maintain and understand the structure.

Results

Running subsystem-benchmarks with 1000 validators 100 fully ocuppied cores and triggering all assignments and approvals for all tranches

Approval does not lags behind.

Master

Chain selection approved  after 72500 ms hash=0x0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a

With this PoC

Chain selection approved  after 3500 ms hash=0x0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a

Gathering enough assignments

Enough assignments are gathered in less than 500ms, so that gives un a guarantee that un-necessary work does not get triggered, on master on the same benchmark because the subsystems fall behind on work, that number goes above 32 seconds on master.

Cpu usage:

Master

CPU usage, seconds                     total   per block
approval-distribution                96.9436      9.6944
approval-voting                     117.4676     11.7468
test-environment                     44.0092      4.4009

With this PoC

CPU usage, seconds                     total   per block
approval-distribution                 0.0014      0.0001 --- unused
approval-voting                       0.0437      0.0044.  --- unused
approval-voting-parallel              5.9560      0.5956
approval-voting-parallel-0           22.9073      2.2907
approval-voting-parallel-1           23.0417      2.3042
approval-voting-parallel-2           22.0445      2.2045
approval-voting-parallel-3           22.7234      2.2723
approval-voting-parallel-4           21.9788      2.1979
approval-voting-parallel-5           23.0601      2.3060
approval-voting-parallel-6           22.4805      2.2481
approval-voting-parallel-7           21.8330      2.1833
approval-voting-parallel-db          37.1954      3.7195.  --- the approval-voting thread.

Enablement strategy

Because just some trivial plumbing is needed in approval-distribution and approval-voting to be able to run things in parallel and because this subsystems plays a critical part in the system this PR proposes that we keep both ways of running the approval work, as separated subsystems and just a single subsystem(approval-voting-parallel) which has multiple workers for the distribution work and one worker for the approval-voting work and switch between them with a comandline flag.

The benefits for this is twofold.

1. With the same polkadot binary we can easily switch just a few validators to use the parallel approach and gradually make this the default way of running, if now issues arise.
2. In the worst case scenario were it becomes the default way of running things, but we discover there are critical issues with it we have the path to quickly disable it by asking validators to adjust their command line flags.

Next steps

• Make sure through various testing we are not missing anything
• Polish the implementations to make them production ready
• Add Unittest Tests for approval-voting-parallel.
• Define and implement the strategy for rolling this change, so that the blast radius is minimal(single validator) in case there are problems with the implementation.
• Versi long running tests.
• Add relevant metrics.

@ordian @eskimor @sandreim @AndreiEres, let me know what you think.

[sandreim]

The last piece of the puzzle is looking good !

Something that I think could be improved is reducing the blast radius of the approval_voting_parallel_enabled flag. I see it in all subsystems that interact with approval voting.

[alexggh]

Something that I think could be improved is reducing the blast radius of the approval_voting_parallel_enabled flag. I see it in all subsystems that interact with approval voting.

Thought about this a bit, one thing to have in mind here is that these are temporary, once the new subsystem becomes the norm they will be remove/cleaned up, so I don't think we will have them more than one/two releases.

	if approval_voting_parallel_enabled {
		send_new_subsystem_message
	} else {
		send_old_subsystem_message
	}

The root cause for them is that with the new parallelised approach we want to:

1. Consolidate the two subsystems approval-distribution and approval-voting in a single one(approval-voting-parallel), hence both the callers of the approval-voting and approval-distribution have to know to send the new messages, so even if we don't introduce a new subsystem and use the existing approval-distribution as the orchestrator of the workers as suggested here we would still need to introduce the above if else in all the callers of the approval-voting.

2. We want to keep the both approaches of running the original 2 subsystems and the parallelised one and switch between them with a runtime knob, if we would just have the parallelised mode then this logic would go away and it will once we built enough trust the parallelised mode is hardened enough to be the default. My opinion, here is that the benefits outweighs the temporary ugliness that this knob needs to be plumbed in the subsystems that interact with approval voting subsystems.

So from my perspective we have three options here:

a) Don't support both modes of running and in this case we rely on discovering all the bugs in our QA phase on versi, westend and rococo and then everyone on kusama and polkadot will start running in the new parallel mode. The rollback path in that case would be to ask people to go to the previous polkadot binary and with that any other unrelated features and bugs will be rolled-back. I don't think we should go this route.

b) Do not add a new subsystem approval-voting-parallel and use approval-distribution as the orchestrator, this would reduce a bit the touched surface, but it won't solve all paths because all approval-voting callers would still need to have this switch logic between the old and new mode. Additionally, I think this will make it a bit harder to review the current PR and distinguished between the normal mode with 2 subsystems and the new proposed parallel mode.

c) Take into consideration that this logic will be clean-up/removed anyway, so I don't think is worth investing adding other abstractions on top of it, while the solution is not the most elegant it is the fastest and simplest to understand and to cleanup once we don't need it anymore. With that said there are a few suggestions I can easily implement to drop the subsystem_disabled and subsystem_enabled flags by having an alternative to validator_overseer_builder where we don't instantiate the unneeded subsystems and use the Dummysubsystem like we do in collator_overseer_builder, but unfortunately we would still need the knobs where we decide if we send the ApprovalVotingParallelMessage or the older one.

[ordian]

Looking good overall from a light review

[ordian]

Great work! 🚀

[sandreim]

Something that I think could be improved is reducing the blast radius of the approval_voting_parallel_enabled flag. I see it in all subsystems that interact with approval voting.

Thought about this a bit, one thing to have in mind here is that these are temporary, once the new subsystem becomes the norm they will be remove/cleaned up, so I don't think we will have them more than one/two releases.

I have no better idea on how to make this better, so best is we stick with current approach.

[sandreim]

Excellent work @alexggh !

Looking forward to seeing this live. We should try to increase Kusama to 750 para validators after the para inherent runtime weight fix is applied.