Introduce basic slot-based collator #3963

skunert · 2024-04-03T11:35:49Z

Part of #3168
On top of #3568

Changes Overview

Introduces a new collator variant in cumulus/client/consensus/aura/src/collators/slot_based/mod.rs
Two tasks are part of that module, one for block building and one for collation building and submission.
Introduces a new variant of cumulus-test-runtime which has 2s slot duration, used for zombienet testing
Zombienet tests for the new collator

Note: This collator is considered experimental and should only be used for testing and exploration for now.

Comparison with `lookahead` collator

The new variant is slot based, meaning it waits for the next slot of the parachain, then starts authoring
The search for potential parents remains mostly unchanged from lookahead
As anchor, we use the current best relay parent
In general, the new collator tends to be anchored to one relay parent earlier. lookahead generally waits for a new relay block to arrive before it attempts to build a block. This means the actual timing of parachain blocks depends on when the relay block has been authored and imported. With the slot-triggered approach we are authoring directly on the slot boundary, were a new relay chain block has probably not yet arrived.

Limitations

Overall, the current implementation focuses on the "happy path"
We assume that we want to collate close to the tip of the relay chain. It would be useful however to have some kind of configurable drift, so that we could lag behind a bit. slot-based-collator: Allow slot drift #3965
The collation task is pretty dumb currently. It checks if we have cores scheduled and if yes, submits all the messages we have received from the block builder until we have something submitted for every core. Ideally we should do some extra checks, i.e. we do not need to submit if the built block is already too old (build on a out of range relay parent) or was authored with a relay parent that is not an ancestor of the relay block we are submitting at. slot-based-collator: Collation & block-builder communication #3966
There is no throttling, we assume that we can submit velocity blocks every relay chain block. There should be communication between the collator task and block-builder task.
The parent search and ConsensusHook are not yet properly adjusted. The parent search makes assumptions about the pending candidate which no longer hold. slot-based-collator: Adjust ConsensusHook and parent search #3967
Custom triggers for block building not implemented.

TBD

PRDOC

sandreim

You also need to enable elastic-scaling-experimental feature when building polkadot-parachain-bin and you should be able to collate properly on all assigned cores.

commit 265365920836bb1d286c9b48b1902a2de278fdd9 Author: Hernando Castano <castano.ha@gmail.com> Date: Wed Jan 29 19:51:15 2020 -0500 Move hc-jp-bridge repo to different folder commit 8271991e95320baba70bd1cb9c4234d0ffd5b638 Merge: 57d0811 304cbc5 Author: Hernando Castano <castano.ha@gmail.com> Date: Wed Jan 29 19:36:41 2020 -0500 Merge branch 'hc-jp-bridge-module' of hc-jp-bridge-module commit 304cbc5f02d003ffa5404c1c01e461e5b8539888 Author: Hernando Castano <HCastano@users.noreply.github.com> Date: Wed Jan 29 00:38:27 2020 -0500 Update bridge pallet to work with the (almost) lastest master (paritytech#4672) * Update decl_error usage * WIP: Update error handling to use DispatchResult * Get module compiling with new error handling * Make tests compile again Main change was updating the usage of InMemoryBackend * Move `sp-state-machine` into dev-dependencies * Bump dependencies to v2.0.0 * Remove some stray comments * Appy code review suggestion commit 510cd6d96372688517496efa61773ea2839f8474 Author: Hernando Castano <HCastano@users.noreply.github.com> Date: Tue Dec 17 12:52:51 2019 -0500 Move Bridge Pallet into FRAME (paritytech#4373) * Move `bridge` crate into `frame` folder * Make `bridge` pallet compile after `the-big-reorg` commit ab54e838ef75e6a3f68fd0944bf22598c10c552f Author: Hernando Castano <castano.ha@gmail.com> Date: Mon Nov 11 21:56:40 2019 +0100 Use new StorageProof type from paritytech#3834 commit 8fc8911fd1b4acc2274c6863fb3dba91b30c90af Author: Hernando Castano <HCastano@users.noreply.github.com> Date: Tue Nov 5 00:50:34 2019 +0100 Verify Ancestry between Headers (paritytech#3963) * Create module for checking ancestry proofs * Use Vec of Headers instead of a HashMap * Move the ancestry verification into the lib.rs file * Change the proof format to exclude `child` and `ancestor` headers * Add a testing function for building header chains * Rename AncestorNotFound error to InvalidAncestryProof * Use ancestor hash instead of header when verifying ancestry * Clean up some stuff missed in the merge commit dbe85738b68358b790cf927b34a804b965a88f96 or: Hernando Castano <HCastano@users.noreply.github.com> Date: Fri Nov 1 15:41:58 2019 +0100 Check given Grandpa validator set against set found in storage (paritytech#3915) * Make StorageProofChecker happy * Update some tests * Check given validator set against set found in storage * Use Finality Grandpa's Authority Id and Weight * Add better error handling * Use error type from decl_error! macro commit 31b09216603d3e9c21144ce8c0b6bf59307a4f97 or: Hernando Castano <HCastano@users.noreply.github.com> Date: Wed Oct 23 14:55:37 2019 +0200 Make tests work after the changes introduced in paritytech#3793 (paritytech#3874) * Make tests work after the changes introduced in paritytech#3793 * Remove unneccessary import commit bce6d804aa86504599ff912387295c58f846cbf3 Author: Jim Posen <jim.posen@gmail.com> Date: Thu Oct 10 12:18:58 2019 +0200 Logic for checking Substrate proofs from within runtime module. (paritytech#3783) commit a7013e94b6c772c1d45a7cacbb445f73f6554fca Author: Hernando Castano <castano.ha@gmail.com> Date: Fri Oct 4 15:21:00 2019 +0300 Allow tracking of multiple bridges commit 3cf648242d631e32bd553a67df54bf5a48912839 Author: Hernando Castano <castano.ha@gmail.com> Date: Tue Oct 1 14:55:04 2019 +0200 Add BridgeId => Bridge mapping commit 001c74c45072213e01857d0a2454379b447c5a76 Author: Hernando Castano <castano.ha@gmail.com> Date: Tue Oct 1 11:10:19 2019 +0200 Get the mock runtime for tests set up commit 38443a1e8b424ed2f148eb95121d009f730e3b5a Author: Hernando Castano <castano.ha@gmail.com> Date: Fri Sep 27 14:52:53 2019 +0200 Clean up some warnings commit bdc3b01401e89c7111f8bf71f84c50750d25089f Author: Hernando Castano <castano.ha@gmail.com> Date: Thu Sep 26 16:41:01 2019 +0200 Add more skeleton code commit 26995efbf4bac2842eb2822322f7ad3c3e88feb8 Author: Hernando Castano <castano.ha@gmail.com> Date: Wed Sep 25 15:16:57 2019 +0200 Create `bridge` module skeleton

commit 657deb4cf4b90f24b9c5bfd62764b197776c262c Author: Hernando Castano <castano.ha@gmail.com> Date: Wed Jan 29 20:14:20 2020 -0500 Move Slava's bridge code into relays folder commit 4868c42c7da959dde7252766996b3ed4e408e439 Author: Hernando Castano <castano.ha@gmail.com> Date: Wed Jan 29 20:01:06 2020 -0500 Move files into `modules/ethereum` commit d1093f3e4238acb1a1a020011452cb928d3f8d7a Merge: 29dc6f9 bfd30ef Author: Hernando Castano <castano.ha@gmail.com> Date: Wed Jan 29 19:59:27 2020 -0500 Merge branch 'master' of slava-async-bridge commit 29dc6f97b1b7d1db99086d35a5336f43d2f0f8af Author: Hernando Castano <castano.ha@gmail.com> Date: Wed Jan 29 19:51:31 2020 -0500 Squashed commit of the following: commit 265365920836bb1d286c9b48b1902a2de278fdd9 Author: Hernando Castano <castano.ha@gmail.com> Date: Wed Jan 29 19:51:15 2020 -0500 Move hc-jp-bridge repo to different folder commit 8271991e95320baba70bd1cb9c4234d0ffd5b638 Merge: 57d0811 304cbc5 Author: Hernando Castano <castano.ha@gmail.com> Date: Wed Jan 29 19:36:41 2020 -0500 Merge branch 'hc-jp-bridge-module' of hc-jp-bridge-module commit 304cbc5f02d003ffa5404c1c01e461e5b8539888 Author: Hernando Castano <HCastano@users.noreply.github.com> Date: Wed Jan 29 00:38:27 2020 -0500 Update bridge pallet to work with the (almost) lastest master (paritytech#4672) * Update decl_error usage * WIP: Update error handling to use DispatchResult * Get module compiling with new error handling * Make tests compile again Main change was updating the usage of InMemoryBackend * Move `sp-state-machine` into dev-dependencies * Bump dependencies to v2.0.0 * Remove some stray comments * Appy code review suggestion commit 510cd6d96372688517496efa61773ea2839f8474 Author: Hernando Castano <HCastano@users.noreply.github.com> Date: Tue Dec 17 12:52:51 2019 -0500 Move Bridge Pallet into FRAME (paritytech#4373) * Move `bridge` crate into `frame` folder * Make `bridge` pallet compile after `the-big-reorg` commit ab54e838ef75e6a3f68fd0944bf22598c10c552f Author: Hernando Castano <castano.ha@gmail.com> Date: Mon Nov 11 21:56:40 2019 +0100 Use new StorageProof type from paritytech#3834 commit 8fc8911fd1b4acc2274c6863fb3dba91b30c90af Author: Hernando Castano <HCastano@users.noreply.github.com> Date: Tue Nov 5 00:50:34 2019 +0100 Verify Ancestry between Headers (paritytech#3963) * Create module for checking ancestry proofs * Use Vec of Headers instead of a HashMap * Move the ancestry verification into the lib.rs file * Change the proof format to exclude `child` and `ancestor` headers * Add a testing function for building header chains * Rename AncestorNotFound error to InvalidAncestryProof * Use ancestor hash instead of header when verifying ancestry * Clean up some stuff missed in the merge commit dbe85738b68358b790cf927b34a804b965a88f96 or: Hernando Castano <HCastano@users.noreply.github.com> Date: Fri Nov 1 15:41:58 2019 +0100 Check given Grandpa validator set against set found in storage (paritytech#3915) * Make StorageProofChecker happy * Update some tests * Check given validator set against set found in storage * Use Finality Grandpa's Authority Id and Weight * Add better error handling * Use error type from decl_error! macro commit 31b09216603d3e9c21144ce8c0b6bf59307a4f97 or: Hernando Castano <HCastano@users.noreply.github.com> Date: Wed Oct 23 14:55:37 2019 +0200 Make tests work after the changes introduced in paritytech#3793 (paritytech#3874) * Make tests work after the changes introduced in paritytech#3793 * Remove unneccessary import commit bce6d804aa86504599ff912387295c58f846cbf3 Author: Jim Posen <jim.posen@gmail.com> Date: Thu Oct 10 12:18:58 2019 +0200 Logic for checking Substrate proofs from within runtime module. (paritytech#3783) commit a7013e94b6c772c1d45a7cacbb445f73f6554fca Author: Hernando Castano <castano.ha@gmail.com> Date: Fri Oct 4 15:21:00 2019 +0300 Allow tracking of multiple bridges commit 3cf648242d631e32bd553a67df54bf5a48912839 Author: Hernando Castano <castano.ha@gmail.com> Date: Tue Oct 1 14:55:04 2019 +0200 Add BridgeId => Bridge mapping commit 001c74c45072213e01857d0a2454379b447c5a76 Author: Hernando Castano <castano.ha@gmail.com> Date: Tue Oct 1 11:10:19 2019 +0200 Get the mock runtime for tests set up commit 38443a1e8b424ed2f148eb95121d009f730e3b5a Author: Hernando Castano <castano.ha@gmail.com> Date: Fri Sep 27 14:52:53 2019 +0200 Clean up some warnings commit bdc3b01401e89c7111f8bf71f84c50750d25089f Author: Hernando Castano <castano.ha@gmail.com> Date: Thu Sep 26 16:41:01 2019 +0200 Add more skeleton code commit 26995efbf4bac2842eb2822322f7ad3c3e88feb8 Author: Hernando Castano <castano.ha@gmail.com> Date: Wed Sep 25 15:16:57 2019 +0200 Create `bridge` module skeleton commit bfd30ef8363b1483ef1107ae1eb958a4e944c93b Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Tue Dec 10 12:10:53 2019 +0300 actually use signer from CLI to sign Substrate transactions commit 504028eac60d9d14ba95b506cd355b0d2f405ce0 Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Tue Dec 10 12:02:22 2019 +0300 go offline for a bit on connection error commit 446d0c8d20187dfd1beb173958ea28f2ad97887d Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Tue Dec 10 11:25:50 2019 +0300 enable info logs by default commit d039c60ec72bc91adfdad85442bc99a93b7f8e8d Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Tue Dec 10 11:12:51 2019 +0300 support basic CLI arguments commit 65c6d48e23576f36e8541878b920a03730226392 Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Mon Dec 9 15:37:48 2019 +0300 fix restart commit 96e94c1c4b22d732078f8c401b872c5f8246c3fe Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Mon Dec 9 14:57:53 2019 +0300 license commit 68f4191e6cdd211ac8975e0b79f8a6f46a3ca953 Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Mon Dec 9 14:56:05 2019 +0300 restart sync when Substrate reorgs && we are unlucky commit 29887c446167d580d73cc03a0b71c31890cafb51 Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Mon Dec 9 13:49:31 2019 +0300 only read genesis hash once commit 832492b8393fe2063adf9c58c2b9e060dc3e4efb Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Mon Dec 9 13:23:26 2019 +0300 changed TODO commit 9dbc130e5fa036ae63d973819daf30f4ed6ffb5b Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Mon Dec 9 13:16:56 2019 +0300 removed obsolete exit future commit d03408cd8284eb0c61e7e96429b4f6199353e030 Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Mon Dec 9 13:16:17 2019 +0300 removed obsolete TODOs + moved a couple of TODOs to runtime module commit ed8bec44b79f9a2ce829e59f10181368b2f42139 Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Mon Dec 9 12:37:05 2019 +0300 explained TODO fix commit aa9c4c66ec2904eeb6072d654718b0ac0b7d8803 Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Mon Dec 9 12:28:09 2019 +0300 fix tx outcome serialization commit 126f8f5484dac8c4af588ae86dc8855919d6c822 Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Mon Dec 9 12:05:05 2019 +0300 prune old ethereum headers when Substrate best header is too far in the future commit c7bd301e631a44fe3263e188d0956081aa84f31e Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Fri Dec 6 12:51:50 2019 +0300 fix trace commit 549bb7acdb30cfdafe6c8600f0410212539ea63d Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Fri Dec 6 12:51:26 2019 +0300 tx hashes are already a part of Block response commit 7864017909f87ea36955d605a924c3c88bc88df3 Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Thu Dec 5 12:29:37 2019 +0300 submit bunch of headers at once + some fixes commit 96485f85d38c144f0771f02ba692216a60356665 Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Wed Dec 4 17:22:13 2019 +0300 print status messages commit ae0ec4c087136db653339537daab7f96a8c21b65 Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Wed Dec 4 17:06:00 2019 +0300 continue actual Substrate client implementation commit 8146293740d70b88904568ff8e5acdfbadf06fd3 Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Wed Dec 4 13:49:30 2019 +0300 fix IncompleteHeader condition commit 767c6201157dabcccf7f62e643681ca298224fb1 Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Wed Dec 4 10:55:06 2019 +0300 actual Substrate client implementation commit 221fd4ccd2b1eea12c9dacf800d80e15ec115c1b Author: Svyatoslav Nikolsky <svyatonik@gmail.com> Date: Wed Nov 20 17:28:13 2019 +0300 initial commit