collation-generation + collator-protocol: collate on multiple assigned cores #3795

sandreim · 2024-03-22T12:34:35Z

This works only for collators that implement the collator_fn allowing collation-generation subsystem to pull collations triggered on new heads.

Also enables request_v2::CollationFetchingResponse::CollationWithParentHeadData for test adder/undying collators.

TODO:

fix tests
new tests
PR doc

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

…reim/collate_on_multiple_cores Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

polkadot/node/collation-generation/src/lib.rs

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

… tests) Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

…reim/collate_on_multiple_cores

sandreim · 2024-03-25T12:20:09Z

Looks like CI gets stuck and it's this PR's fault: TRY 1 SLOW [>480.000s] test-parachain-undying-collator::integration collating_using_undying_collator TRY 1 SLOW [>540.000s] test-parachain-adder-collator::integration collating_using_adder_collator TRY 1 SLOW [>540.000s] test-parachain-undying-collator::integration collating_using_undying_collator

alindima

Looking good

cumulus/client/consensus/aura/src/collators/lookahead.rs

polkadot/node/collation-generation/src/lib.rs

polkadot/node/network/collator-protocol/src/collator_side/validators_buffer.rs

sandreim · 2024-03-25T13:05:17Z

For some weird reason finality halts

sandreim · 2024-03-25T13:05:42Z

2024-03-25 15:05:01.006 WARN tokio-runtime-worker parachain::approval-voting: Failed to create assignment bitfield block_hash=0x7467bd478b3f36afa7be045b1ae79aef482410b8046335fb8d8f5f42fbea6ee8 err=NullAssignment

sandreim · 2024-03-25T14:16:49Z

For some weird reason finality halts

#3826

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

paritytech-cicd-pr · 2024-03-25T14:36:12Z

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-clippy
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5645876

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

eskimor

Looks good. My only real concern is: Do we have a breaking change here for Babe? If so, what can we do to make this as pain free as possible for parachains?

cumulus/client/consensus/aura/src/collators/lookahead.rs

polkadot/node/network/collator-protocol/src/collator_side/mod.rs

polkadot/node/primitives/src/lib.rs

sandreim · 2024-03-26T11:47:18Z

Looks good. My only real concern is: Do we have a breaking change here for Babe? If so, what can we do to make this as pain free as possible for parachains?

There is no change that touches Babe in this PR, SubmitCollationParams is a polkadot node side primitive

ordian

Looking good modulo missing PRDoc

cumulus/client/consensus/aura/src/collators/lookahead.rs

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

…reim/collate_on_multiple_cores

eskimor

Looks good!

polkadot/node/network/collator-protocol/src/collator_side/mod.rs

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Somehow #3795 was merged but tests are failing now on master. I suspect that CI is not even running these tests anymore which is a big issue. --------- Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

…d cores (paritytech#3795) This works only for collators that implement the `collator_fn` allowing `collation-generation` subsystem to pull collations triggered on new heads. Also enables `request_v2::CollationFetchingResponse::CollationWithParentHeadData` for test adder/undying collators. TODO: - [x] fix tests - [x] new tests - [x] PR doc --------- Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Somehow paritytech#3795 was merged but tests are failing now on master. I suspect that CI is not even running these tests anymore which is a big issue. --------- Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim added 14 commits March 19, 2024 17:46

DistributeCollation includes CoreIndex

8f5d259

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

SubmitCollationParams includes CoreIndex

dc84796

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

collator-protocol updates

665ef06

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

collation-generation pull support for chained collations

78ee8b2

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

adjust validator buffer capacity for elastic scaling

a61d64a

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

fmt

1cbb090

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Per core index tracking of multiple collations

493a7a2

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

fix tests

8497413

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

enable collator-protocol elastic scaling extension for adder colaltor

0dbc63a

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Use backing state to get candidates pending availability

3906d34

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Merge branch 'master' of github.com:paritytech/polkadot-sdk into sand…

eb7d842

…reim/collate_on_multiple_cores Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

use ok_or

d344a2d

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

subsystem-util request_para_backing_state

16e1cb4

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Use the exposed claim queue

91f9c40

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim changed the title ~~collation-generation: collate on multiple assigned cores~~ collation-generation + collator-protoocl: collate on multiple assigned cores Mar 22, 2024

sandreim changed the title ~~collation-generation + collator-protoocl: collate on multiple assigned cores~~ collation-generation + collator-protocol: collate on multiple assigned cores Mar 22, 2024

skunert reviewed Mar 22, 2024

View reviewed changes

polkadot/node/collation-generation/src/lib.rs Outdated Show resolved Hide resolved

sandreim added 6 commits March 22, 2024 15:41

refactor

154f3dd

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

remove type annotation

60ec79f

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

remove more merge damage

a6c4afe

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

make tests compile

62c13dc

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

happy clippy is good clippy

ddf057e

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

lookahead use core_index in SubmitCollationParams

13b149f

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim added the R0-silent Changes should not be mentioned in any release notes label Mar 25, 2024

sandreim added 4 commits March 25, 2024 11:23

taplo fix

64a53f4

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

enable elastic scaling in undying collator (also enables in zombienet…

732440b

… tests) Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Collation generation test fixes and new tests

f80897d

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Merge branch 'master' of github.com:paritytech/polkadot-sdk into sand…

7d8e6b5

…reim/collate_on_multiple_cores

sandreim marked this pull request as ready for review March 25, 2024 11:51

alindima approved these changes Mar 25, 2024

View reviewed changes

review feedback

9721b69

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

await await

4eb804e

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

eskimor reviewed Mar 26, 2024

View reviewed changes

sandreim mentioned this pull request Mar 26, 2024

Look-ahead collator: build on all assigned cores #3837

Closed

ordian approved these changes Mar 27, 2024

View reviewed changes

cumulus/client/consensus/aura/src/collators/lookahead.rs Show resolved Hide resolved

sandreim added 3 commits March 27, 2024 13:31

review feedback

8b70bc2

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

prdoc

16607dd

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Merge branch 'master' of github.com:paritytech/polkadot-sdk into sand…

ba80669

…reim/collate_on_multiple_cores

eskimor approved these changes Mar 27, 2024

View reviewed changes

polkadot/node/network/collator-protocol/src/collator_side/mod.rs Outdated Show resolved Hide resolved

nice new syntax

69a63cc

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim enabled auto-merge March 27, 2024 14:39

sandreim added this pull request to the merge queue Mar 27, 2024

Merged via the queue into master with commit 417c54c Mar 27, 2024
121 of 132 checks passed

sandreim deleted the sandreim/collate_on_multiple_cores branch March 27, 2024 15:08

sandreim mentioned this pull request Mar 28, 2024

collation-genereation: fix tests #3883

Merged

sandreim mentioned this pull request Apr 8, 2024

Elastic Scaling: Request multiple collations in collator protocol #3582

Closed

sandreim mentioned this pull request Apr 10, 2024

polkadot-parachain v1.10.0 breaks collating on polkadot system chains #4067

Closed

2 tasks

alindima mentioned this pull request May 31, 2024

Fix collation-generation for on-demand/coretime #3327

Closed

TDemeco mentioned this pull request Sep 27, 2024

feat: ⏫ upgrade to Polkadot SDK v1.10.0 Moonsong-Labs/storage-hub#210

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

collation-generation + collator-protocol: collate on multiple assigned cores #3795

collation-generation + collator-protocol: collate on multiple assigned cores #3795

sandreim commented Mar 22, 2024 •

edited

Loading

sandreim commented Mar 25, 2024

alindima left a comment

sandreim commented Mar 25, 2024

sandreim commented Mar 25, 2024

sandreim commented Mar 25, 2024

paritytech-cicd-pr commented Mar 25, 2024

eskimor left a comment

sandreim commented Mar 26, 2024 •

edited

Loading

ordian left a comment

eskimor left a comment

collation-generation + collator-protocol: collate on multiple assigned cores #3795

collation-generation + collator-protocol: collate on multiple assigned cores #3795

Conversation

sandreim commented Mar 22, 2024 • edited Loading

sandreim commented Mar 25, 2024

alindima left a comment

Choose a reason for hiding this comment

sandreim commented Mar 25, 2024

sandreim commented Mar 25, 2024

sandreim commented Mar 25, 2024

paritytech-cicd-pr commented Mar 25, 2024

eskimor left a comment

Choose a reason for hiding this comment

sandreim commented Mar 26, 2024 • edited Loading

ordian left a comment

Choose a reason for hiding this comment

eskimor left a comment

Choose a reason for hiding this comment

sandreim commented Mar 22, 2024 •

edited

Loading

sandreim commented Mar 26, 2024 •

edited

Loading