Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

collation-generation: use v2 receipts #5908

Merged
merged 159 commits into from
Nov 4, 2024

Conversation

alindima
Copy link
Contributor

@alindima alindima commented Oct 2, 2024

Part of #5047

Plus some cleanups

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
sandreim and others added 11 commits October 17, 2024 16:24
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
…reim/node_v2_descriptors

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
…to alindima/collation-generation-v2-receipts
Base automatically changed from sandreim/node_v2_descriptors to master October 25, 2024 16:12
@tdimitrov tdimitrov self-requested a review October 29, 2024 06:35
Copy link
Contributor

@sandreim sandreim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! just a few questions and suggestion before approval.

#[case(0)]
#[case(1)]
#[case(2)]
fn distribute_collation_for_free_cores_with_async_backing_enabled_and_elastic_scaling(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still have a test that covers this ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Since we no longer care about occupied cores, this test and distribute_collation_for_occupied_cores_with_async_backing_enabled_and_elastic_scaling have been merged into distribute_collation_with_elastic_scaling

#[rstest]
#[case(RuntimeApiRequest::CLAIM_QUEUE_RUNTIME_REQUIREMENT - 1)]
#[case(RuntimeApiRequest::CLAIM_QUEUE_RUNTIME_REQUIREMENT)]
fn requests_validation_data_for_scheduled_matches(#[case] runtime_version: u32) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the reasoning to remove this test ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a test that verifies the handling of the active leaves, assuming we can have multiple activations at once. This is no longer true (probably used to be)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name of the test was misleading in this case.

descriptor: CandidateDescriptorV2::new(
para_id,
relay_parent,
core_index,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can simplify the collator code and also avoid checking the core index here if we use commitments.committed_core_index(). We need to however use the PreparedCollation core index if there is no core index commitment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how would this simplify the collator code?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean we don't have to look into the commitments because we just do it here when assembling the committed candidate receipt.

polkadot/node/collation-generation/src/lib.rs Outdated Show resolved Hide resolved
polkadot/node/collation-generation/src/lib.rs Show resolved Hide resolved
polkadot/node/collation-generation/src/lib.rs Show resolved Hide resolved
},
Err(e @ RuntimeApiError::Execution { .. }) => Err(e.into()),
}
Ok(())
}

fn erasure_root(
n_validators: usize,
persisted_validation: PersistedValidationData,
pov: PoV,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the above comment, this should be receipt.core_index().unwrap_or(core_index)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand how this helps. We are creating the receipt just above with this core index

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main point here is that we should pick the core index that the candidate commits to. There is a bit of confusion since there are 2 core indices, one from the PreparedCollation and potentially another one in the commitments. If we are using v2 receipts, the collator code retrieves the core index from the commitments and puts it in the SubmitCollationParams and then we check it once again later here. This is likely prone to errors if the core_index is different from the one passed in the commitments. Maybe we should have a separate SubmitCollationParamsV2 that doesn't have a core_index field at all (which should be used when the candidate commits to a core_index), so there is no confusion.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok. what you're saying seems doable. However, I'm not sure it simplifies things.
We'd need to handle two message types here in collation-generation and also decide which message type to use in the collator based on the node feature and whether the core index commitment is present. I'm not sure it simplifies things.
Would be an improvement idea for when we completely switch to v2 receipts and make the commitment mandatory

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The commitments being optional indeed introduces more complexity, but for v2 receipts, the collator doesn't actually need to know the core index, just the amount of cores assigned. We don't even need a runtime API to get the next core selector, we just build a block and we look at the commitments, if there is a core committed, we just push the collation to collation generation. We can create a ticket to handle this at a later time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I remember why I coded it this way. What you're suggesting has the caveat that we're breaking the generic nature of the core selection policy of the parachain runtime, since we assume that the runtime will choose a core that is assigned. If it won't, it would have built a block and only after that see that the selected core is not assigned.

polkadot/node/collation-generation/src/lib.rs Outdated Show resolved Hide resolved
polkadot/node/collation-generation/src/lib.rs Outdated Show resolved Hide resolved
@alindima alindima added this pull request to the merge queue Nov 4, 2024
Merged via the queue into master with commit 68e0563 Nov 4, 2024
207 of 208 checks passed
@alindima alindima deleted the alindima/collation-generation-v2-receipts branch November 4, 2024 08:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T8-polkadot This PR/Issue is related to/affects the Polkadot network.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants