Unify dependency aliases #4633

ggwpez · 2024-05-29T14:58:06Z

Inherited workspace dependencies cannot be renamed by the crate using them (see 1, 2).
Since we want to use inherited workspace dependencies everywhere, we first need to unify all aliases that we use for a dependency throughout the workspace.
The umbrella crate is currently excluded from this procedure, since it should be able to export the crates by their original name without much hassle.

For example: one crate may alias parity-scale-codec to codec, while another crate does not alias it at all. After this change, all crates have to use codec as name. The problematic combinations were:

conflicting aliases: most crates aliases as A but some use B.
missing alias: most of the crates alias a dep but some dont.
superfluous alias: most crates dont alias a dep but some do.

The script that i used first determines whether most crates opted to alias a dependency or not. From that info it decides whether to use an alias or not. If it decided to use an alias, the most common one is used everywhere.

To reproduce, i used this python script in combination with this error output from Zepter.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

bkchr · 2024-05-29T19:35:49Z

substrate/primitives/consensus/grandpa/src/lib.rs

@@ -35,7 +35,7 @@ use sp_runtime::{
 };

 /// The log target to be used by client code.
-pub const CLIENT_LOG_TARGET: &str = "grandpa";
+pub const CLIENT_LOG_TARGET: &str = "finality_grandpa";


Suggested change

pub const CLIENT_LOG_TARGET: &str = "finality_grandpa";

pub const CLIENT_LOG_TARGET: &str = "grandpa";

Ah, yea it did a string replace. I also replaced the string with quotes, since there are feature gates that we call the same as the dependency.
But i can run it again and print all the cases where it replaced an exact quoted string match.

I checked the diff with git diff -U0 $(git merge-base HEAD^ origin/master) -- ':!**/*.toml' | egrep '"(\w|_|-)+"' and it seems like only a few cases where the string replace messed up.

polkadot/parachain/test-parachains/adder/collator/Cargo.toml

Check with: git diff -U0 $(git merge-base HEAD^ origin/master) -- ':!**/*.toml' | egrep '"(\w|_|-)+"' Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Co-authored-by: Bastian Köcher <git@kchr.de> Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

polkadot/node/subsystem-bench/Cargo.toml

prdoc/pr_4633.prdoc

Co-authored-by: Bastian Köcher <git@kchr.de>

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

franciscoaguirre

Cool script 🚀

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

paritytech-cicd-pr · 2024-06-05T12:21:50Z

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-clippy
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6405457

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* master: (29 commits) Append overlay optimization. (#1223) finalization: Skip tree route calculation if no forks present (#4721) Remove unncessary call remove_from_peers_set (#4742) add pov-recovery unit tests and support for elastic scaling (#4733) approval-voting: Add no shows debug information (#4726) Revamp the Readme of the parachain template (#4713) Update README.md to move the PSVM link under a "Tooling" section under the "Releases" section (#4734) frame/proc-macro: Refactor code for better readability (#4712) Contracts: update wasmi to 0.32 (#3679) Backport style changes from P<>K bridge to R<>W bridge (#4732) New reference doc for Custom RPC V2 (#4654) Frame Pallets: Clean a lot of test setups (#4642) Fix occupied core handling (#4691) statement-distribution: Fix false warning (#4727) Update the README to include a link to the Polkadot SDK Version Manager (#4718) Cleanup PVF artifact by cache limit and stale time (#4662) Update link to a latest polkadot release (#4711) [CI] Delete cargo-deny config (#4677) fix build on MacOS: bump secp256k1 and secp256k1-sys to patched versions (#4709) Unify dependency aliases (#4633) ...

After preparing in #4633, we can lift also all internal dependencies up to the workspace. This does not actually change anything, but uses `workspace = true` for all dependencies. You can check it with: ```bash git checkout -q $(git merge-base oty-lift-all-deps origin/master) cargo tree -e features > master.out git checkout -q oty-lift-all-deps cargo tree -e features > new.out diff master.out new.out ``` It did not yet lift 100% of dependencies, some inside of `target.*` or some that had conflicting aliases introduced recently. But i will do these together in a follow-up with CI checks. Can be reproduced with [zepter](https://github.com/ggwpez/zepter/): `zepter transpose d lift-to-workspace "regex:.*" --version-resolver highest --skip-package "polkadot-sdk" --ignore-errors --fix`. --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Inherited workspace dependencies cannot be renamed by the crate using them (see [1](rust-lang/cargo#12546), [2](https://stackoverflow.com/questions/76792343/can-inherited-dependencies-in-rust-be-aliased-in-the-cargo-toml-file)). Since we want to use inherited workspace dependencies everywhere, we first need to unify all aliases that we use for a dependency throughout the workspace. The umbrella crate is currently excluded from this procedure, since it should be able to export the crates by their original name without much hassle. For example: one crate may alias `parity-scale-codec` to `codec`, while another crate does not alias it at all. After this change, all crates have to use `codec` as name. The problematic combinations were: - conflicting aliases: most crates aliases as `A` but some use `B`. - missing alias: most of the crates alias a dep but some dont. - superfluous alias: most crates dont alias a dep but some do. The script that i used first determines whether most crates opted to alias a dependency or not. From that info it decides whether to use an alias or not. If it decided to use an alias, the most common one is used everywhere. To reproduce, i used [this](https://github.com/ggwpez/substrate-scripts/blob/master/uniform-crate-alias.py) python script in combination with [this](https://github.com/ggwpez/zepter/blob/38ad10585fe98a5a86c1d2369738bc763a77057b/renames.json) error output from Zepter. --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de>

After preparing in paritytech#4633, we can lift also all internal dependencies up to the workspace. This does not actually change anything, but uses `workspace = true` for all dependencies. You can check it with: ```bash git checkout -q $(git merge-base oty-lift-all-deps origin/master) cargo tree -e features > master.out git checkout -q oty-lift-all-deps cargo tree -e features > new.out diff master.out new.out ``` It did not yet lift 100% of dependencies, some inside of `target.*` or some that had conflicting aliases introduced recently. But i will do these together in a follow-up with CI checks. Can be reproduced with [zepter](https://github.com/ggwpez/zepter/): `zepter transpose d lift-to-workspace "regex:.*" --version-resolver highest --skip-package "polkadot-sdk" --ignore-errors --fix`. --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

closes #849 ## Context For the background on this and the long-term fix, see #849 (comment). ## Changes * The weigh files are renamed from `runtime_(parachains|common).*` to `polkadot_runtime_(parachains|common).*`. The reason for it is the renaming introduced in #4633. The new weight command and files are generated now include `polkadot_` prefix. * The WeightInfo for `paras_inherent` now includes `enter_empty` which calculates the cost of processing an empty parachains inherent. This cost is subtracted dynamically when calculating other weights (so the other weights remain the same) ## Benefits See #849 (comment), but TL;DR is that we are not blocked on weights for scaling the number of validators and cores further. Resolved questions: - [x] why new benchmarks for westend are doing fewer db IOPS? Is it due polkadot-sdk update (db IOPS diff)? or the bench setup is no longer valid? https://github.com/polkadot-fellows/runtimes/blob/7723274a2c5cbb10213379271094d5180716ca7d/relay/polkadot/src/weights/runtime_parachains_paras_inherent.rs#L131-L196 Answer: see background section of #5270 TODOs: - [x] Rerun benchmarks for Rococo and Westend - [x] PRDoc --------- Co-authored-by: command-bot <>

Inherited workspace dependencies cannot be renamed by the crate using them (see [1](rust-lang/cargo#12546), [2](https://stackoverflow.com/questions/76792343/can-inherited-dependencies-in-rust-be-aliased-in-the-cargo-toml-file)). Since we want to use inherited workspace dependencies everywhere, we first need to unify all aliases that we use for a dependency throughout the workspace. The umbrella crate is currently excluded from this procedure, since it should be able to export the crates by their original name without much hassle. For example: one crate may alias `parity-scale-codec` to `codec`, while another crate does not alias it at all. After this change, all crates have to use `codec` as name. The problematic combinations were: - conflicting aliases: most crates aliases as `A` but some use `B`. - missing alias: most of the crates alias a dep but some dont. - superfluous alias: most crates dont alias a dep but some do. The script that i used first determines whether most crates opted to alias a dependency or not. From that info it decides whether to use an alias or not. If it decided to use an alias, the most common one is used everywhere. To reproduce, i used [this](https://github.com/ggwpez/substrate-scripts/blob/master/uniform-crate-alias.py) python script in combination with [this](https://github.com/ggwpez/zepter/blob/38ad10585fe98a5a86c1d2369738bc763a77057b/renames.json) error output from Zepter. --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de>

After preparing in paritytech#4633, we can lift also all internal dependencies up to the workspace. This does not actually change anything, but uses `workspace = true` for all dependencies. You can check it with: ```bash git checkout -q $(git merge-base oty-lift-all-deps origin/master) cargo tree -e features > master.out git checkout -q oty-lift-all-deps cargo tree -e features > new.out diff master.out new.out ``` It did not yet lift 100% of dependencies, some inside of `target.*` or some that had conflicting aliases introduced recently. But i will do these together in a follow-up with CI checks. Can be reproduced with [zepter](https://github.com/ggwpez/zepter/): `zepter transpose d lift-to-workspace "regex:.*" --version-resolver highest --skip-package "polkadot-sdk" --ignore-errors --fix`. --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

ggwpez added 9 commits May 27, 2024 14:29

DNM: disable umbrella crate

5533ffe

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

DNM: Manual fix

02d5122

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Manual rename

5610dc5

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Manual fixes

40fa5f3

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Merge remote-tracking branch 'origin/master' into oty-uniform-dep-alias

d0db9d7

Auto fix

62d9724

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Manual fix

d32966b

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

fmt

d28bce5

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Manual fix

0ed8e32

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

ggwpez requested review from athei, andresilva and a team as code owners May 29, 2024 14:58

paritytech-review-bot bot requested a review from a team May 29, 2024 14:58

ggwpez marked this pull request as draft May 29, 2024 15:01

ggwpez added 2 commits May 29, 2024 17:09

Toml format

11320d5

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Fix test

46a7f8d

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

ggwpez added R0-silent Changes should not be mentioned in any release notes and removed R0-silent Changes should not be mentioned in any release notes labels May 29, 2024

bkchr reviewed May 29, 2024

View reviewed changes

ggwpez added 2 commits May 31, 2024 16:44

Undo some string replacements

b6620c2

Check with: git diff -U0 $(git merge-base HEAD^ origin/master) -- ':!**/*.toml' | egrep '"(\w|_|-)+"' Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Manuall fix

c4204a6

Co-authored-by: Bastian Köcher <git@kchr.de> Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

ggwpez marked this pull request as ready for review May 31, 2024 16:55

ggwpez added 3 commits May 31, 2024 19:01

Manuall fix

42115da

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

fmt

4b7f9d1

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Fix compile

a974d13

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

ggwpez added the T8-polkadot This PR/Issue is related to/affects the Polkadot network. label Jun 3, 2024

ggwpez added 2 commits June 3, 2024 12:59

Test with prdoc

3b4a279

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Fix docs

a9386e9

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

bkchr approved these changes Jun 4, 2024

View reviewed changes

polkadot/node/subsystem-bench/Cargo.toml Outdated Show resolved Hide resolved

prdoc/pr_4633.prdoc Outdated Show resolved Hide resolved

ggwpez and others added 2 commits June 4, 2024 23:59

Update prdoc/pr_4633.prdoc

23e7928

Co-authored-by: Bastian Köcher <git@kchr.de>

Unalias pyro

59d7034

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

franciscoaguirre approved these changes Jun 5, 2024

View reviewed changes

kianenigma approved these changes Jun 5, 2024

View reviewed changes

bkontur approved these changes Jun 5, 2024

View reviewed changes

Fix doctests

5767a6c

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

ggwpez enabled auto-merge June 5, 2024 12:15

ggwpez added 2 commits June 5, 2024 14:32

🤦

4c035b8

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

doctest...

7b43b07

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

ggwpez added this pull request to the merge queue Jun 5, 2024

Merged via the queue into master with commit d2fd536 Jun 5, 2024
156 checks passed

ggwpez deleted the oty-uniform-dep-alias branch June 5, 2024 15:07

ggwpez mentioned this pull request Jun 5, 2024

Lift all dependencies (the big one) #4716

Merged

ordian mentioned this pull request Aug 7, 2024

short-term fix for para inherent weight overestimation #5082

Merged

3 tasks

This was referenced Aug 21, 2024

Update polkadot-sdk from v1.11.0 to stable2407 moondance-labs/tanssi#659

Closed

Update polkadot-sdk from v1.11.0 to stable2407 moonbeam-foundation/moonbeam#2912

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unify dependency aliases #4633

Unify dependency aliases #4633

ggwpez commented May 29, 2024 •

edited

Loading

bkchr May 29, 2024

ggwpez May 30, 2024

ggwpez Jun 3, 2024

franciscoaguirre left a comment

paritytech-cicd-pr commented Jun 5, 2024

	pub const CLIENT_LOG_TARGET: &str = "finality_grandpa";
	pub const CLIENT_LOG_TARGET: &str = "grandpa";

Unify dependency aliases #4633

Unify dependency aliases #4633

Conversation

ggwpez commented May 29, 2024 • edited Loading

bkchr May 29, 2024

Choose a reason for hiding this comment

ggwpez May 30, 2024

Choose a reason for hiding this comment

ggwpez Jun 3, 2024

Choose a reason for hiding this comment

franciscoaguirre left a comment

Choose a reason for hiding this comment

paritytech-cicd-pr commented Jun 5, 2024

ggwpez commented May 29, 2024 •

edited

Loading