Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

All-benchmarks script and CI workflow improvements #758

Merged
merged 8 commits into from
Aug 25, 2022

Conversation

ghzlatarev
Copy link
Contributor

@ghzlatarev ghzlatarev commented Aug 17, 2022

Signed-off-by: Georgi Zlatarev georgi.zlatarev@manta.network

Description

closes: #759

  • Adds a run_all_benchmarks.sh script which runs pallet, storage and machine benchmark and outputs results, same as this Parity script Create Script to Run All Benchmarks paritytech/substrate#11493

  • Adds new CI workflow which calls the script and replaces the 3 duplicated weights benchmarking workflows for dolphin, calamari and manta.

  • The chain-spec is a mandatory input field on the new workflow.

  • The storage benchmark can be skipped if no URL is provided for the second input field that specifies a tar.gz archive of a production chain snapshot.

  • Changed all CI workflows to always try to upload artifacts even if previous steps have failed, which is useful for debugging failed jobs.

  • Disable parachain block parsing until polkadot-js recognizes we that our chain is a special case like Moonbeam with nimbus keys.

  • Update release process template to account for aforementioned changes.

  • Example run - https://github.com/Manta-Network/Manta/runs/7880566445?check_suite_focus=true


Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

  • Linked to Github issue with discussion and accepted design OR have an explanation in the PR that describes this work.
  • Added one label out of the L- group to this PR
  • Added one or more labels out of A- and C- groups to this PR
  • Re-reviewed Files changed in the Github PR explorer.

Situational Notes:

  • If adding functionality, write unit tests!
  • If importing a new pallet, choose a proper module index for it, and allow it in BaseFilter. Ensure every extrinsic works from front-end. If there's corresponding tool, ensure both work for each other.
  • If needed, update our Javascript/Typescript APIs. These APIs are officially used by exchanges or community developers.
  • If modifying existing runtime storage items, make sure to implement storage migrations for the runtime and test them with try-runtime. This includes migrations inherited from upstream changes, and you can search the diffs for modifications of #[pallet::storage] items to check for any.
  • If runtime changes, need to update the version numbers properly:
    • authoring_version: The version of the authorship interface. An authoring node will not attempt to author blocks unless this is equal to its native runtime.
    • spec_version: The version of the runtime specification. A full node will not attempt to use its native runtime in substitute for the on-chain Wasm runtime unless all of spec_name, spec_version, and authoring_version are the same between Wasm and native.
    • impl_version: The version of the implementation of the specification. Nodes are free to ignore this; it serves only as an indication that the code is different; as long as the other two versions are the same then while the actual code may be different, it is nonetheless required to do the same thing. Non-consensus-breaking optimizations are about the only changes that could be made which would result in only the impl_version changing.
    • transaction_version: The version of the extrinsics interface. This number must be updated in the following circumstances: extrinsic parameters (number, order, or types) have been changed; extrinsics or pallets have been removed; or the pallet order in the construct_runtime! macro or extrinsic order in a pallet has been changed. You can run the metadata_diff.yml workflow for help. If this number is updated, then the spec_version must also be updated
  • Verify benchmarks & weights have been updated for any modified runtime logics

…ven if previous steps failed in all CI workflows. Disable parachain block time parsing temporarily

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
@ghzlatarev ghzlatarev self-assigned this Aug 17, 2022
@ghzlatarev ghzlatarev added A-ci Area: Continuous Integration L-added Log: Issues and PRs related to addition labels Aug 17, 2022
@ghzlatarev ghzlatarev marked this pull request as ready for review August 17, 2022 16:52
@Garandor Garandor changed the title All-benchmarks script and CI workflow improvlemetns All-benchmarks script and CI workflow improvemetns Aug 19, 2022
Apokalip
Apokalip previously approved these changes Aug 19, 2022
Copy link
Contributor

@Apokalip Apokalip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty good to me! Now it looks a little cleaner than before and the tests' always() is a good change as we would want to see not only until the failing test

Copy link
Contributor

@Garandor Garandor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing a C- label

@ghzlatarev ghzlatarev added the C-enhancement Category: An issue proposing an enhancement or a PR with one label Aug 19, 2022
@ghzlatarev ghzlatarev requested a review from Garandor August 19, 2022 14:36
@Apokalip
Copy link
Contributor

Apokalip commented Aug 19, 2022

Missing a C- label

@Garandor
We should probably add the other labels in the label checker workflow? I keep forgetting about them too

Garandor
Garandor previously approved these changes Aug 19, 2022
Copy link
Contributor

@Garandor Garandor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, maybe get rid of the yamllint warning before merge @ghzlatarev

@Garandor Garandor changed the title All-benchmarks script and CI workflow improvemetns All-benchmarks script and CI workflow improvements Aug 19, 2022
@Garandor Garandor modified the milestones: v3.3.0, v3.4.0 Aug 19, 2022
Dengjianping
Dengjianping previously approved these changes Aug 22, 2022
Copy link
Contributor

@Dengjianping Dengjianping left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but there's conflict to be resolved.

Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
@ghzlatarev ghzlatarev dismissed stale reviews from Dengjianping, Garandor, and Apokalip via e2883a8 August 23, 2022 06:43
Signed-off-by: Georgi Zlatarev <georgi.zlatarev@manta.network>
@ghzlatarev ghzlatarev merged commit 8339372 into manta Aug 25, 2022
@ghzlatarev ghzlatarev deleted the ghzlatarev/all-benchmarks-2 branch August 25, 2022 07:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-ci Area: Continuous Integration C-enhancement Category: An issue proposing an enhancement or a PR with one L-added Log: Issues and PRs related to addition
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Integrate new benchmarks and improve benchmarks CI
4 participants