Skip to content

Commit

Permalink
Merge branch 'master' into ao-fix-backing-filtering-on-disputes
Browse files Browse the repository at this point in the history
* master: (167 commits)
  Upgrade accidentally downgraded deps (#5365)
  [Pools] Fix issues with member migration to `DelegateStake` (#4822)
  Unify `no_genesis` check (#5360)
  [CI] Fix prdoc command (#5358)
  Beefy: add benchmarks for `report_fork_voting()` (#5188)
  Fix OurViewChange small race (#5356)
  Make ticket non-optional and add ensure_successful method to Consideration trait (#5359)
  [tests] dedup test code, add more tests, improve naming and docs (#5338)
  Stop running the wishlist workflow on forks (#5297)
  Migrate foreign assets v3::Location to v4::Location (#4129)
  Minor clean up (#5284)
  [Pools] Ensure members can always exit the pool gracefully (#4998)
  StorageWeightReclaim: set to node pov size if higher (#5281)
  [Bot] Add prdoc generation (#5331)
  Small nits found accidentally along the way (#5341)
  Create subsystem-benchmarks.yml (#5325)
  Bump libp2p-identity from 0.2.8 to 0.2.9 (#5232)
  Bump authoring duration for async backing to 2s. (#5195)
  Fix spelling issues (#5206)
  Bump the known_good_semver group across 1 directory with 3 updates (#5315)
  ...
  • Loading branch information
ordian committed Aug 14, 2024
2 parents 683d0ea + 53f4274 commit 55f5782
Show file tree
Hide file tree
Showing 1,093 changed files with 32,091 additions and 13,240 deletions.
2 changes: 1 addition & 1 deletion .config/zepter.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ workflows:
]
# The umbrella crate uses more features, so we to check those too:
check_umbrella:
- [ $check.0, '--features=serde,experimental,with-tracing,tuples-96,with-tracing', '-p=polkadot-sdk' ]
- [ $check.0, '--features=serde,experimental,runtime,with-tracing,tuples-96,with-tracing', '-p=polkadot-sdk' ]
# Same as `check_*`, but with the `--fix` flag.
default:
- [ $check.0, '--fix' ]
Expand Down
76 changes: 76 additions & 0 deletions .github/commands-readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ The current available command actions are:

- [Command FMT](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-fmt.yml)
- [Command Update UI](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-update-ui.yml)
- [Command Prdoc](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-prdoc.yml)
- [Command Sync](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-sync.yml)
- [Command Bench](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-bench.yml)
- [Command Bench All](https://github.com/paritytech/polkadot-sdk/actions/workflows/command-bench-all.yml)
Expand All @@ -25,12 +26,36 @@ Each command will have the same two required values, but it could have more.

GitHub's official documentation: [Manually running a workflow](https://docs.github.com/en/actions/using-workflows/manually-running-a-workflow)

#### Running from CLI

You can use [`gh cli`](https://cli.github.com/) to run the commands too. Refers to the [`gh workflow run`](https://cli.github.com/manual/gh_workflow_run) section from the documentation for more information.

### Number of the Pull Request

The number of the pull request. Required so the action can fetch the correct branch and comment if it fails.

## Action configurations

### FMT

For FMT you only need the PR number.

You can use the following [`gh cli`](https://cli.github.com/) inside the repo:

```bash
gh workflow run command-fmt.yml -f pr=1000
```

### Update UI

For Update UI you only need the PR number.

You can use the following [`gh cli`](https://cli.github.com/) inside the repo:

```bash
gh workflow run command-update-ui.yml -f pr=1000
```

### Bench

Runs `benchmark pallet` or `benchmark overhead` against your PR and commits back updated weights.
Expand Down Expand Up @@ -136,6 +161,12 @@ Posible combinations based on the `benchmark` dropdown.
- Requires `Runtime Dir` to be `testing`
- Requires `Target Directory` to be `cumulus`

You can use the following [`gh cli`](https://cli.github.com/) inside the repo:

```bash
gh workflow run command-bench.yml -f pr=1000 -f benchmark=polkadot-pallet -f subcommand=pallet -f runtime=rococo -f pallet=pallet_name -f target_dir=polkadot
```

### Bench-all

This is a wrapper to run `bench` for all pallets.
Expand Down Expand Up @@ -174,6 +205,12 @@ Posible combinations based on the `benchmark` dropdown.
- `people-westend`
- Requires `Target Directory` to be `cumulus`

You can use the following [`gh cli`](https://cli.github.com/) inside the repo:

```bash
gh workflow run command-bench-all.yml -f pr=1000 -f benchmark=pallet -f pallet=pallet_name -f target_dir=polkadot -f runtime=rococo
```

### Bench-overhead

Run benchmarks overhead and commit back results to PR.
Expand All @@ -193,6 +230,45 @@ Posible combinations based on the `benchmark` dropdown.
- `asset-hub-westend`
- Requires `Target directory` to be `cumulus`

You can use the following [`gh cli`](https://cli.github.com/) inside the repo:

```bash
gh workflow run command-bench-overheard.yml -f pr=1000 -f benchmark=substrate -f runtime=rococo -f target_dir=substrate
```

### PrDoc

Generate a PrDoc with the crates populated by all modified crates.

Options:
- `pr`: The PR number to generate the PrDoc for.
- `audience`: The audience of whom the changes may concern.
- `bump`: A default bump level for all crates. The PrDoc will likely need to be edited to reflect the actual changes after generation.
- `overwrite`: Whether to overwrite any existing PrDoc.

### Sync

Run sync and commit back results to PR.

Posible combinations based on the `benchmark` dropdown.

- `chain`
- Requires one of the following:
- `rococo`
- `westend`
- `sync-type`
- Requires one of the following:
- `warp`
- `full`
- `fast`
- `fast-unsafe`

You can use the following [`gh cli`](https://cli.github.com/) inside the repo:

```bash
gh workflow run command-sync.yml -f pr=1000 -f chain=rococo -f sync-type=full
```

## How to modify an action

If you want to modify an action and test it, you can do by simply pushing your changes and then selecting your branch in the `Use worflow from` option.
Expand Down
8 changes: 6 additions & 2 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,17 @@ updates:
directory: '/'
labels: ["A1-insubstantial", "R0-silent"]
schedule:
interval: daily
interval: weekly
groups:
ci_dependencies:
patterns:
- "*"
# Update Rust dependencies:
- package-ecosystem: "cargo"
directory: "/"
labels: ["A1-insubstantial", "R0-silent"]
schedule:
interval: "daily"
interval: "weekly"
groups:
# We assume these crates to be semver abiding and can therefore group them together.
known_good_semver:
Expand Down
26 changes: 26 additions & 0 deletions .github/scripts/common/lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,7 @@ function import_gpg_keys() {
) &
done
wait
gpg -k $SEC
}

# Check the GPG signature for a given binary
Expand Down Expand Up @@ -444,3 +445,28 @@ get_latest_release_tag() {
latest_release_tag=$(curl -s -H "$TOKEN" $api_base/paritytech/polkadot-sdk/releases/latest | jq -r '.tag_name')
printf $latest_release_tag
}

function get_polkadot_node_version_from_code() {
# list all the files with node version
git grep -e "\(NODE_VERSION[^=]*= \)\".*\"" |
# fetch only the one we need
grep "primitives/src/lib.rs:" |
# Print only the version
awk '{ print $7 }' |
# Remove the quotes
sed 's/"//g' |
# Remove the semicolon
sed 's/;//g'
}

validate_stable_tag() {
tag="$1"
pattern='^stable[0-9]+(-[0-9]+)?$'

if [[ $tag =~ $pattern ]]; then
echo $tag
else
echo "The input '$tag' does not match the pattern."
exit 1
fi
}
112 changes: 112 additions & 0 deletions .github/scripts/generate-prdoc.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
#!/usr/bin/env python3

"""
Generate the PrDoc for a Pull Request with a specific number, audience and bump level.
It downloads and parses the patch from the GitHub API to opulate the prdoc with all modified crates.
This will delete any prdoc that already exists for the PR if `--force` is passed.
Usage:
python generate-prdoc.py --pr 1234 --audience "TODO" --bump "TODO"
"""

import argparse
import os
import re
import sys
import subprocess
import toml
import yaml
import requests

from github import Github
import whatthepatch
from cargo_workspace import Workspace

# Download the patch and pass the info into `create_prdoc`.
def from_pr_number(n, audience, bump, force):
print(f"Fetching PR '{n}' from GitHub")
g = Github()

repo = g.get_repo("paritytech/polkadot-sdk")
pr = repo.get_pull(n)

patch_url = pr.patch_url
patch = requests.get(patch_url).text

create_prdoc(n, audience, pr.title, pr.body, patch, bump, force)

def create_prdoc(pr, audience, title, description, patch, bump, force):
path = f"prdoc/pr_{pr}.prdoc"

if os.path.exists(path):
if force == True:
print(f"Overwriting existing PrDoc for PR {pr}")
else:
print(f"PrDoc already exists for PR {pr}. Use --force to overwrite.")
sys.exit(1)
else:
print(f"No preexisting PrDoc for PR {pr}")

prdoc = { "doc": [{}], "crates": [] }

prdoc["title"] = title
prdoc["doc"][0]["audience"] = audience
prdoc["doc"][0]["description"] = description

workspace = Workspace.from_path(".")

modified_paths = []
for diff in whatthepatch.parse_patch(patch):
modified_paths.append(diff.header.new_path)

modified_crates = {}
for p in modified_paths:
# Go up until we find a Cargo.toml
p = os.path.join(workspace.path, p)
while not os.path.exists(os.path.join(p, "Cargo.toml")):
p = os.path.dirname(p)

with open(os.path.join(p, "Cargo.toml")) as f:
manifest = toml.load(f)

if not "package" in manifest:
print(f"File was not in any crate: {p}")
continue

crate_name = manifest["package"]["name"]
if workspace.crate_by_name(crate_name).publish:
modified_crates[crate_name] = True
else:
print(f"Skipping unpublished crate: {crate_name}")

print(f"Modified crates: {modified_crates.keys()}")

for crate_name in modified_crates.keys():
entry = { "name": crate_name }

if bump == 'silent' or bump == 'ignore' or bump == 'no change':
entry["validate"] = False
else:
entry["bump"] = bump

print(f"Adding crate {entry}")
prdoc["crates"].append(entry)

# write the parsed PR documentation back to the file
with open(path, "w") as f:
yaml.dump(prdoc, f)

def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument("--pr", type=int, required=True)
parser.add_argument("--audience", type=str, default="TODO")
parser.add_argument("--bump", type=str, default="TODO")
parser.add_argument("--force", type=str)
return parser.parse_args()

if __name__ == "__main__":
args = parse_args()
force = True if args.force.lower() == "true" else False
print(f"Args: {args}, force: {force}")
from_pr_number(args.pr, args.audience, args.bump, force)
79 changes: 79 additions & 0 deletions .github/scripts/update-wishlist-leaderboard.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
from github import Github
import re
import os
from datetime import date

g = Github(os.getenv("GH_TOKEN"))

# Regex pattern to match wish format:
wish_pattern = re.compile(
r"I wish for:? (https://github\.com/([a-zA-Z0-9_.-]+)/([a-zA-Z0-9_.-]+)/(issues|pull)/(\d+))"
)

wishlist_issue = g.get_repo(os.getenv("WISHLIST_REPOSITORY")).get_issue(
int(os.getenv("WISHLIST_ISSUE_NUMBER"))
)
new_leaderboard = (
"| Feature Request | Summary | Votes | Status |\n| --- | --- | --- | --- |\n"
)
wishes = {}
issue_details = {}

for comment in wishlist_issue.get_comments():
# in the comment body, if there is a string `#(\d)`, replace it with
# https://github.com/paritytech/polkadot-sdk/issues/(number)
updated_body = re.sub(
r"#(\d+)", r"https://github.com/paritytech/polkadot-sdk/issues/\1", comment.body
)

matches = wish_pattern.findall(updated_body)
for match in matches:
url, org, repo_name, _, issue_id = match
issue_key = (url, org, repo_name, issue_id)
if issue_key not in wishes:
wishes[issue_key] = []

# Get the author and upvoters of the wish comment.
wishes[issue_key].append(comment.user.id)
wishes[issue_key].extend(
[
reaction.user.id
for reaction in comment.get_reactions()
if reaction.content in ["+1", "heart", "rocket"]
]
)

# Get upvoters of the desired issue.
desired_issue = g.get_repo(f"{org}/{repo_name}").get_issue(int(issue_id))
wishes[issue_key].extend(
[
reaction.user.id
for reaction in desired_issue.get_reactions()
if reaction.content in ["+1", "heart", "rocket"]
]
)
issue_details[url] = [
desired_issue.title,
"👾 Open" if desired_issue.state == "open" else "✅Closed",
]

# Count unique wishes - the author of the wish, upvoters of the wish, and upvoters of the desired issue.
for key in wishes:
wishes[key] = len(list(set(wishes[key])))

# Sort wishes by count and add to the markdown table
sorted_wishes = sorted(wishes.items(), key=lambda x: x[1], reverse=True)
for (url, _, _, _), count in sorted_wishes:
[summary, status] = issue_details.get(url, "No summary available")
new_leaderboard += f"| {url} | {summary} | {count} | {status} |\n"
new_leaderboard += f"\n> Last updated: {date.today().strftime('%Y-%m-%d')}\n"
print(new_leaderboard)

new_content = re.sub(
r"(\| Feature Request \|)(.*?)(> Last updated:)(.*?\n)",
new_leaderboard,
wishlist_issue.body,
flags=re.DOTALL,
)

wishlist_issue.edit(body=new_content)
Loading

0 comments on commit 55f5782

Please sign in to comment.