Add an auto-cherry-pick GitHub Action #19021

thejcannon · 2023-05-17T02:02:20Z

This adds a new GitHub action that runs after a PR is merged which is labeled "needs-cherrypick" as well as can be run manually.

I've done a decent amount of testing on my fork, but wasn't able to test the actual PR creation (it failed on me not configuring git over there)

huonw

Woohoo, automation!

build-support/bin/cherry_pick.sh

.github/workflows/auto-cherry-picker.yaml

huonw · 2023-05-17T03:03:26Z

.github/workflows/auto-cherry-picker.yaml

+          git config --local user.email "pantsbuild+github-automation@gmail.com"
+          git config --local user.name "Worker Pants (Pantsbuild GitHub Automation Bot) "


If this is happening in several automations, it might be good to centralise, and I see two options:

they could be factored into a composite action in this repo (https://docs.github.com/en/actions/creating-actions/about-custom-actions#choosing-a-location-for-your-action)

the email/name content could be stored as a repo/org level 'variable' https://docs.github.com/en/actions/learn-github-actions/variables#defining-configuration-variables-for-multiple-workflows rather than hard-code

I haven't tried it yet, but I suspect there's just a missing env var to handle it

.github/workflows/auto-cherry-picker.yaml

thejcannon · 2023-05-17T15:12:47Z

Oh sorry @huonw I just realized this branch wasn't up to date with my latest changes 😬

thejcannon · 2023-05-17T16:11:14Z

build-support/bin/cherry_pick.sh

@@ -25,13 +27,11 @@ if [[ -z $PR_NUM ]]; then
  read -r PR_NUM
 fi

-TARGET_MILESTONE=$2


The milestone changes should've been made on the "every milestone" PR, sorry y'all

thejcannon · 2023-05-17T16:11:38Z

build-support/bin/cherry_pick.sh

@@ -51,6 +51,7 @@ COMMIT=$(gh pr view "$PR_NUM" --json mergeCommit --jq '.mergeCommit.oid')
 if [[ -z $COMMIT ]]; then
  fail "Wasn't able to retrieve merge commit for $PR_NUM."
 fi
+git fetch https://github.com/pantsbuild/pants "$COMMIT"


This means the person running the script doesn't need an up-to-date main, we'll fetch the ref.

thejcannon · 2023-05-17T16:12:02Z

build-support/bin/cherry_pick.sh

+    if [[ -n $REMOTE ]]; then
+      git push -u "$REMOTE" "$BRANCH_NAME"
+    fi


Since the GitHub Action doesn't get to participate in prompting, we need to push the branch.

I assume the GitHub cli won't even try to prompt in this case, it'll detect the branch already exists on a remote?

thejcannon · 2023-05-17T16:12:24Z

.github/workflows/auto-cherry-picker.yaml

+          git config --local user.email "pantsbuild+github-automation@gmail.com"
+          git config --local user.name "Worker Pants (Pantsbuild GitHub Automation Bot)"


Ugh: https://github.com/actions/checkout/blob/f095bcc56b7c2baf48f3ac70d6d6782f4f553222/adrs/0153-checkout-v2.md?plain=1#L111

stuhood

Thanks!

Ideally this would be generated by

pants/build-support/bin/generate_github_workflows.py

Lines 1071 to 1173 in f9188ab

    
           def generate() -> dict[Path, str]: 
        
               """Generate all YAML configs with repo-relative paths.""" 
        
               pr_jobs = test_workflow_jobs([PYTHON37_VERSION], cron=False) 
        
               pr_jobs.update(**classify_changes()) 
        
               for key, val in pr_jobs.items(): 
        
                   if key in {"check_labels", "classify_changes"}: 
        
                       continue 
        
                   needs = val.get("needs", []) 
        
                   if isinstance(needs, str): 
        
                       needs = [needs] 
        
                   needs.extend(["classify_changes"]) 
        
                   val["needs"] = needs 
        
                   if_cond = val.get("if") 
        
                   not_docs_only = "needs.classify_changes.outputs.docs_only != 'true'" 
        
                   val["if"] = not_docs_only if if_cond is None else f"({if_cond}) && ({not_docs_only})" 
        
               pr_jobs.update(merge_ok(sorted(pr_jobs.keys()))) 
        
               test_workflow_name = "Pull Request CI" 
        
               test_yaml = yaml.dump( 
        
                   { 
        
                       "name": test_workflow_name, 
        
                       "concurrency": { 
        
                           "group": "${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}", 
        
                           "cancel-in-progress": True, 
        
                       }, 
        
                       "on": {"pull_request": {}, "push": {"branches-ignore": ["dependabot/**"]}}, 
        
                       "jobs": pr_jobs, 
        
                       "env": global_env(), 
        
                   }, 
        
                   width=120, 
        
                   Dumper=NoAliasDumper, 
        
               ) 
        
               test_cron_yaml = yaml.dump( 
        
                   { 
        
                       "name": "Daily Extended Python Testing", 
        
                       # 08:45 UTC / 12:45AM PST, 1:45AM PDT: arbitrary time after hours. 
        
                       "on": {"schedule": [{"cron": "45 8 * * *"}]}, 
        
                       "jobs": test_workflow_jobs([PYTHON38_VERSION, PYTHON39_VERSION], cron=True), 
        
                       "env": global_env(), 
        
                   }, 
        
                   Dumper=NoAliasDumper, 
        
               ) 
        
               ignore_advisories = " ".join( 
        
                   f"--ignore {adv_id}" for adv_id in CARGO_AUDIT_IGNORED_ADVISORY_IDS 
        
               ) 
        
               audit_yaml = yaml.dump( 
        
                   { 
        
                       "name": "Cargo Audit", 
        
                       "on": { 
        
                           # 08:11 UTC / 12:11AM PST, 1:11AM PDT: arbitrary time after hours. 
        
                           "schedule": [{"cron": "11 8 * * *"}], 
        
                           # Allow manually triggering this workflow 
        
                           "workflow_dispatch": None, 
        
                       }, 
        
                       "jobs": { 
        
                           "audit": { 
        
                               "runs-on": "ubuntu-latest", 
        
                               "if": IS_PANTS_OWNER, 
        
                               "steps": [ 
        
                                   *checkout(), 
        
                                   { 
        
                                       "name": "Cargo audit (for security vulnerabilities)", 
        
                                       "run": f"./cargo install --version 0.17.5 cargo-audit\n./cargo audit {ignore_advisories}\n", 
        
                                   }, 
        
                               ], 
        
                           } 
        
                       }, 
        
                   } 
        
               ) 
        
               cc_jobs, cc_inputs = cache_comparison_jobs_and_inputs() 
        
               cache_comparison_yaml = yaml.dump( 
        
                   { 
        
                       "name": "Cache Comparison", 
        
                       # Kicked off manually. 
        
                       "on": {"workflow_dispatch": {"inputs": cc_inputs}}, 
        
                       "jobs": cc_jobs, 
        
                   }, 
        
                   Dumper=NoAliasDumper, 
        
               ) 
        
               release_jobs, release_inputs = release_jobs_and_inputs() 
        
               release_yaml = yaml.dump( 
        
                   { 
        
                       "name": "Record Release Commit", 
        
                       "on": { 
        
                           "push": {"tags": ["release_*"]}, 
        
                           "workflow_dispatch": {"inputs": release_inputs}, 
        
                       }, 
        
                       "jobs": release_jobs, 
        
                   }, 
        
                   Dumper=NoAliasDumper, 
        
               ) 
        
               return { 
        
                   Path(".github/workflows/audit.yaml"): f"{HEADER}\n\n{audit_yaml}", 
        
                   Path(".github/workflows/cache_comparison.yaml"): f"{HEADER}\n\n{cache_comparison_yaml}", 
        
                   Path(".github/workflows/test.yaml"): f"{HEADER}\n\n{test_yaml}", 
        
                   Path(".github/workflows/test-cron.yaml"): f"{HEADER}\n\n{test_cron_yaml}", 
        
                   Path(".github/workflows/release.yaml"): f"{HEADER}\n\n{release_yaml}", 
        
               }

rather than hand written, but fine as a followup.

.github/workflows/auto-cherry-picker.yaml

thejcannon · 2023-05-17T16:45:18Z

What do we gain from generating? Seems like extra steps for no/little benefit?

stuhood · 2023-05-17T16:48:01Z

What do we gain from generating? Seems like extra steps for no/little benefit?

Type checking, avoiding yaml gotchas, consistency, reuse.

thejcannon · 2023-05-17T19:58:36Z

build-support/bin/cherry_pick.sh

@@ -60,7 +61,8 @@ if [[ -z $CATEGORY_LABEL ]]; then
  echo "Couldn't detect category label on PR. What's the label? (E.g., category:bugfix)"
  read -r CATEGORY_LABEL
 fi
-REVIEWERS=$(gh pr view "$PR_NUM" --json reviews --jq '.reviews.[].author.login' | sort | uniq)
+REVIEWERS=$(gh pr view "$PR_NUM" --json author --jq '.author.login')


I just tried, adding a PR author as a reviewer of their own PR is a no-op, so this is safe to do unconditionally.

huonw

Nice

thejcannon added 3 commits May 16, 2023 15:42

first try

0e4da29

update

5630c06

okie dokie

09cf85b

thejcannon added the category:internal CI, fixes for not-yet-released features, etc. label May 17, 2023

thejcannon requested review from stuhood, benjyw and huonw May 17, 2023 02:02

huonw reviewed May 17, 2023

View reviewed changes

thejcannon added 2 commits May 17, 2023 11:10

thereeee we go

43f0800

dont need this

d933b37

thejcannon commented May 17, 2023

View reviewed changes

stuhood approved these changes May 17, 2023

View reviewed changes

.github/workflows/auto-cherry-picker.yaml Show resolved Hide resolved

Add author as reviewer

6795598

thejcannon commented May 17, 2023

View reviewed changes

huonw approved these changes May 17, 2023

View reviewed changes

stuhood approved these changes May 17, 2023

View reviewed changes

thejcannon merged commit 2a976d5 into pantsbuild:main May 17, 2023

thejcannon deleted the tryaction branch May 17, 2023 20:47

stuhood mentioned this pull request May 18, 2023

Prepare 2.17.0a0. #19038

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an auto-cherry-pick GitHub Action #19021

Add an auto-cherry-pick GitHub Action #19021

thejcannon commented May 17, 2023 •

edited

Loading

huonw left a comment •

edited

Loading

huonw May 17, 2023

thejcannon May 17, 2023

thejcannon commented May 17, 2023

thejcannon May 17, 2023

thejcannon May 17, 2023

thejcannon May 17, 2023

huonw May 17, 2023

thejcannon May 17, 2023

stuhood left a comment

thejcannon commented May 17, 2023

stuhood commented May 17, 2023

thejcannon May 17, 2023

huonw left a comment

		git config --local user.email "pantsbuild+github-automation@gmail.com"
		git config --local user.name "Worker Pants (Pantsbuild GitHub Automation Bot) "

	def generate() -> dict[Path, str]:
	"""Generate all YAML configs with repo-relative paths."""

	pr_jobs = test_workflow_jobs([PYTHON37_VERSION], cron=False)
	pr_jobs.update(**classify_changes())
	for key, val in pr_jobs.items():
	if key in {"check_labels", "classify_changes"}:
	continue
	needs = val.get("needs", [])
	if isinstance(needs, str):
	needs = [needs]
	needs.extend(["classify_changes"])
	val["needs"] = needs
	if_cond = val.get("if")
	not_docs_only = "needs.classify_changes.outputs.docs_only != 'true'"
	val["if"] = not_docs_only if if_cond is None else f"({if_cond}) && ({not_docs_only})"
	pr_jobs.update(merge_ok(sorted(pr_jobs.keys())))

	test_workflow_name = "Pull Request CI"
	test_yaml = yaml.dump(
	{
	"name": test_workflow_name,
	"concurrency": {
	"group": "${{ github.workflow }}-${{ github.event.pull_request.number \|\| github.sha }}",
	"cancel-in-progress": True,
	},
	"on": {"pull_request": {}, "push": {"branches-ignore": ["dependabot/**"]}},
	"jobs": pr_jobs,
	"env": global_env(),
	},
	width=120,
	Dumper=NoAliasDumper,
	)

	test_cron_yaml = yaml.dump(
	{
	"name": "Daily Extended Python Testing",
	# 08:45 UTC / 12:45AM PST, 1:45AM PDT: arbitrary time after hours.
	"on": {"schedule": [{"cron": "45 8 * * *"}]},
	"jobs": test_workflow_jobs([PYTHON38_VERSION, PYTHON39_VERSION], cron=True),
	"env": global_env(),
	},
	Dumper=NoAliasDumper,
	)
	ignore_advisories = " ".join(
	f"--ignore {adv_id}" for adv_id in CARGO_AUDIT_IGNORED_ADVISORY_IDS
	)
	audit_yaml = yaml.dump(
	{
	"name": "Cargo Audit",
	"on": {
	# 08:11 UTC / 12:11AM PST, 1:11AM PDT: arbitrary time after hours.
	"schedule": [{"cron": "11 8 * * *"}],
	# Allow manually triggering this workflow
	"workflow_dispatch": None,
	},
	"jobs": {
	"audit": {
	"runs-on": "ubuntu-latest",
	"if": IS_PANTS_OWNER,
	"steps": [
	*checkout(),
	{
	"name": "Cargo audit (for security vulnerabilities)",
	"run": f"./cargo install --version 0.17.5 cargo-audit\n./cargo audit {ignore_advisories}\n",
	},
	],
	}
	},
	}
	)

	cc_jobs, cc_inputs = cache_comparison_jobs_and_inputs()
	cache_comparison_yaml = yaml.dump(
	{
	"name": "Cache Comparison",
	# Kicked off manually.
	"on": {"workflow_dispatch": {"inputs": cc_inputs}},
	"jobs": cc_jobs,
	},
	Dumper=NoAliasDumper,
	)

	release_jobs, release_inputs = release_jobs_and_inputs()
	release_yaml = yaml.dump(
	{
	"name": "Record Release Commit",
	"on": {
	"push": {"tags": ["release_*"]},
	"workflow_dispatch": {"inputs": release_inputs},
	},
	"jobs": release_jobs,
	},
	Dumper=NoAliasDumper,
	)

	return {
	Path(".github/workflows/audit.yaml"): f"{HEADER}\n\n{audit_yaml}",
	Path(".github/workflows/cache_comparison.yaml"): f"{HEADER}\n\n{cache_comparison_yaml}",
	Path(".github/workflows/test.yaml"): f"{HEADER}\n\n{test_yaml}",
	Path(".github/workflows/test-cron.yaml"): f"{HEADER}\n\n{test_cron_yaml}",
	Path(".github/workflows/release.yaml"): f"{HEADER}\n\n{release_yaml}",
	}

Add an auto-cherry-pick GitHub Action #19021

Add an auto-cherry-pick GitHub Action #19021

Conversation

thejcannon commented May 17, 2023 • edited Loading

huonw left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thejcannon commented May 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood left a comment

Choose a reason for hiding this comment

thejcannon commented May 17, 2023

stuhood commented May 17, 2023

Choose a reason for hiding this comment

huonw left a comment

Choose a reason for hiding this comment

thejcannon commented May 17, 2023 •

edited

Loading

huonw left a comment •

edited

Loading