Skip to content

Conversation

@amoghrajesh
Copy link
Contributor

@amoghrajesh amoghrajesh commented Nov 26, 2025

The automatic backport workflow was intermittently skipping PRs that should be backported, even when they had backport-to-* labels. After some investigation, I could see a race condition with GitHub's API.

Examples:

For successful runs:

Run actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea
  with:
    script: const { data: pullRequest } = await github.rest.repos.listPullRequestsAssociatedWithCommit({
      owner: context.repo.owner,
      repo: context.repo.repo,
      commit_sha: process.env.GITHUB_SHA
  });
  if (pullRequest.length > 0) {
      const pr = pullRequest[0];
      const backportBranches = pr.labels
            .filter(label => label.name.startsWith('backport-to-'))
            .map(label => label.name.replace('backport-to-', ''));
  
      console.log(`Commit ${process.env.GITHUB_SHA} is associated with PR ${pr.number}`);
      console.log(`Backport branches: ${backportBranches}`);
      core.setOutput('branches', JSON.stringify(backportBranches));
  } else {
      console.log('No pull request found for this commit.');
      core.setOutput('branches', '[]');
  }
  
    github-token: ***
    debug: false
    user-agent: actions/github-script
    result-encoding: json
    retries: 0
    retry-exempt-status-codes: 400,401,403,404,422
  env:
    COMMIT_SHA: c03fc7943009b02072ad275dd31e54f927bff889
    GITHUB_TOKEN: ***
Commit c03fc7943009b02072ad275dd31e54f927bff889 is associated with PR 57853
Backport branches: v3-1-test

For failed runs:

1s
Run actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea
  with:
    script: const { data: pullRequest } = await github.rest.repos.listPullRequestsAssociatedWithCommit({
      owner: context.repo.owner,
      repo: context.repo.repo,
      commit_sha: process.env.GITHUB_SHA
  });
  if (pullRequest.length > 0) {
      const pr = pullRequest[0];
      const backportBranches = pr.labels
            .filter(label => label.name.startsWith('backport-to-'))
            .map(label => label.name.replace('backport-to-', ''));
  
      console.log(`Commit ${process.env.GITHUB_SHA} is associated with PR ${pr.number}`);
      console.log(`Backport branches: ${backportBranches}`);
      core.setOutput('branches', JSON.stringify(backportBranches));
  } else {
      console.log('No pull request found for this commit.');
      core.setOutput('branches', '[]');
  }
  
    github-token: ***
    debug: false
    user-agent: actions/github-script
    result-encoding: json
    retries: 0
    retry-exempt-status-codes: 400,401,403,404,422
  env:
    COMMIT_SHA: caafc704462cc15bedde67ef416d870ceabd13e8
    GITHUB_TOKEN: ***
No pull request found for this commit.

The root cause seems to be that when a PR is merged to main, the workflow immediately queries GH listPullRequestsAssociatedWithCommit API to find the PR and check for backport labels. However, this API call seems to need time to process the merge and associate the commit with the PR.

Performed analysis on couple of PRs and this is my finding:

PR # Commit (short) Merge Time Workflow Start API Call Time Time to API Call Result Current Status
57853 c03fc79 2025-11-17 13:42:26 13:42:29 (+3s) 13:42:33 7 seconds Found PR associated
58575 caafc70 2025-11-22 10:23:24 10:23:26 (+2s) 10:23:30 6 seconds Not found Now associated
58587 19e0a9a 2025-11-26 00:35:10 00:35:13 (+3s) 00:35:18 8 seconds Not found Now associated
58696 6d36d7b 2025-11-26 00:47:54 00:47:57 (+3s) 00:48:01 7 seconds Not found Now associated

To solve this, I have two options:

  1. Add some sort of retry mechanism while calling the API. Complex implementation but could work.

  2. [Chosen Now] Added a 15 second delay before querying the API. This gives the backend sufficient time to process the merge commit and establish the PR association, ensuring backport labels are consistently detected.

Why 15 seconds?

  • Observed failures occurred at 6-8 seconds for my dataset
  • 15 seconds provides ~2 times safety margin
  • Simple solution that handles worst case timing

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg bot added area:dev-tools backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch labels Nov 26, 2025
@amoghrajesh amoghrajesh self-assigned this Nov 26, 2025
Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sleep is not cool but also no better idea. Thanks for the investigation! Wow this is complex :-D

@amoghrajesh
Copy link
Contributor Author

@jscheffl I think I can also get it done by using retries in the API calls. But that will probably be more complex and time consuming to do

@amoghrajesh amoghrajesh merged commit 1aa9b7a into apache:main Nov 26, 2025
118 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 26, 2025
(cherry picked from commit 1aa9b7a)

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
@github-actions
Copy link

Backport successfully created: v3-1-test

Status Branch Result
v3-1-test PR Link

github-actions bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Nov 26, 2025
)

(cherry picked from commit 1aa9b7a)

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
potiuk pushed a commit that referenced this pull request Nov 27, 2025
…58706)

(cherry picked from commit 1aa9b7a)

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
ephraimbuddy pushed a commit that referenced this pull request Dec 3, 2025
…58706)

(cherry picked from commit 1aa9b7a)

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
Copilot AI pushed a commit to jason810496/airflow that referenced this pull request Dec 5, 2025
itayweb pushed a commit to itayweb/airflow that referenced this pull request Dec 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants