Skip to content

Conversation

dscho
Copy link
Member

@dscho dscho commented Sep 2, 2025

In preparation for supporting projects other than Git, let's combine the sync-git-gui and sync-gitster-git workflows into a single, matrix one. The next step after that will be to define the project-specific parts in a project configuration and consume it from there.

While at it, also rename the workflow that synchronizes the mailing list mirror, so that it does not talk about Git specifically.

In the upcoming work to support a project other than Git, we will need
to synchronize a different upstream repository to a different "PR
repository".

To that end, let's consolidate `sync-git-gui` and `sync-gitster-git`
into a single, matrix workflow that will continue to synchronize the Git
GUI and the fine-grained branches from `gitster/git` to
`gitgitgadget/git`, but can be overridden in another fork of
`gitgitgadget-workflows` via a repository variable that contains the
project config to perform the equivalent job of synchronizing upstream
branches into the repository where GitGitGadget handles the Pull
Requests for said project.

This is step 1 to support that: renaming the `sync-git-gui` workflow to
`sync-upstream-branches`. Subsequent commits will transmogrify that
workflow into the desired shape.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The workflow still only handles the `git-gui` branches, but is now
almost generic enough to become a matrix workflow that also handles the
`gitster/git` synchronization.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This still does the exact same thing, but avoids storing intermediate
states of the `data` array's transformation.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
…orkflow

With a few touch-ups, the workflow can become generic enough to handle
also the synchronization of `gitster/git`'s branches to
`gitgitgadget/git`.

The overall goal, of course, is to prepare for supporting projects other
than Git, via configurations stored in repository variables. That's the
task of the next commit(s).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
I want to use the same workflow in a fork to synchronize _Cygwin's_
mailing list mirror; Let's rename the workflow and its job so that it
stops claiming to synchronize _Git's_ mailing list mirror.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Rather than hard-coding the exact minutes at which the workflow should
run, just specify that it should run every five minutes, and leave it to
GitHub to figure out the specifics.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
@dscho dscho force-pushed the consolidate-sync-branches-workflows branch from 36d1292 to 3c327fc Compare September 4, 2025 22:39
@dscho
Copy link
Member Author

dscho commented Sep 4, 2025

I also noticed that it makes no sense to introduce kebap-cased matrix variables only to switch them to CamelCased ones later. Instead, I now introduce CamelCased ones right away.

Range-diff
  • 1: 5eb6540 ! 1: 7a1ca71 sync-upstream-branches: turn the workflow into a matrix

    @@ .github/workflows/sync-upstream-branches.yml: concurrency:
     +    strategy:
     +      matrix:
     +        spec:
    -+          - source-repo: j6t/git-gui
    -+            target-repo: gitgitgadget/git
    -+            target-ref-namespace: git-gui/
    ++          - sourceRepo: j6t/git-gui
    ++            targetRepo: gitgitgadget/git
    ++            targetRefNamespace: git-gui/
     +
          steps:
            - name: check which refs need to be synchronized
    @@ .github/workflows/sync-upstream-branches.yml: concurrency:
              with:
                script: |
     -            const [targetRepoOwner, targetRepoName] = process.env.TARGET_REPOSITORY.split('/')
    -+            const sourceRepo = ${{ toJSON(matrix.spec.source-repo) }}
    -+            const targetRepo = ${{ toJSON(matrix.spec.target-repo) }}
    -+            const targetRefNamespace = ${{ toJSON(matrix.spec.target-ref-namespace) }}
    ++            const sourceRepo = ${{ toJSON(matrix.spec.sourceRepo) }}
    ++            const targetRepo = ${{ toJSON(matrix.spec.targetRepo) }}
    ++            const targetRefNamespace = ${{ toJSON(matrix.spec.targetRefNamespace) }} || ''
     +
     +            const [targetRepoOwner, targetRepoName] = targetRepo.split('/')
                  core.setOutput('target-repo-owner', targetRepoOwner)
    @@ .github/workflows/sync-upstream-branches.yml: jobs:
     +            const targetRefs = await getRefs(targetRepo, targetRefNamespace)
      
     -            const targetPrefix = `refs/heads/${process.env.TARGET_REF_NAMESPACE}`
    -+            const targetPrefix = targetRefNamespace ? `refs/heads/${targetRefNamespace}` : 'refs/heads/'
    ++            const targetPrefix = `refs/heads/${targetRefNamespace}`
      
                  const refspecs = []
                  const toFetch = new Set()
    @@ .github/workflows/sync-upstream-branches.yml: jobs:
                git init --bare
      
     -          git remote add source "${{ github.server_url }}/$SOURCE_REPOSITORY"
    -+          git remote add source '${{ github.server_url }}/${{ matrix.spec.source-repo }}'
    ++          git remote add source '${{ github.server_url }}/${{ matrix.spec.sourceRepo }}'
                # pretend to be a partial clone
                git config remote.source.promisor true
                git config remote.source.partialCloneFilter blob:none
    @@ .github/workflows/sync-upstream-branches.yml: jobs:
                printf '%s' '${{ steps.check.outputs.refspec }}' |
                xargs -d ' ' -r git -c http.extraHeader='${{ steps.auth.outputs.header }}' \
     -            push "${{ github.server_url }}/$TARGET_REPOSITORY"
    -+            push '${{ github.server_url }}/${{ matrix.spec.target-repo }}'
    ++            push '${{ github.server_url }}/${{ matrix.spec.targetRepo }}'
  • 2: 5cbbf3c = 2: 6cc9af2 sync-upstream-branches: adopt a more functional style

  • 3: 792313e ! 3: 265711c sync-upstream-branches: take over the job of the sync-gitster-git workflow

    @@ .github/workflows/sync-gitster-git.yml (deleted)
     
      ## .github/workflows/sync-upstream-branches.yml ##
     @@ .github/workflows/sync-upstream-branches.yml: jobs:
    -           - source-repo: j6t/git-gui
    -             target-repo: gitgitgadget/git
    -             target-ref-namespace: git-gui/
    -+          - source-repo: gitster/git
    -+            target-repo: gitgitgadget/git
    -+            source-ref-regex: "^refs/heads/(maint-\\d|[a-z][a-z]/)"
    +           - sourceRepo: j6t/git-gui
    +             targetRepo: gitgitgadget/git
    +             targetRefNamespace: git-gui/
    ++          - sourceRepo: gitster/git
    ++            targetRepo: gitgitgadget/git
    ++            sourceRefRegex: "^refs/heads/(maint-\\d|[a-z][a-z]/)"
      
          steps:
            - name: check which refs need to be synchronized
     @@ .github/workflows/sync-upstream-branches.yml: jobs:
              with:
                script: |
    -             const sourceRepo = ${{ toJSON(matrix.spec.source-repo) }}
    -+            const sourceRefRegexp = ((p) => p ? new RegExp(p) : null)(${{ toJSON(matrix.spec.source-ref-regex) }})
    -             const targetRepo = ${{ toJSON(matrix.spec.target-repo) }}
    -             const targetRefNamespace = ${{ toJSON(matrix.spec.target-ref-namespace) }}
    +             const sourceRepo = ${{ toJSON(matrix.spec.sourceRepo) }}
    ++            const sourceRefRegexp = ((p) => p ? new RegExp(p) : null)(${{ toJSON(matrix.spec.sourceRefRegex) }})
    +             const targetRepo = ${{ toJSON(matrix.spec.targetRepo) }}
    +             const targetRefNamespace = ${{ toJSON(matrix.spec.targetRefNamespace) }} || ''
      
     @@ .github/workflows/sync-upstream-branches.yml: jobs:
                          await github.rest.git.listMatchingRefs({
  • 4: c682136 = 4: ca0a54d sync-git-mailing-list-mirror: start making the workflow more generic

  • 5: 36d1292 = 5: 3c327fc sync-mailing-list-mirror: use more elegant cron definition

@dscho dscho merged commit 039ebd7 into main Sep 5, 2025
@dscho dscho deleted the consolidate-sync-branches-workflows branch September 5, 2025 05:56
Copy link
Collaborator

@webstech webstech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll let you complete this so you can control the timing.

dscho added a commit to cygwingitgadget/gitgitgadget-workflows that referenced this pull request Sep 28, 2025
Authorization is hard. So hard, in fact, that it seems that (maybe
due to incorrect optimizations?) pushes to GitHub with an installation
access token that has _just_ been obtained sometimes fail with a 403,
even though it seems that a re-run "fixes" the problem.

The first instances of this problem was
https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/actions/runs/17543752351/job/49820347084#step:6:11
on Sep 8, 2025, 9:52 AM GMT+2:

  Run git push ***github.com/$TARGET_GITHUB_REPOSITORY lore-$LORE_EPOCH
  remote: Permission to gitgitgadget/git-mailing-list-mirror.git denied to gitgitgadget[bot].
  fatal: unable to access 'https://github.com/gitgitgadget/git-mailing-list-mirror/': The requested URL returned error: 403
  Error: Process completed with exit code 128.

This happened relatively shortly after I had merged
gitgitgadget#12 on
Sep 5, 2025, 7:55 AM GMT+2, and there were 30 such occurrences of
denied pushes due to apparently not-yet-valid installation tokens in
https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/actions/workflows/sync-mailing-list-mirror.yml
(which isn't bad, given that there have been 1,892 runs of that workflow
since Sep 5, 8:44 AM GMT+2, as of time of writing).

So it _might_ seem as if the flake has something to do with this change,
but then, it occurs too rarely to be _caused_ by that change, _and_ it
seems to happen a tad more frequently than once a day _but_ started 3
days after merging that PR. As I stated above, I have come to the
conclusion that it is something on GitHub's side where they changed
something on their side that _sometimes_ lets installation access tokens
be _not quite ready for pushing_ for a little while right after the
tokens have been issued.

Corroborating evidence for that suspicion are the 8 occurrences of the
same issue in the sister repository I established for Cygwin:
https://github.com/cygwingitgadget/gitgitgadget-workflows/actions/workflows/sync-mailing-list-mirror.yml
And in that repository, essentially the same workflow had been running a
quite a few days longer _without_ any issues, more concretely, it has
done its job without problems since Aug 30, 2025, 6:40 PM GMT+2 in
https://github.com/cygwingitgadget/gitgitgadget-workflows/actions/runs/17346140096
until it failed for the first time on Sep 22, 2025, 7:24 PM GMT+2 in
https://github.com/cygwingitgadget/gitgitgadget-workflows/actions/runs/17923242729/job/50963073473#step:6:11
with the same 403.

Let's just work around that by trying again to push, if necessary a
couple of times (with a back-off strategy).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho added a commit to gitgitgadget-workflows/gitgitgadget-workflows that referenced this pull request Sep 28, 2025
Authorization is hard. So hard, in fact, that it seems that (maybe
due to incorrect optimizations?) pushes to GitHub with an installation
access token that has _just_ been obtained sometimes fail with a 403,
even though it seems that a re-run "fixes" the problem.

The first instances of this problem was
https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/actions/runs/17543752351/job/49820347084#step:6:11
on Sep 8, 2025, 9:52 AM GMT+2:

  Run git push ***github.com/$TARGET_GITHUB_REPOSITORY lore-$LORE_EPOCH
  remote: Permission to gitgitgadget/git-mailing-list-mirror.git denied to gitgitgadget[bot].
  fatal: unable to access 'https://github.com/gitgitgadget/git-mailing-list-mirror/': The requested URL returned error: 403
  Error: Process completed with exit code 128.

This happened relatively shortly after I had merged
gitgitgadget#12 on
Sep 5, 2025, 7:55 AM GMT+2, and there were 30 such occurrences of
denied pushes due to apparently not-yet-valid installation tokens in
https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/actions/workflows/sync-mailing-list-mirror.yml
(which isn't bad, given that there have been 1,892 runs of that workflow
since Sep 5, 8:44 AM GMT+2, as of time of writing).

So it _might_ seem as if the flake has something to do with this change,
but then, it occurs too rarely to be _caused_ by that change, _and_ it
seems to happen a tad more frequently than once a day _but_ started 3
days after merging that PR. As I stated above, I have come to the
conclusion that it is something on GitHub's side where they changed
something on their side that _sometimes_ lets installation access tokens
be _not quite ready for pushing_ for a little while right after the
tokens have been issued.

Corroborating evidence for that suspicion are the 8 occurrences of the
same issue in the sister repository I established for Cygwin:
https://github.com/cygwingitgadget/gitgitgadget-workflows/actions/workflows/sync-mailing-list-mirror.yml
And in that repository, essentially the same workflow had been running a
quite a few days longer _without_ any issues, more concretely, it has
done its job without problems since Aug 30, 2025, 6:40 PM GMT+2 in
https://github.com/cygwingitgadget/gitgitgadget-workflows/actions/runs/17346140096
until it failed for the first time on Sep 22, 2025, 7:24 PM GMT+2 in
https://github.com/cygwingitgadget/gitgitgadget-workflows/actions/runs/17923242729/job/50963073473#step:6:11
with the same 403.

Let's just work around that by trying again to push, if necessary a
couple of times (with a back-off strategy).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho added a commit to cygwingitgadget/gitgitgadget-workflows that referenced this pull request Sep 29, 2025
Authorization is hard. So hard, in fact, that it seems that (maybe
due to incorrect optimizations?) pushes to GitHub with an installation
access token that has _just_ been obtained sometimes fail with a 403,
even though it seems that a re-run "fixes" the problem.

The first instances of this problem was
https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/actions/runs/17543752351/job/49820347084#step:6:11
on Sep 8, 2025, 9:52 AM GMT+2:

  Run git push ***github.com/$TARGET_GITHUB_REPOSITORY lore-$LORE_EPOCH
  remote: Permission to gitgitgadget/git-mailing-list-mirror.git denied to gitgitgadget[bot].
  fatal: unable to access 'https://github.com/gitgitgadget/git-mailing-list-mirror/': The requested URL returned error: 403
  Error: Process completed with exit code 128.

This happened relatively shortly after I had merged
gitgitgadget#12 on
Sep 5, 2025, 7:55 AM GMT+2, and there were 30 such occurrences of
denied pushes due to apparently not-yet-valid installation tokens in
https://github.com/gitgitgadget-workflows/gitgitgadget-workflows/actions/workflows/sync-mailing-list-mirror.yml
(which isn't bad, given that there have been 1,892 runs of that workflow
since Sep 5, 8:44 AM GMT+2, as of time of writing).

So it _might_ seem as if the flake has something to do with this change,
but then, it occurs too rarely to be _caused_ by that change, _and_ it
seems to happen a tad more frequently than once a day _but_ started 3
days after merging that PR. As I stated above, I have come to the
conclusion that it is something on GitHub's side where they changed
something on their side that _sometimes_ lets installation access tokens
be _not quite ready for pushing_ for a little while right after the
tokens have been issued.

Corroborating evidence for that suspicion are the 8 occurrences of the
same issue in the sister repository I established for Cygwin:
https://github.com/cygwingitgadget/gitgitgadget-workflows/actions/workflows/sync-mailing-list-mirror.yml
And in that repository, essentially the same workflow had been running a
quite a few days longer _without_ any issues, more concretely, it has
done its job without problems since Aug 30, 2025, 6:40 PM GMT+2 in
https://github.com/cygwingitgadget/gitgitgadget-workflows/actions/runs/17346140096
until it failed for the first time on Sep 22, 2025, 7:24 PM GMT+2 in
https://github.com/cygwingitgadget/gitgitgadget-workflows/actions/runs/17923242729/job/50963073473#step:6:11
with the same 403.

Let's just work around that by trying again to push, if necessary a
couple of times (with a back-off strategy).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants