Release not running for tagged commit.

[jaraco]

I recently merged the skeleton into pytest-enabler, but after doing so, the CI is no longer running for tagged commits:

 pytest-enabler main @ git push
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 8 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 1.81 KiB | 1.81 MiB/s, done.
Total 4 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com/jaraco/pytest-enabler
   a70b97a..0079f80  main -> main
 * [new tag]         v3.0.0 -> v3.0.0

[jaraco]

Here are the changes applied to that project.

[jaraco]

The change from #93 seem to me to be the most likely cause.

[jaraco]

Indeed, after reverting 8bff8b0, the tagged commits once again run:

@webknjaz Do you know why the merge queue support broke CI runs of tagged commits?

[webknjaz]

Pretty sure this change doesn't cause your problem. The push trigger is still there. When your push tags, the trigger is always push. Let me follow your links and see what's happening there.

[webknjaz]

@jaraco when was the first tag push attempt? Do you have a record that push being successful somewhere? Scratch that — I overlooked your initial log. Looks like GitHub had a few incidents over the past week: https://www.githubstatus.com/history.

In general, though, due to how the GitHub platform (including GitHub Apps / Actions) works, sometimes the internal events don't get delivered to the listening apps. It's impossible to debug and they also don't re-deliver those events.
At times, they also deliberately suspend sending out any event webhooks, when they have platform issues and don't attempt delivering them either.

I'm assuming that your git push succeeded, you didn't delete the workflow run from https://github.com/jaraco/pytest-enabler/actions?query=branch%3Av3.0.0, and it just didn't show up there, right?
This indicates that GHA didn't get a push event in the first place.

So my verdict is that you've been unlucky. Feel free to re-revert and I'm sure it'll work.

Also, watch out for how GHA behaves today, across all the repos in general. When they have issues, this tends to get weird and unpredictable at scale.

[webknjaz]

This incident https://www.githubstatus.com/incidents/476y2pqmb3hc, although marked as resolved, gives me suspicion — maybe they didn't fully fix their platform problem and it manifested itself in no triggers for you.

[jaraco]

I thought it might have been a transient issue. I'll cut a release of something else where I haven't yet performed the revert (I did it only in pytest-enabler) and see what happens.

[jaraco]

I tried again in jaraco.logging:

 jaraco.logging main @ git push
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 8 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 1.71 KiB | 1.71 MiB/s, done.
Total 4 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com/jaraco/jaraco.logging
   ad24708..67a144f  main -> main
 * [new tag]         v3.2.1 -> v3.2.1

And again, no CI was generated for that tag:

assuming... you didn't delete the workflow run

That's correct. I haven't deleted any workflow runs. They just don't appear.

This indicates that GHA didn't get a push event in the first place.

I'm not so sure. GHA is running the actions for the pushed commit. It's just not additionally running GHA for the tag on that commit.

Do you use this config on other projects that are getting CI runs for tags?

[webknjaz]

This indicates that GHA didn't get a push event in the first place.

I'm not so sure. GHA is running the actions for the pushed commit. It's just not additionally running GHA for the tag on that commit.

The docs on this are a bit confusing but here's some things to understand:

• the events aren't directly tied to commits, but instead are connected to Git refs
• when you push new commits to a pre-existing branch a push event with a branch type happens
• when you push new commits to a non-existing branch, two events happen — a push event with a branch type and a create event
• when you push a new tag, two events happen — a push event with a tag type and a create event
• when you force-push a tag (without previously deleting it), one event happens — push event with a tag type
• some event trigger declarations in GHA workflow definitions allow for filtering out some of the events, so they don't trigger the runs
• when such filtering isn't supported, you end up having to stick if: instructions, evaluated during the runs

I used to prefer the create trigger with runtime ref filtering.

At this point, I'd complain to GitHub Support, because what you describe shouldn't be happening according to my previous experience..

Do you use this config on other projects that are getting CI runs for tags?

I switched to using workflow_dispatch in a lot of places, but I think @aio-libs projects still use tag pushes.

Here's a run from a few weeks ago: https://github.com/aio-libs/yarl/actions/runs/7113771461/workflow#L11.

Now I noticed that it also has the tags push type set. Perhaps, using branches-ignore somehow changes the defaults regarding matching tags... Though, I don't recall seeing anything related in the docs.

FWIW, I still maintain that adding an extra event trigger doesn't influence others.

[jaraco]

Thanks for the context. I'm going to try dropping the changes from #93 until the pushed tag runs CI: first, empty branches-ignore, then remove branches-ignore, then remove merge_group, then reformat as a list. When the first step succeeds, that'll give me a more precise indication of the cause after which I'll (a) apply that change to skeleton and (b) contact GitHub support for guidance.

[jaraco]

Step 1

 jaraco.logging skeleton-103-troubleshoot @ git diff
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index a079bbf..f8ea239 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -4,7 +4,6 @@ on:
   merge_group:
   push:
     branches-ignore:
-    - gh-readonly-queue/**  # Temporary merge queue-related GH-made branches
   pull_request:
 
 permissions:
 jaraco.logging skeleton-103-troubleshoot @ git commit -a -m "Remove sole entry for branches-ignore. Ref jaraco/skeleton#103."
[skeleton-103-troubleshoot d91f503] Remove sole entry for branches-ignore. Ref jaraco/skeleton#103.
 1 file changed, 1 deletion(-)
 jaraco.logging skeleton-103-troubleshoot @ git atag -f v3.2.1
Updated tag 'v3.2.1' (was c3fa95f)
 jaraco.logging skeleton-103-troubleshoot @ git push origin -f v3.2.1
Enumerating objects: 10, done.
Counting objects: 100% (10/10), done.
Delta compression using up to 8 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 1.84 KiB | 1.84 MiB/s, done.
Total 6 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com/jaraco/jaraco.logging
 + c3fa95f...f795a90 v3.2.1 -> v3.2.1 (forced update)

That did the trick. Somehow ignoring gh-readonly-queue/** is causing tags to be ignored for GHA.

[jaraco]

I've created a support ticket (private) to track the possibility of an upstream issue with GitHub support.

[webknjaz]

I wonder if you were to add

tags:
- >-
  **

would it be a reasonable workaround?

[jaraco]

Yes, maybe, but only if that's what's recommended. I'd not want to incorporate a workaround without an upstream bug to track the deficiency it's working around.

[jaraco]

Github responded:

The following documentation covers this behavior - mentioning it as:

If you define only tags/tags-ignore or only branches/branches-ignore, the workflow won't run for events affecting the undefined Git ref. If you define neither tags/tags-ignore or branches/branches-ignore, the workflow will run for events affecting either branches or tags.

In this case, it looks like your workflow configuration only included a branches-ignore key - so while defined, push events to tags are expected to not trigger a run. A tags / tags-ignore key will need to be added for push events to tags to trigger runs as well. If there is no specific filter you wish to apply to tag names, you can pass a wildcard value so that all tag names match. For example:

push:
 branches-ignore:    
   - 'branchname'
 tags:
   - '**'

So it does seem that the behavior is documented, as counterintuitive as it seems. Since this entangled behavior is documented, I'm satisfied that the proposed workaround is appropriate.