Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build(publish-release): race between release view and release create #551

Closed
ee7 opened this issue Apr 12, 2022 · 2 comments · Fixed by #792
Closed

build(publish-release): race between release view and release create #551

ee7 opened this issue Apr 12, 2022 · 2 comments · Fixed by #792
Labels
kind: ci

Comments

@ee7
Copy link
Member

ee7 commented Apr 12, 2022 •
edited
Loading

Follow-up from something I noticed in #550. There's a race condition here:

configlet/.github/bin/publish-release

Lines 5 to 9 in 4ddb23d

if ! gh release view "${build_tag}"; then
gh release create --draft "${build_tag}" "${ARTIFACT_FILE}"
else
gh release upload "${build_tag}" "${ARTIFACT_FILE}"
fi

@ee7 wrote:

Note that we still have another race condition: a separate job can
create a release in the time between gh release view indicating
"there is no such release" and gh release create running. We should
address that later, but it hasn't caused a problem yet - the Linux job
consistently finishes significantly faster than the others. And if we
encounter such a failure in the meantime, the workaround is simple: just
restart the failing job.

One possible way to resolve this is to ensure that gh release create can only run once:

  1. Make each job use actions/upload-artifact only.
  2. Add a dependent job that uses actions/download-artifact, and then runs gh release create.
@ee7 ee7 added the kind: ci label May 15, 2022
@ee7
Copy link
Member Author

ee7 commented Jul 1, 2022 •
edited
Loading

Hit this in 4.0.0-beta.5.

The publish-release script for the macOS job ran within 2 seconds of the Linux one, and caused 2 draft releases to be created (one with Linux archive + Windows archive + checksum file, the other with just the macOS archive).

https://github.com/exercism/configlet/runs/7157826327
https://github.com/exercism/configlet/runs/7157826359

2022-07-01T22:57:51.5826678Z Requested labels: ubuntu-20.04
2022-07-01T22:57:51.5826725Z Job defined at: exercism/configlet/.github/workflows/build.yml@refs/tags/4.0.0-beta.5
2022-07-01T22:57:51.5826752Z Waiting for a runner to pick up this job...
[...]
2022-07-01T22:58:53.7766389Z ##[group]Run ./.github/bin/publish-release
2022-07-01T22:58:53.7766751Z ./.github/bin/publish-release
2022-07-01T22:58:53.7826253Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-07-01T22:58:53.7826541Z env:
2022-07-01T22:58:53.7826830Z   ARTIFACT_FILE: artifacts/configlet-linux-64bit.tgz
2022-07-01T22:58:53.7827744Z   GITHUB_TOKEN: ***
2022-07-01T22:58:53.7828025Z   REF: refs/tags/4.0.0-beta.5
2022-07-01T22:58:53.7828274Z ##[endgroup]
2022-07-01T22:58:55.0117111Z release not found
2022-07-01T22:58:56.7510141Z https://github.com/exercism/configlet/releases/tag/untagged-2e82d65ff20df39ab2ff
2022-07-01T22:58:56.7588138Z Post job cleanup.

2022-07-01T22:57:51.6320017Z Requested labels: macos-11
2022-07-01T22:57:51.6320068Z Job defined at: exercism/configlet/.github/workflows/build.yml@refs/tags/4.0.0-beta.5
2022-07-01T22:57:51.6320118Z Waiting for a runner to pick up this job...
[...]
2022-07-01T22:58:55.4451400Z ##[group]Run ./.github/bin/publish-release
2022-07-01T22:58:55.4451940Z ./.github/bin/publish-release
2022-07-01T22:58:55.4774620Z shell: /bin/bash --noprofile --norc -e -o pipefail {0}
2022-07-01T22:58:55.4774930Z env:
2022-07-01T22:58:55.4775210Z   ARTIFACT_FILE: artifacts/configlet-mac-64bit.tgz
2022-07-01T22:58:55.4775730Z   GITHUB_TOKEN: ***
2022-07-01T22:58:55.4775960Z   REF: refs/tags/4.0.0-beta.5
2022-07-01T22:58:55.4776340Z ##[endgroup]
2022-07-01T22:58:56.8352040Z release not found
2022-07-01T22:58:58.7860570Z https://github.com/exercism/configlet/releases/tag/untagged-2123a3615eee3f93cec1
2022-07-01T22:58:58.7956820Z Post job cleanup.

@ErikSchierboom
Copy link
Member

Ah, race conditions. Fun.

This was referenced Aug 14, 2023
Merged
Merged
@ee7 ee7 closed this as completed in #792 Aug 16, 2023
ee7 added a commit that referenced this issue Aug 16, 2023
@ee7
.github: separate release creation from uploading assets (#792)
88f367f 
Before this commit, the publish-release script handled both:

- creating the draft release

- uploading assets to the release

with the intention that the first build job to finish would create the
release with one asset, and each later job would upload an asset.

The logic was like:

    if ! gh release view "${build_tag}"; then
      gh release create [...]
    else
      gh release upload [...]
    fi

But this had a race. For example, if two build jobs finished at nearly
the same moment, they could each create a draft release. This race was
almost never encountered in practice, and was easily resolved by
deleting a draft release and restarting a build job, but it's worth
resolving.

Add an initial build job to create a draft release that has no assets,
and make the later build jobs only upload to that release.

This fixes the race, and makes it easier to share logic between the
native build jobs and the cross-compiling build jobs.

Fixes: #551
Refs: #789
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind: ci
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

.github: separate release creation from uploading assets ee7/exercism-configlet
2 participants
@ErikSchierboom @ee7