Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exit 94 if a mirror lock times out #3023

Merged
merged 1 commit into from
Oct 8, 2024
Merged

Conversation

DrJosh9000
Copy link
Contributor

@DrJosh9000 DrJosh9000 commented Oct 3, 2024

Description

Lockfiles (used for serialising access to git mirrors) aren't totally reliable in the face of agents dying without releasing them. As a stopgap, if a lockfile can't be acquired before the timeout, exit with a distinctive code (94, chosen randomly).

Context

https://coda.io/d/_dHnUHNps1YO#Escalations-Pipelines_tuQbCBf6/r711

Changes

  • Create a new error type, ErrTimedOutAcquiringLock.
  • Return it if the git mirror lockfiles fail for context.DeadlineExceeded
  • Check for it in the checkout phase, and wrap it in a shell.ExitError
  • Update the one other use of shell.ExitError

Testing

  • Tests have run locally (with go test ./...). Buildkite employees may check this if the pipeline has run automatically.
  • Code is formatted (with go fmt ./...)

@DrJosh9000 DrJosh9000 force-pushed the exit-94-for-lock-timeout branch from 8c02be0 to 88433a9 Compare October 3, 2024 08:18
@DrJosh9000 DrJosh9000 force-pushed the exit-94-for-lock-timeout branch from 88433a9 to ef8d246 Compare October 3, 2024 08:26
Copy link
Contributor

@wolfeidau wolfeidau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, should we add something to the docs for agent error codes?

@DrJosh9000
Copy link
Contributor Author

Great work, should we add something to the docs for agent error codes?

Yes we should!

@DrJosh9000 DrJosh9000 merged commit b93fff7 into main Oct 8, 2024
1 check passed
@DrJosh9000 DrJosh9000 deleted the exit-94-for-lock-timeout branch October 8, 2024 01:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants