Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry requests to rekor, fulcio ? #1148

Open
jku opened this issue Sep 30, 2024 · 1 comment
Open

Retry requests to rekor, fulcio ? #1148

jku opened this issue Sep 30, 2024 · 1 comment
Labels
enhancement New feature or request

Comments

@jku
Copy link
Member

jku commented Sep 30, 2024

I've been keeping an eye on intermittent failures in various CI workflows that run sigstore tools (root-signing, root-signing-staging, sigstore-probers)... and my gut feeling is that sigstore-python fails a little more often than the other clients.

Looking at the client implementations, at least cosign, sigstore-java and sigstore-js seem to have some built-in retries for the requests they make to rekor and fulcio. I wonder if we should have that too?

It's not an obvious decision:

  • Interactive use and CI use have different expectations WRT responsiveness -- maybe we should only retry on CI?
  • It's not entirely trivial to recognize which responses should lead to retries -- potential ones could be 5xx, 429
  • by far most failures seem to be on rekor
  • Some error responses have Retry-After header but most do not seem to (I can't see the actual responses in the load balancer logs so this is partly guesswork but I believe 503 and 429 include the header). 429 specifically only makes sense to retry with the Retry-After value
@jku jku added the enhancement New feature or request label Sep 30, 2024
@woodruffw
Copy link
Member

Adding retries makes sense to me! I think sigstore-go (or maybe gh attestation as its wrapper?) also has some retry logic to make requests more reliable.

If we complete #1040, we could use urllib3's retry APIs: https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html#urllib3.util.Retry

@woodruffw woodruffw changed the title retry requests to rekor, fulcio ? Retry requests to rekor, fulcio ? Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants