Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

resource temporarily unavailable error occurs frequently #2576

Closed
suzuki-shunsuke opened this issue Dec 27, 2022 · 9 comments
Closed

resource temporarily unavailable error occurs frequently #2576

suzuki-shunsuke opened this issue Dec 27, 2022 · 9 comments
Labels
bug Something isn't working

Comments

@suzuki-shunsuke
Copy link

suzuki-shunsuke commented Dec 27, 2022 •
edited
Loading

Description

When I run cosign verify-blob, resource temporarily unavailable error occurs frequently.

https://github.com/aquaproj/aqua/actions/runs/3784672207/jobs/6434146871#step:39:26

Error: verifying blob [/tmp/257978516]: getting Fulcio roots: initializing tuf: updating local metadata and targets: creating cached local store: resource temporarily unavailable
main.go:62: error during command execution: verifying blob [/tmp/257978516]: getting Fulcio roots: initializing tuf: updating local metadata and targets: creating cached local store: resource temporarily unavailable

The command passed by retrying.

Version

https://github.com/aquaproj/aqua/blob/d37dec79a9b96c85592eb24d69f9972cbd176f9a/pkg/cosign/version.go#L3

v1.13.1

environment

At the moment, this error occurs only in GitHub Actions ubuntu-latest.
This error haven't occurred in my laptop.

How to reproduce

Run the following command, then it failed temporarily.

env COSIGN_EXPERIMENTAL=1 cosign verify-blob \
  --signature https://github.com/terraform-linters/tflint/releases/download/v0.43.0/checksums.txt.keyless.sig \
  --certificate https://github.com/terraform-linters/tflint/releases/download/v0.43.0/checksums.txt.pem \
  checksums.txt

assets: https://github.com/terraform-linters/tflint/releases/tag/v0.43.0
@suzuki-shunsuke suzuki-shunsuke added the bug Something isn't working label Dec 27, 2022
@suzuki-shunsuke
Copy link
Author

Hmm. I tried to create a reproduction code, but the command succeeded...
I retried workflows several times, but all passed. 🤔
suzuki-shunsuke/cosign-issue-2576#1

@haydentherapper
Copy link
Contributor

Are you still seeing this issue? This could occur if running cosign processes in parallel.

@suzuki-shunsuke
Copy link
Author

suzuki-shunsuke commented Jan 10, 2023 •
edited
Loading

Thank you for your answer!

Are you still seeing this issue?

I removed some test code in CI to prevent the error, so currently I'm not seeing this issue.

This could occur if running cosign processes in parallel.

Probably this is the cause of the issue.

In my usecase, I use Cosign in my CLI tool aqua, which is a CLI Version Manager, for verifying packages signed by Cosign.
And aqua could execute Cosign in parallel when multiple packages are installed in parallel.
So I'll try the following solutions.

  1. Retry to execute Cosign if this error occurs
  2. Get a lock before executing Cosign to prevent Cosign from being executing in parallel

I'll appreciate if you have any advice.

Anyway, this is an issue of not Cosign but my usage of Cosign.
So I close this issue now.

Thank you for your answer and great project!

@suzuki-shunsuke
Copy link
Author

Sorry, I have a question.
Does this issue occur only in keyless signing?
Or is keyless signing unrelated?

This was referenced Jan 10, 2023
Closed
Closed
@haydentherapper
Copy link
Contributor

If you upload signatures to Rekor after signing with a key, it can still occur. If you only use Cosign with a key to sign an artifact or container and don't upload to Rekor, then it should not, but of course this would mean no artifact transparency.

@suzuki-shunsuke
Copy link
Author

suzuki-shunsuke commented Jan 10, 2023 •
edited
Loading

Thank you for your explanation. It means this issue could occur without keyless signing.

cosign sign has an option --upload to skip uploading signature to Rekor, but cosign sign-blob seems not to have the same option. So I guess cosign sign-blob always uploads signature to Rekor.

$ cosign version
  ______   ______        _______. __    _______ .__   __.
 /      | /  __  \      /       ||  |  /  _____||  \ |  |
|  ,----'|  |  |  |    |   (----`|  | |  |  __  |   \|  |
|  |     |  |  |  |     \   \    |  | |  | |_ | |  . `  |
|  `----.|  `--'  | .----)   |   |  | |  |__| | |  |\   |
 \______| \______/  |_______/    |__|  \______| |__| \__|
cosign: A tool for Container Signing, Verification and Storage in an OCI registry.

GitVersion:    v1.13.1
GitCommit:     d1c6336475b4be26bb7fb52d97f56ea0a1767f9f
GitTreeState:  clean
BuildDate:     2022-10-17T18:00:05Z
GoVersion:     go1.19.2
Compiler:      gc
Platform:      darwin/arm64
$ cosign sign-blob --help
Sign the supplied blob, outputting the base64-encoded signature to stdout.

Usage:
  cosign sign-blob [flags]

Examples:
  cosign sign-blob --key <key path>|<kms uri> <blob>

  # sign a blob with Google sign-in (experimental)
  COSIGN_EXPERIMENTAL=1 cosign --timeout 90s sign-blob <FILE>

  # sign a blob with a local key pair file
  cosign sign-blob --key cosign.key <FILE>

  # sign a blob with a key pair stored in Azure Key Vault
  cosign sign-blob --key azurekms://[VAULT_NAME][VAULT_URI]/[KEY] <FILE>

  # sign a blob with a key pair stored in AWS KMS
  cosign sign-blob --key awskms://[ENDPOINT]/[ID/ALIAS/ARN] <FILE>

  # sign a blob with a key pair stored in Google Cloud KMS
  cosign sign-blob --key gcpkms://projects/[PROJECT]/locations/global/keyRings/[KEYRING]/cryptoKeys/[KEY] <FILE>

  # sign a blob with a key pair stored in Hashicorp Vault
  cosign sign-blob --key hashivault://[KEY] <FILE>

Flags:
      --allow-insecure-registry                                                                  whether to allow insecure connections to registries. Don't use this for anything but testing
      --attachment-tag-prefix [AttachmentTagPrefix]sha256-[TargetImageDigest].[AttachmentName]   optional custom prefix to use for attached image tags. Attachment images are tagged as: [AttachmentTagPrefix]sha256-[TargetImageDigest].[AttachmentName]
      --b64                                                                                      whether to base64 encode the output (default true)
      --bundle string                                                                            write everything required to verify the blob to a FILE
      --fulcio-url string                                                                        [EXPERIMENTAL] address of sigstore PKI server (default "https://fulcio.sigstore.dev")
  -h, --help                                                                                     help for sign-blob
      --identity-token string                                                                    [EXPERIMENTAL] identity token to use for certificate from fulcio
      --insecure-skip-verify                                                                     [EXPERIMENTAL] skip verifying fulcio published to the SCT (this should only be used for testing).
      --k8s-keychain                                                                             whether to use the kubernetes keychain instead of the default keychain (supports workload identity).
      --key string                                                                               path to the private key file, KMS URI or Kubernetes Secret
      --oidc-client-id string                                                                    [EXPERIMENTAL] OIDC client ID for application (default "sigstore")
      --oidc-client-secret-file string                                                           [EXPERIMENTAL] Path to file containing OIDC client secret for application
      --oidc-disable-ambient-providers                                                           [EXPERIMENTAL] Disable ambient OIDC providers. When true, ambient credentials will not be read
      --oidc-issuer string                                                                       [EXPERIMENTAL] OIDC provider to be used to issue ID token (default "https://oauth2.sigstore.dev/auth")
      --oidc-provider string                                                                     [EXPERIMENTAL] Specify the provider to get the OIDC token from (Optional). If unset, all options will be tried. Options include: [spiffe, google, github, filesystem]
      --oidc-redirect-url string                                                                 [EXPERIMENTAL] OIDC redirect URL (Optional). The default oidc-redirect-url is 'http://localhost:0/auth/callback'.
      --output string                                                                            write the signature to FILE
      --output-certificate string                                                                write the certificate to FILE
      --output-signature string                                                                  write the signature to FILE
      --rekor-url string                                                                         [EXPERIMENTAL] address of rekor STL server (default "https://rekor.sigstore.dev")
      --sk                                                                                       whether to use a hardware security key
      --slot string                                                                              security key slot to use for generated key (default: signature) (authentication|signature|card-authentication|key-management)
  -y, --yes                                                                                      skip confirmation prompts for non-destructive operations

Global Flags:
      --output-file string   log output to a file
  -t, --timeout duration     timeout for commands (default 3m0s)
  -d, --verbose              log debug output

@haydentherapper
Copy link
Contributor

IIRC you can set rekor-url to "" to disable uploading. We also added a flag in Cosign 2.0 to dictate whether or not you're uploading to Rekor, which will hopefully be out by end of month.

@suzuki-shunsuke
Copy link
Author

Looks good. I'm looking forward to Cosign 2.0.
Thank you!

@suzuki-shunsuke
Copy link
Author

📝 Noted.

@LucaGuerra LucaGuerra mentioned this issue Sep 22, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@haydentherapper @suzuki-shunsuke