verify command: support keyless verification using only a provided certificate chain with non-fulcio roots

[dmitris]

This is a continuation of the work started by @nsmith5 in #2631 and uses the code of that PR rebased on the current trunk.

Fixes #2630

Summary

This change adds support for verifying keyless signatures from the specified --certificate-chain instead of assuming Fulcio and loading its roots and without requiring to pass the certificate in addition to the certificate chain.

As @haydentherapper noted in the comment below, the keyless verification without the Fulcio certificate roots is already possible but requires passing both certificate-chain and certificate.
This would allow folks to more easily verify signatures from a custom CA (see #2630 for additional discussion).

Release Note

Allow verifying signatures from a custom CA without needing leaf certificate
Documentation

Will attach an PR for docs here if folks agree on the parent issue that its worth doing

TODO

The TODO list as this idea is still in discussion etc.

• Add PR for docs with examples of this kind of verification (done: 'cosign verify' for keyless verification with non-Fulio roots - '--cert-chain' without '--cert' (sigstore/cosign pr2845) docs#153)

• Add some end to end tests of this (done: https://github.com/dmitris/cosign-keyless)

[codecov]

Codecov Report

Merging #2845 (ff1b5c1) into main (ef1b2a0) will increase coverage by 0.35%.
The diff coverage is 17.24%.

❗ Current head ff1b5c1 differs from pull request most recent head 6196278. Consider uploading reports for the commit 6196278 to get more accurate results

@@            Coverage Diff             @@
##             main    #2845      +/-   ##
==========================================
+ Coverage   30.32%   30.67%   +0.35%     
==========================================
  Files         151      151              
  Lines        9451     9469      +18     
==========================================
+ Hits         2866     2905      +39     
+ Misses       6140     6111      -29     
- Partials      445      453       +8     

Impacted Files	Coverage Δ
cmd/cosign/cli/verify.go	0.00% <0.00%> (ø)
cmd/cosign/cli/verify/verify.go	32.84% <20.00%> (+10.42%)	⬆️

[nsmith5]

Oh lovely! Thanks for picking this up @dmitris

[dmitris]

(planning to work on this the weak after Easter - after April 11)

[haydentherapper]

cc @znewman01 @woodruffw This relates to the conversation we've been having. If we go through with this, it would assume that the chain in the bundle contains the root. Note that Cosign already does this, with cosign sign --certificate cert.pem --certificate-chain chain.pem <artifact>, so this PR is mostly an optimization that would allow you to provide --bundle instead of --certificate & --certificate-chain. We could either say that this is no different than what we're doing currently, so it's fine, or argue that this behavior actually should be removed and we should be explicit about the root.

nit for the title, can we update to reflect this, something like "Support providing a bundle to verify with non-Fulcio roots", because identity-based verification is already suppported

[dmitris]

so this PR is mostly an optimization that would allow you to provide --bundle instead of --certificate & --certificate-chain.

We are running for an internal PoC a version of cosign with this modification, and I think we do not change the command-line parameters - we call it like this:

./cosign verify --insecure-ignore-tlog --insecure-ignore-sct --check-claims=true --certificate-identity-regexp spiffe://<ci.internal.identity> --certificate-oidc-issuer-regexp '.*' --certificate-chain path/to/internal-cert-bundle.pem 

@haydentherapper - do you mean that we should introduce the --bundle parameter for verify and use it instead of the --certificate-chain one?

nit for the title, can we update to reflect this, something like "Support providing a bundle to verify with non-Fulcio roots", because identity-based verification is already suppported

identity-based verification is, but you cannot verify the image without the Fulcio roots, and I believe this Is the essence of this change (please correct me if I'm wrong!). I think with this we can fix the issue on the "Bad News" slide of @nsmith5's Seattle talk https://youtu.be/d5vROySf1I8?t=1726 28:45.
So I wonder if we could agree on the title like "verify command: support keyless verification using provided certificate chain with non-fulcio roots"?

[haydentherapper]

Sorry @dmitris, I got my wires crossed, I was thinking this was about verify-blob and a bundle instead of verify. We don't need to add support for --bundle, as per your question on Slack.

Your PR title suggestion sounds good. For a small change, can we add "only", "verify command: support keyless verification using only a provided certificate chain with non-fulcio roots", because it is possible to do keyless verification with non-fulcio roots now, you just have to specify the leaf cert with --certificate in addition to the chain

[woodruffw]

Thanks for pinging me in here!

We could either say that this is no different than what we're doing currently, so it's fine, or argue that this behavior actually should be removed and we should be explicit about the root.

Making sure I understand: would the changes in this PR mean that including a root in the bundle's chain would be an intended use case? If that's correct, then I'd argue this should be removed entirely rather than expanded upon: including (and trusting) the bundled root is no different from accepting a self-signed signature.

(It's possible I completely misunderstood, in which case please ignore me!)

[haydentherapper]

The current state is that you can specify the certificate chain to verify the provided certificate. The root is not provided out of band - It's been noted on another issue that we need to correct this, I think we'll probably just tackle that when we implement support for the new bundle format.

This proposed change, for only cosign verify (containers) lets you specify only the certificate chain without specifying the leaf, because the leaf is in the OCI annotations.verify-blob is unaffected because you always have to specify the certificate (or the --bundle flag, which is more of an optimization). This is no worse than what we've got right now, but still has the issue of providing the root in-band.

So the question is, would we rather make the current, suboptimal behavior more uniform, or stop making any changes to the chain until we mandate provided roots out of band?

[woodruffw]

Got it, thanks for explaining!

That sounds like a logistical choice for cosign, but my outsider's preference would be for unifying the different Sigstore clients on enforcing OOB trust materials 🙂

[znewman01]

IMO this all goes back to #2472 — the "chain" concept is really confusing, and the --certificate-chain flag should be called something like --trusted-root-chain.

Given that, I think it's reasonable to make the behavior be (1) configure the trusted certificates, then (2) get the cert from wherever you can find it.

[dmitris]

for the first TODO in the description, "Add PR for docs with examples of this kind of verification", see sigstore/docs#153, preview https://deploy-preview-153--docssigstore.netlify.app/cosign/verify/#keyless-verification-using-only-a-provided-certificate-chain-with-non-fulcio-roots or the attached screenshots:

[dmitris]

@haydentherapper would you (or other reviewers) insist on having an e2e test for this feature in this PR, or could I do it in a follow-up? I did test it manually and it works [for me] as expected. I'm not familiar with the e2e tests, and the related documentation I believe is still TODO (https://github.com/sigstore/cosign/blob/main/CONTRIBUTING.md) 😄 Trying to run test/e2e_test.sh, I'm getting a failure (below), both in this branch and in trunk - asked on Slack [1] but didn't get any replies (yet).

test/e2e_test.sh
[...]
2023/04/25 18:45:18 GET /v2/cosign-e2e/referrers/sha256:a43304747e8d511d77dcbab6ece8dafc57e8dab2eed5cf2afb50ce95a85262c7 404 METHOD_UNKNOWN We don't understand your method + url
2023/04/25 18:45:18 GET /v2/cosign-e2e/manifests/sha256-a43304747e8d511d77dcbab6ece8dafc57e8dab2eed5cf2afb50ce95a85262c7 404 MANIFEST_UNKNOWN Unknown manifest
2023/04/25 18:45:19 GET /v2/
2023/04/25 18:45:19 GET /v2/cosign-e2e/manifests/latest
2023/04/25 18:45:19 GET /v2/
2023/04/25 18:45:19 GET /v2/cosign-e2e/manifests/sha256-a43304747e8d511d77dcbab6ece8dafc57e8dab2eed5cf2afb50ce95a85262c7.sig
2023/04/25 18:45:19 GET /v2/cosign-e2e/blobs/sha256:8a3a2d50844ea1a43c3a308952e8b3b5b4ad08cacc36b10fc7555671c0c57c13
2023/04/25 18:45:20 GET /v2/
2023/04/25 18:45:20 DELETE /v2/cosign-e2e/manifests/latest
2023/04/25 18:45:21 GET /v2/
2023/04/25 18:45:21 DELETE /v2/cosign-e2e/manifests/sha256-a43304747e8d511d77dcbab6ece8dafc57e8dab2eed5cf2afb50ce95a85262c7.sig
FAIL
FAIL	github.com/sigstore/cosign/v2/test	542.888s
FAIL

Full output of bash -x test/e2e_test.sh: https://gist.github.com/dmitris/44b78d3de43c645fa55ca4c5ae324fb3

[1] https://sigstore.slack.com/archives/C01PZKDL4DP/p1681925607818049

[tuminoid]

Full output of bash -x test/e2e_test.sh: https://gist.github.com/dmitris/44b78d3de43c645fa55ca4c5ae324fb3

[1] https://sigstore.slack.com/archives/C01PZKDL4DP/p1681925607818049

Yeah, this e2e is not really documented, but looking at the GH workflows .github/workflows/tests.yaml it seems to require kind cluster and GODEBUG env set up:

      - name: setup kind cluster
        run: |
          # Used to test: cosign generate-key-pair k8s://...
          go install sigs.k8s.io/kind@v0.17.0
          kind create cluster

      - name: Run end-to-end tests
        env:
          # See #2091 for the issue describing this temp workaround.
          GODEBUG: x509sha1=1
        run: ./test/e2e_test.sh

You also need ko installed: https://github.com/ko-build/ko/releases then after kind create cluster and export GODEBUG=x509sha1=1 e2e with ./test/e2e_test.sh works.

[dmitris]

I have put together a "one-shot" / automated script that goes through the keyless verification with non-fulcio root flow: https://github.com/dmitris/cosign-keyless - https://github.com/dmitris/cosign-keyless/blob/main/run.sh is the script,
https://github.com/dmitris/cosign-keyless/blob/main/sample.out contains sample output of a test run.

As expected, it fails with the trunk version of cosign and works OK using the branch of this PR. Maybe we could use it as the test evidence to merge? I would follow up with the unit tests afterwards.
/cc @haydentherapper

[haydentherapper]

Thanks for trying to get tests working, but I think it's clear that there's no great way to test this out besides a manual test. Given you've shown it works, I'm fine with this change. Can you just clean up the duplicated tests?

[dmitris]

@haydentherapper removed the TestVerifyKeylessVerification test and also squashed some commits. Thanks for your help! 👋

[haydentherapper]

@znewman01 Any other thoughts? I'm fine with this since we already do support providing the trust root via --cert-chain, this is just an optimization to also not require --certificate.

[znewman01]

@znewman01 Any other thoughts? I'm fine with this since we already do support providing the trust root via --cert-chain, this is just an optimization to also not require --certificate.

[haydentherapper]

Thanks for fixing this!