feat(cd): sign released container images and Helm chart

[hainenber]

Description

Sign released container images with Cosign and Helm chart with helm package --sign

Related Issue

If this pull request is related to any issue, please mention it here. Additionally, make sure that the issue is assigned to you before submitting this pull request.

Closes #141

Checklist

• I have read the contributing documantation.
• I signed and signed-off the commits (git commit -S -s ...)
• I have correctly attributed the author(s) of the code.
• I have tested the changes locally.
• I have followed the project's style guidelines.
• I have updated the documentation, if necessary.
• I have added tests, if applicable.

Screenshots (if applicable)

N/A

Additional Notes

In order for testing this change, please create a GPG private key and a Cosign private key and deposit as GH secrets for this repo

---

Please refer to the CONTRIBUTING.md file for more information on how to contribute to this project.

[rbtr]

@hainenber thanks for picking this up, I'll give this a thorough review ASAP.

One initial question - I noticed in the GH sample workflow they don't have any explicit keys. I think this is because they are using OIDC integration to do "keyless" signing?

This seems preferable since we wouldn't need to manage GPG keys as secrets. Do you think that's something we can do here?

[hainenber]

Sure, I think it's feasible if GH decided to demonstrate their sample workflow as such. Will tinker around a bit

[hainenber]

All relevant container images and Helm chart are now signed with cosign in keyless manner. There's 1 caveat with signing Helm chart though; it'll generate a cosign.bundle - a base64-encoded string containing both certificate and signature.

I decide to make it into a build artifact so release engineers can bundle (no pun, I swear) into a release if needed.

[rbtr]

This is great, thanks! Left some feedback based on my testing

[rbtr]

I did some experimenting with this and I couldn't quite get it to work: https://github.com/rbtr/retina/actions/runs/8474428281
Here are the changes I made: rbtr@04c7fe6

Ran in to a few problems:

• the signing step uses the image digest. We don't have that image loaded in docker, it's in the buildx context, so we have to pull it or get the digest earlier and save it. If we try to pull it, we run in to platform issues - it's tricky to pull an image for a platform that's different from the host, especially if it's Windows.
• I can't figure out the command to verify a signature...it should be something like cosign verify ghcr.io/rbtr/retina/retina-agent@sha256:1d38a1d76f032a95ea502ec1d52dc2fc1475aa18c6a6c9ed41841c9a7fe2f0d8 --certificate-identity "https://github.com/rbtr/retina" --certificate-oidc-issuer "https://token.actions.githubusercontent.com" but I just get Error: no matching signatures. I think the certificate-identity is wrong? But when I use certificate-identity-regexp "*", still nothing.

Overall I think this is very close with the changes suggested earlier, if we can figure out a solution to the digest problem. For acceptance criteria I would love to see a published signed image (and be able to verify it with cosign) to demonstrate that this functions 🙂

[hainenber]

I did some experimenting with this and I couldn't quite get it to work: https://github.com/rbtr/retina/actions/runs/8474428281 Here are the changes I made: rbtr@04c7fe6

Ran in to a few problems:

• the signing step uses the image digest. We don't have that image loaded in docker, it's in the buildx context, so we have to pull it or get the digest earlier and save it. If we try to pull it, we run in to platform issues - it's tricky to pull an image for a platform that's different from the host, especially if it's Windows.
• I can't figure out the command to verify a signature...it should be something like cosign verify ghcr.io/rbtr/retina/retina-agent@sha256:1d38a1d76f032a95ea502ec1d52dc2fc1475aa18c6a6c9ed41841c9a7fe2f0d8 --certificate-identity "https://github.com/rbtr/retina" --certificate-oidc-issuer "https://token.actions.githubusercontent.com" but I just get Error: no matching signatures. I think the certificate-identity is wrong? But when I use certificate-identity-regexp "*", still nothing.

Overall I think this is very close with the changes suggested earlier, if we can figure out a solution to the digest problem. For acceptance criteria I would love to see a published signed image (and be able to verify it with cosign) to demonstrate that this functions 🙂

Thanks for your effort on testing out the changes on your forked repo! My apologies for having this under the radar as I tend to disable GH actions in my forked repos to cut CI costs

Re: 1st point, I'm trying to store image digest with this --metadata-file. This should be able to cover this concern once I'm done with another round of tinkering.

Onto the 2nd point, from cosign docs, it seems the OIDC issuer for GH-stored images is "https://github.com/login/oauth" and the cert's identity is the signer's address. Another round of tinkering is needed to validate this ofc.

Onto the lab, hopefully a short round this time.

To save up your effort to other Retina's more pressuring issues, I'll convert this one to Draft and will seek your reviews once it deems as worthy.

[rbtr]

Thanks for your effort on testing out the changes on your forked repo! My apologies for having this under the radar as I tend to disable GH actions in my forked repos to cut CI costs

Re: 1st point, I'm trying to store image digest with this --metadata-file. This should be able to cover this concern once I'm done with another round of tinkering.

Onto the 2nd point, from cosign docs, it seems the OIDC issuer for GH-stored images is "https://github.com/login/oauth" and the cert's identity is the signer's address. Another round of tinkering is needed to validate this ofc.

Onto the lab, hopefully a short round this time.

To save up your effort to other Retina's more pressuring issues, I'll convert this one to Draft and will seek your reviews once it deems as worthy.

Happy to collab on this, no problem at all. This is an important feature to me 🙂
I can keep pulling it into my fork or you can PR there to get Actions runs? Let me know how I can help here.

[hainenber]

hi @rbtr, I've fixed the signing for released images, charts and manifest. These are the successful actions in my forked repo

You can verify the artifacts via cosign verify <oci_image_url> --certificate-oidc-issuer https://token.actions.githubusercontent.com --certificate-identity="https://github.com/hainenber/retina/.github/workflows/release-images.yaml@refs/heads/sign-released-images"". I'm verifying a manifest here

[rbtr]

thanks @hainenber! I will test this out 🙂

[hainenber]

Please do! Bon apetit :D

[nddq]

left some nit comments, thanks!

[rbtr]

Functionally LGTM! This is great, thanks @hainenber 🍻
Aside from anyone else's comments, my last ask would be to updated the README and docs.
I'm thinking we add a block such as:

### Verify signed images
Retina images published to GHCR are cryptographically signed. You can verify their provenance with [`sigstore/cosign`](https://github.com/sigstore/cosign):

```shell
REPO=microsoft/retina # or your repo
IMAGE=retina-operator # or other image to verify
TAG=v0.0.6 # or other tag to verify OR replace with the image SHA256
cosign verify ghcr.io/$REPO/$IMAGE:$TAG --certificate-oidc-issuer https://token.actions.githubusercontent.com --certificate-identity-regexp="https://github.com/$REPO" -o text
```

to the README, and duplicate it to a site page such as docs/installation/verify-signed-images.md so that it's on retina.sh also 🙂

[hainenber]

@rbtr I've addressed Quang's comments and added relevant documentation as suggested by you. I think yours is already succinct so pardon me for a whole copy-paste :D

[rbtr]

@hainenber I committed newlines at the end of your markdown files to make the linter happy, everything else LGTM