Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSDOCS#12209: Sigstore signature verification for core OCP images #81985

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
159 changes: 159 additions & 0 deletions modules/security-understanding-container-validation.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
// Module included in the following assemblies:
//
// * security/container_security/security-understanding.adoc

:_mod-docs-content-type: CONCEPT
[id="security-understanding-container-validation_{context}"]
= Container security and validation
Comment on lines +3 to +7
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pedantic: “container security” is much larger than the scope of this document.

“Container image autenticity?” “Preventing supply chain attacks on container images”?


{product-title} is released as a set of container images that comprise the different functions of the software.
These images must be validated in order to ensure that they have not been tampered with by an external party, and to ensure that they are the intended images provided by Red{nbsp}Hat.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can rephrase to highlight "customer admins can use ClusterImagePolicy for their own policy too!" with something like:

{product-title} runs containers from containers images.
These images may be validated in order to ensure that they have not been tampered with by an external party.

Then the following sections are:

  • Image integrity validation at pull time, which can be dropped or tweaked.
  • Image authenticity validation, introducing ClusterImagePolicy as a generic tool.
  • Trusting an {product-title} installation, which gets into Red Hat images for the first time.
    • Unfortunately for continuity, we have to take a bit of a detour into bootstrapping/installer trust, which is complicated.
    • Trusting a connected installation, still talking about client/installer trust.
    • Trusting a disconnected installation, still talking about client/installer trust.

Hrm, and seems like we want an additional section like Trusting an {product-title} post-install, that points out the trusted installer will roll out a trusted release image with the openshift ClusterImagePolicy (for TechPreview clusters), and that that pulls in the ClusterImagePolicy security discussed under Image authenticity validation for images in the quay.io/openshift-release-dev/ocp-release scope. That prevents folks from running an quay.io/openshift-release-dev/ocp-release image that hasn't been signed by the trusted Red Hat key used to sign OCP release images, e.g. in the event that quay.io or a customer-configured mirror registry is compromised.

Or maybe that only matters when we start to talk about updates, and we're puning on that for now, because really, you'd need a compromised mirror and an admin tricked into updating to a malicious digest or a by-tag release image reference.

Copy link
Contributor

@mtrmac mtrmac Sep 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where I’m a little lost is that the goal, per the title of the PR (not of the document), is to talk about OCP product images … which don’t currently use sigstore / ClusterImagePolicy IIRC (just in a tech preview). So if that is the goal, why bring ClusterImagePolicy into the picture at all, now? It would certainly be relevant in a future release, but I don’t think we are there yet.

OTOH the title of the document (as opposed to the PR) talks about container “security and validation”, i.e. not restricted to core OCP, and in that case ClusterImagePolicy is certainly the most relevant part, and the OCP trust bootstrapping a side issue.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OCP trust bootstrapping is more of an installer issue, and I'm fine with them leading there, and doc'ing the long-GA processes around that.

ClusterImagePolicy is more of a node issue, and I'm fine with the node folks leading on that. There was some work associated with the 4.16 tech-preview of the type in #76613, #77203, #78396.

I'm here because 4.17 is releasing the tech-preview openshift ClusterImagePolicy roughly as laid out in this enhancement. There's a release note landed today via #81997, but it doesn't cover mechanics, and I'd expect customer security folks to have some questions about mechanics. But maybe they don't? I'm fine letting this float until there's some clarity around the kinds of security questions customers ask, and if anyone is particularly interested in if/how to manage installer trust or have customers create their own ClusterImagePolicies.


[id="security-understanding-image-integrity_{context}"]
== Image integrity validation at pull time

When images are downloaded, or pulled, from a registry, an {product-title} cluster performs the following actions to ensure the integrity of each image:

. The cluster downloads an image manifest and calculates its digest.

. The cluster compares the calculated digest of the image manifest with the known digest.
If the digest values do not match, the image is rejected without further processing.

. The cluster downloads an image config file and calculates its digest.

. The cluster compares the calculated digest of the image config file with the digest that is provided in the image manifest.
If the digest values do not match, the image is rejected without further processing.

. For each file, or layer, that comprises the container image, the cluster downloads the file and calculates its digest.

. The cluster compares the calculated digest of each file that makes up the image with the corresponding digest that is provided in the image manifest.
If the digest values do not match, the image is rejected.

Once the cluster completes all of the previous steps, the integrity of the image has been validated.

[NOTE]
====
The image manifest is validated only when the manifest digest is known, which is true in the following cases:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this should go into that level of detail:

  • Yes, if an image is being pulled by an unsigned tag, we don’t know the manifest digest, and we trust the registry to return the correct and unmodified manifest
  • (But the config and layer integrity checks do happen.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any additional content you'd suggest to add this info in?

At the end of this note, should I add something like "Even if the image manifest is not validated in these cases, the image config and layers are still checked for integrity"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added my suggested sentence for now

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

“for integrity against the [untrusted] manifest”? I’m very unsure about the “untrusted” word — it’s correct in the security jargon, but it might carry the connotation of “specifically known to be dangerous”.

My general opinion is that none of the details of configs/layers need to be included, and then we don’t need to discuss this either; but if they are discussed, like that.


. The image is pulled by specifying the SHA256 image digest, for example `quay.io/openshift-release-dev/ocp-release@sha256:ea7ac3ad42169b39fce07e5e53403a028644810bee9a212e7456074894df40f3`.

. The image signature is also being validated. See “Image authenticity validation” for more information.

However, if the image manifest is not validated in these cases, the image config and layers are still checked for integrity.
====

[id="security-understanding-image-authenticity_{context}"]
== Image authenticity validation

Cluster administrators can use `ClusterImagePolicy` resources to declare an image policy for the cluster, requiring signature validation and other policy checks in addition to image integrity.

:FeatureName: Configuring cluster image policies
include::snippets/technology-preview.adoc[]

.Example `ClusterImagePolicy` resource 1
[source,yaml]
----
apiVersion: config.openshift.io/v1alpha1
kind: ClusterImagePolicy
metadata:
name: mypolicy-0
spec:
scopes:
- test0.com <1>
- test1.com
policy:
rootOfTrust:
policyType: PublicKey
publicKey:
keyData: dGVzdC1rZXktZGF0YQ==
----
<1> This policy for `test0.com` and the policy from `mypolicy-1` will be appended together.


.Example `ClusterImagePolicy` resource 2
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’d recommend leading with the PublicKey example, that’s really more relevant to enterprises (and it’s how RH images are signed). Similarly the signedIdentity section may be critically important for some customers but should not be necessary in the default deployments.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I can switch the order of the examples.

[source,yaml]
----
apiVersion: config.openshift.io/v1alpha1
kind: ClusterImagePolicy
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
kind: ClusterImagePolicy
apiVersion: config.openshift.io/v1alpha1
kind: ClusterImagePolicy

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, does this need to be added to the other example too?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, all the ClusterImagePolicy YAML examples will need this apiVersion line

metadata:
name: mypolicy-1
spec:
scopes:
- test0.com
policy:
rootOfTrust:
policyType: FulcioCAWithRekor
fulcioCAWithRekor:
fulcioCAData: dGVzdC1jYS1kYXRhLWRhdGE=
rekorKeyData: dGVzdC1yZWtvci1rZXktZGF0YQ==
fulcioSubject: <1>
oidcIssuer: https://OIDC.example.com
signedEmail: test-user@example.com
signedIdentity:
matchPolicy: RemapIdentity
remapIdentity:
prefix: test-remap-prefix
signedPrefix: test-remap-signed-prefix
Comment on lines +95 to +96
Copy link
Contributor

@mtrmac mtrmac Sep 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(For the record, these values are invalid and will either be rejected, or won’t do anything — real prefixes contain at least a host name with at least one dot, or a host:port. But then again, the …Data fields don’t contain really realistic inputs either.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
prefix: test-remap-prefix
signedPrefix: test-remap-signed-prefix
prefix: test.remap.prefix
signedPrefix: test.remap.signed.prefix

What do you think about this? theses are valid for the regex validation, and look more like a prefix of the repository

Copy link
Contributor

@mtrmac mtrmac Sep 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that if we care about the example values, it would be better to make this all realistic. scopes: … mirror.internal/vendorA/productA; prefix: mirror.internal/vendorA/productA; signedPrefix: vendorA.example/productA or something (and maybe the same vendorA.example in the Fulcio conditions).

But, also, it seems to me that details of these CRs are really not the primary focus of the article.


----
<1> Specifies the OIDC issuer and the email of the Fulcio authentication configuration.

The previous example resources define two policies, both of which apply to `test0.com` pullspecs, and only one of which applies to `test1.com` pullspecs.

When image integrity is validated at pull time for images whose pullspecs have configured policies with matching `scopes` values, CRI-O will retrieve any Sigstore signature images associated with the image and check them against the configured policies.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be a bit more precise about what “matching” means … and I should double-check.
In the underlying c/image, only the closest matching scope is used; I’m not immediately sure wether {Cluster,}ImagePolicy also uses the “closest” scope semantics or whether the wider scopes are “inherited” down into the narrower ones.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ClusterImagePolicy API Godocs include:

If multiple scopes match a given image, only the policy requirements for the most specific scope apply. The policy requirements for more general scopes are ignored.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would the wording change look like in light of this feedback? Or do I need to add another sentence to clarify what matching means?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd just inline the Godocs sentence after the one we're commenting on here, so folks here about how "maching scopes..." matter, and then the next sentence gives more precision about what qualifies as "matching".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I added the text snippet you provided

If multiple scopes match a given image, only the policy requirements for the most specific scope apply.
The policy requirements for more general scopes are ignored.

If signature verification fails for any configured policy, the associated image pull will be rejected.
Events will be created in the associated namespace explaining the verification failure, and the Pod `status` for containers consuming that image will set a relevant `state`.

[id="security-understanding-trusting-install_{context}"]
== Trusting an {product-title} installation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you also want to have a section about trusting updates, describing the Cincinnati stream?

It might not be relevant for this article, I don’t know.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can add that section from this blog, which I'm using as a reference for this overall PR, if you think it would be valuable to add. Or if it's something that can wait until a v2 of this doc, then I'm okay with that, too.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can exclude Cincinnati/OCP updates to ship a v1 of this content, and then we can think through if/how to add something in that space in follow-up work.


In order to validate the integrity and authenticity of an {product-title} installation, all of its components must be verified, including container images as well as the binaries needed to install a cluster.

The binaries required for a cluster installation are the installation program (`openshift-install`) and one of the {op-system} base image distributions.

If the OpenShift CLI (`oc`) is used during an installation, it is verified through the following steps:

. The OpenShift CLI (`oc`) archive is signed and therefore trusted.

. The OpenShift CLI (`oc`) binary is transitively trusted.

Each version of {product-title} has a unique release image.
A release image represents a single release and includes a manifest containing all of the images that comprise the release.
The method of validating an installation depends on whether the cluster is being installed in a connected or disconnected environment.

[id="security-understanding-trusting-connected-install_{context}"]
=== Trusting a connected installation

For a connected installation of {product-title}, the chain of trust is established in the following order:

. The `openshift-install` archive is trusted using a SHA256sum signature.

. The `openshift-install` binary is transitively trusted.

. The release image digest is hard coded into the `openshift-install` binary, therefore the digest is transitively trusted.

. Image digests of the images that comprise the release are hard coded in the release image, therefore the digests are transitively trusted.

. Because images are referenced using a digest, their integrity is verified when they are pulled from a registry.

[id="security-understanding-trusting-disconnected-install_{context}"]
=== Trusting a disconnected installation

For a disconnected installation of {product-title}, which requires mirroring images to a registry that is available in the disconnected environment, the chain of trust is established in the following order:

. The `openshift-install` binary is obtained by the trusted `oc` binary passing a trusted release image pull spec, in digest format, from the mirrored repository.
Therefore, the `openshift-install` binary is transitively trusted.
For each release, a manifest of the release is published and signed.
From this manifest, you can obtain a trusted digest of a release image.

. Image mirroring is configured in the `install-config` manifest.
The rest of the installation, which involves verifying the digests for the release image and the other images that comprise {product-title}, proceeds in the same way as a connected installation.

[NOTE]
====
There is no process to verify the mirroring process or the mirror registry.
If images have not been mirrored correctly, the installation program will fail to pull the correct images, and the installation will fail.
====
3 changes: 3 additions & 0 deletions security/container_security/security-understanding.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -66,3 +66,6 @@ include::modules/security-understanding-openshift.adoc[leveloffset=+1]
.Additional resources
* xref:../../architecture/architecture.adoc#architecture[{product-title} architecture]
* link:https://www.redhat.com/en/resources/openshift-security-guide-ebook[OpenShift Security Guide]

// Container security and validation
include::modules/security-understanding-container-validation.adoc[leveloffset=+1]