Added support for cosign.

closes pulp#1165
closes pulp#1166
closes pulp#1167

CHANGES/1165.feature

@@ -0,0 +1 @@
+As a user I can serve cosign produced signatures, SBOMS and attestations.

CHANGES/1166.feature

@@ -0,0 +1 @@
+Added ability to mirror cosign signatures, SBOMs and attestations.

CHANGES/1167.feature

@@ -0,0 +1 @@
+As a user I can push a cosign signature, attestation or SBOM to Pulp Registry.

docs/workflows/cosign-support.rst

@@ -0,0 +1,101 @@
+.. _cosign-support:
+
+
+Mirror
+======
+Being a OCI compliant registry Pulp Container registry can natively mirror cosign signatures
+wich are stored as an OCI image::
+
+ {
+    "schemaVersion": 2,
+    "mediaType": "application/vnd.oci.image.manifest.v1+json",
+    "config": {
+        "mediaType": "application/vnd.oci.image.config.v1+json",
+        "digest": "sha256:f35028aa1563f37ccbaa0b32c57777ffbd8e9e3d81d739fec0022995e58a375a",
+        "size": 153
+    },
+    "layers": [
+        {
+            "mediaType": "application/vnd.dev.cosign.simplesigning.v1+json",
+            "digest": "sha256:d3370bd32b32aba43de2b45bb4a2de2fb5c95fd2edbe738acbc3bc595b80c456",
+            "size": 305,
+            "annotations": {
+                "dev.cosignproject.cosign/signature": "MEUCIBWDnTKhbf5x3mSuEHWkv3ixloIFXeDpfXipF9szqrd5AiEA+UU5J84gQ9JnmT6QZAXiPXqSoDVW0CXQYssGh63e9Ro="
+            }
+        }
+    ]
+ }
+
+
+During the syncronization task, Pulp will automatically mirror cosign signatures or atomic
+signatures (accessible via signatures extentions API).
+
+
+Sign
+====
+Pulp Container registry can host cosign signature which can be pushed via cosign or podman clients:
+
+Cosign::
+
+
+  # This command creates an ECDSA-P256 key pair (a private and a public key). 
+  cosign generate-key-pair
+  cosign sign --key cosign.key pulp-registry/ipanova/cosign-test:latest
+
+or via Podman::
+
+  podman push pulp-registry/ipanova/cosign-test:latest --sign-by-sigstore-private-key cosign.key
+
+ .. warning:
+ To use this with images hosted on image registries, the relevant registry or repository must have
+ the use-sigstore-attachments option enabled in containers-registries.d(5). This specifies whether
+ sigstore image attachments (signatures, attestations and the like) are going to be read/written
+ along with the image. If disabled, the images are treated as if no attachments exist; attempts to
+ write attachments fail.
+
+As a result of this operation ``ipanova/cosign-test:latest`` image is signed and its
+cosign signature is stored in the registry as an OCI image. Cosign uses a fixed naming convention
+to decide the name for a separate image, at which we can store the signature. The tag name resolved
+to a fixed digest of the image/or manifest list which is being signed in a form of ``sha256-12345.sig``
+
+The payload of the signature will be store as an image layer::
+
+  {
+    "critical": {
+        "identity": {
+            "docker-reference": "pulp-registry/ipanova/cosigned:latest"
+        },
+        "image": {
+            "docker-manifest-digest": "sha256:81cd171c4eda75046c31d6ed26f1241bbfa9326640613430be780ea931b02c24"
+        },
+        "type": "cosign container image signature"
+    },
+    "optional": {
+        "creator": "containers/image 5.23.1",
+        "timestamp": 1673006074
+    }
+  }
+
+
+  .. note:
+      Besides cosign signature Pulp Container Registry can mirror and host SBOMs and attestations.
+
+
+The verify
+==========
+
+Signature verification can be done via cosign or podman clients::
+
+  cosign verify --key cosign.pub pulp-registry/ipanova/cosign-test:latest
+
+When using podman client the policy.json file should be properly configured per specs.
+A new requirement type ``sigstoreSigned`` has been introduced:
+
+https://github.com/containers/image/blob/main/docs/containers-policy.json.5.md#sigstoresigned
+
+ .. warning::
+ To use this with images hosted on image registries, the relevant registry or repository must have
+ the use-sigstore-attachments option enabled in containers-registries.d(5). This specifies whether
+ sigstore image attachments (signatures, attestations and the like) are going to be read/written
+ along with the image. If disabled, the images are treated as if no attachments exist; attempts to
+ write attachments fail.

docs/workflows/index.rst

@@ -67,4 +67,5 @@ Managing Signatures
    :maxdepth: 2
 
    sign-images
+   cosign-support
    verify-images

docs/workflows/sign-images.rst

@@ -1,7 +1,7 @@
 .. _sign-images:
 
-Image Signature Configuration
-=============================
+Atomic Container Image Signature Configuration via a Pulp signing service
+=========================================================================
 
 Administrators can add a container manifest signing service to The Pulp Registry using the command
 line tools. Users may then associate the signing service with container repositories.
@@ -12,10 +12,10 @@ The example below demonstrates how a manifest signing service can be created usi
    hardware cryptographic device.
 
 2. Create a signing script that accepts a manifest path as the only argument. The script invokes
-   ``skopeo standalone-sign`` command that generates a container signature for the image manifest,
-   using the key specified via the ``PULP_SIGNING_KEY_FINGERPRINT`` environment variable. The script
-   should then print out a JSON structure with the following format. The path of the created
-   signature is a relative path inside the current working directory::
+   ``skopeo standalone-sign`` command that generates an atomic container signature for the image
+   manifest, using the key specified via the ``PULP_SIGNING_KEY_FINGERPRINT`` environment variable.
+   The script should then print out a JSON structure with the following format. The path of the
+   created signature is a relative path inside the current working directory::
 
        {"signature_path": "signature"}
 
@@ -30,7 +30,7 @@ The example below demonstrates how a manifest signing service can be created usi
         IMAGE_REFERENCE="$REFERENCE"
         SIGNATURE_PATH="$SIG_PATH"
 
-        # Create container signature
+        # Create atomic container signature
         skopeo standalone-sign $MANIFEST_PATH $IMAGE_REFERENCE $FINGEPRINT --output $SIGNATURE_PATH
         # Check the exit status
         STATUS=$?
@@ -80,9 +80,12 @@ The example below demonstrates how a manifest signing service can be created usi
 
 Afterwards, users are able to sign selected content by the provided script.
 
+   .. warning::
+       Underlying singing facility with sign anything what represents an OCI container image.
 
-Sign Images Pushed to the Registry
-==================================
+
+Sign Images Pushed to the Registry via Pulp signing service
+-----------------------------------------------------------
 
 Given that an image is pushed to the Pulp Registry via ``podman/docker push`` or via the standard
 DockerRegistry v2 push API, a repository is created containing it::
@@ -140,7 +143,7 @@ operation or it can be explictly specified in the following manner::
              "worker": "/pulp/api/v3/workers/eb65c2d9-31b2-47dc-847e-dad0e744c539/"
          }
 
-Upon task complection, a signature is created and added to the repository::
+Upon task complection, an atomic container signature is created and added to the repository::
 
       $ http GET https://pulp.example.com/pulp/api/v3/repositories/container/container-push/3b279a32-b313-44bf-ad41-4359a92cae24/versions/10/
         {
@@ -201,8 +204,8 @@ Upon task complection, a signature is created and added to the repository::
         }
 
 
-Sign Images Mirrored into the Registry
-======================================
+Sign Images Mirrored into the Registry via Pulp signing service
+---------------------------------------------------------------
 
 It is possible to sign content that was synchronized from remote registries.
 If the content was synced together with signatures, upon signing task completion new signatures will be
@@ -306,15 +309,15 @@ by proviging ``tags_list`` option to the call.
 Note that ``manifest lists`` are not signed, instead all the image manifests that manifest lists
 contain, are signed.
 
-Manage Signatures via the Extensions API
-========================================
+Manage Atomic Container Signatures via the Extensions API
+=========================================================
 
 This API exposes an endpoint for reading and writing image signatures. Users should configure the
 sigstore section in the `registries.d file <https://github.com/containers/image/blob/main/docs/containers-registries.d.5.md>`_
 accordingly to benefit from the API.
 
-Reading Image Signatures
-------------------------
+Reading Atomic Container Image Signatures
+-----------------------------------------
 
 To read existing signatures, issue the following GET request::
 
@@ -323,8 +326,8 @@ To read existing signatures, issue the following GET request::
 Signatures are retrieved by container clients automatically if the policy requires so. The policy is
 defined in the file ``/etc/containers/policy.json``.
 
-Writing Image Signatures
-------------------------
+Writing Atomic Container Image Signatures
+-----------------------------------------
 
 To add a new signature to an image, execute the following PUT request::
 

docs/workflows/sync.rst

@@ -72,9 +72,9 @@ the ``sigstore`` field when creating a Remote.
 
 .. note::
     Some registries provide docker API extensions for ``atomic container signature`` type only, or
-    have ``cosign`` type signatures that are stored as a separate OCI artifact in a registry.
-    Pulp will automatically sync signatures provided via the docker API extension. At the moment,
-    `cosign` signatures are not supported.
+    have ``cosign`` type signatures that are stored as a separate OCI image in a registry.
+    Pulp will automatically sync signatures provided via the docker API extension or cosign
+    signatures stored as an OCI image.
 
 
 Reference: `Container Remote Usage <../restapi.html#tag/Remotes:-Container>`_

pulp_container/app/json_schemas.py

@@ -2,11 +2,15 @@
 
 
 def get_descriptor_schema(
-    allowed_media_types, additional_properties=None, additional_required=None
+    allowed_media_types=None, additional_properties=None, additional_required=None
 ):
     """Return a concrete descriptor schema for manifests."""
+
+    media_type = {"type": "string"}
+    if allowed_media_types:
+       media_type["enum"] = allowed_media_types
     properties = {
-        "mediaType": {"type": "string", "enum": allowed_media_types},
+        "mediaType": media_type,
         "size": {"type": "number"},
         "digest": {"type": "string"},
         "annotations": {"type": "object", "additionalProperties": True},
@@ -71,16 +75,7 @@ def get_descriptor_schema(
         "config": get_descriptor_schema([MEDIA_TYPE.CONFIG_BLOB_OCI]),
         "layers": {
             "type": "array",
-            "items": get_descriptor_schema(
-                [
-                    MEDIA_TYPE.REGULAR_BLOB_OCI_TAR,
-                    MEDIA_TYPE.REGULAR_BLOB_OCI_TAR_GZIP,
-                    MEDIA_TYPE.REGULAR_BLOB_OCI_TAR_ZSTD,
-                    MEDIA_TYPE.FOREIGN_BLOB_OCI_TAR,
-                    MEDIA_TYPE.FOREIGN_BLOB_OCI_TAR_GZIP,
-                    MEDIA_TYPE.FOREIGN_BLOB_OCI_TAR_ZSTD,
-                ]
-            ),
+            "items": get_descriptor_schema(),
         },
     },
     "required": ["schemaVersion", "config", "layers"],

pulp_container/app/tasks/sync_stages.py

@@ -133,15 +133,9 @@ async def run(self):
         to_download = []
         BATCH_SIZE = 500
 
+        # it can be wether a separate sigstore location or registry with extended signatures API
         signature_source = await self.get_signature_source()
 
-        if signature_source is None and self.signed_only:
-            raise ValueError(
-                "It is requested to sync only signed content but no sigstore URL is "
-                "provided. Please configure a `sigstore` on your Remote or set "
-                "`signed_only` to `False` for your sync request."
-            )
-
         async with ProgressReport(
             message="Downloading tag list", code="sync.downloading.tag_list", total=1
         ) as pb:
@@ -176,11 +170,25 @@ async def run(self):
                 saved_artifact, content_data, raw_data, response = await artifact
 
                 digest = response.artifact_attributes["sha256"]
+                tag_name = response.url.split("/")[-1]
+
+                # Look for cosign signatures
+                # cosign signature has a tag convention 'sha256-1234.sig'
+                if signed_only and not signature_source:
+                    if not tag_name.enswith('.sig') and digest+'.sig' not in tag_list:
+                        # skip this tag, there is no corresponding signature
+                        log.info(
+                                    "The unsigned image {digest} can't be synced "
+                                    "due to a requirement to sync signed content "
+                                    "only.".format(digest=digest)
+                                )
+                         # Count the skipped tagks as parsed too.
+                         await pb_parsed_tags.aincrement()
+                        continue
 
                 media_type = determine_media_type(content_data, response)
                 validate_manifest(content_data, media_type, digest)
 
-                tag_name = response.url.split("/")[-1]
                 tag_dc = DeclarativeContent(Tag(name=tag_name))
 
                 if media_type in (MEDIA_TYPE.MANIFEST_LIST, MEDIA_TYPE.INDEX_OCI):