podman pull from pulp fails with: unsupported schema version 2

[ipanova]

Steps to reproduce

1. create a container repo
2. create a remote http https://pulp3-source-fedora34.fluffy.example.com/pulp/api/v3/remotes/container/container/ name=lala upstream_name=riadh_hamdi/ceph-zabbix url=https://quay.io
3. sync repo
4. distribute repo http https://pulp3-source-fedora34.fluffy.example.com/pulp/api/v3/distributions/container/container/ name=ina base_path=ina repository_version= /pulp/api/v3/repositories/container/container/dc8ce8e4-1cc0-495c-be5d-513e63dd931d/versions/1/
5. [erform podman pull podman --log-level=debug pull pulp3-source-fedora34.fluffy.example.com/ina
6. Observe traceback:

$ podman --log-level=debug pull pulp3-source-fedora34.fluffy.example.com/ina
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called pull.PersistentPreRunE(podman --log-level=debug pull pulp3-source-fedora34.fluffy.example.com/ina) 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/vagrant/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] systemd-logind: Unknown object '/'.          
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/vagrant/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/vagrant/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/vagrant/.local/share/containers/storage/volumes 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /home/vagrant/.config/cni/net.d/87-podman.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 13             
DEBU[0000] Pulling image pulp3-source-fedora34.fluffy.example.com/ina (policy: always) 
DEBU[0000] Looking up image "pulp3-source-fedora34.fluffy.example.com/ina" in local containers storage 
DEBU[0000] Trying "pulp3-source-fedora34.fluffy.example.com/ina" ... 
DEBU[0000] Trying "pulp3-source-fedora34.fluffy.example.com/ina:latest" ... 
DEBU[0000] Trying "pulp3-source-fedora34.fluffy.example.com/ina:latest" ... 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" 
DEBU[0000] Attempting to pull candidate pulp3-source-fedora34.fluffy.example.com/ina:latest for pulp3-source-fedora34.fluffy.example.com/ina 
DEBU[0000] parsed reference into "[overlay@/home/vagrant/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]pulp3-source-fedora34.fluffy.example.com/ina:latest" 
Trying to pull pulp3-source-fedora34.fluffy.example.com/ina:latest...
DEBU[0000] Copying source image //pulp3-source-fedora34.fluffy.example.com/ina:latest to destination image [overlay@/home/vagrant/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]pulp3-source-fedora34.fluffy.example.com/ina:latest 
DEBU[0000] Trying to access "pulp3-source-fedora34.fluffy.example.com/ina:latest" 
DEBU[0000] No credentials for pulp3-source-fedora34.fluffy.example.com found 
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  No signature storage configuration found for pulp3-source-fedora34.fluffy.example.com/ina:latest, using built-in default file:///home/vagrant/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/pulp3-source-fedora34.fluffy.example.com 
DEBU[0000] GET https://pulp3-source-fedora34.fluffy.example.com/v2/ 
DEBU[0000] Ping https://pulp3-source-fedora34.fluffy.example.com/v2/ status 401 
DEBU[0000] GET https://pulp3-source-fedora34.fluffy.example.com/token/?scope=repository%3Aina%3Apull&service=pulp3-source-fedora34.fluffy.example.com 
DEBU[0000] GET https://pulp3-source-fedora34.fluffy.example.com/v2/ina/manifests/latest 
DEBU[0000] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v1+prettyjws" 
DEBU[0000] Using blob info cache at /home/vagrant/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] IsRunningImageAllowed for image docker:pulp3-source-fedora34.fluffy.example.com/ina:latest 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
DEBU[0000] Error pulling candidate pulp3-source-fedora34.fluffy.example.com/ina:latest: initializing image from source docker://pulp3-source-fedora34.fluffy.example.com/ina:latest: unsupported schema version 2 
Error: initializing image from source docker://pulp3-source-fedora34.fluffy.example.com/ina:latest: unsupported schema version 2

There is a discrepancy from what we store in DB and the actual manifest.json artifact

$ http   https://pulp3-source-fedora34.fluffy.example.com/pulp/api/v3/content/container/manifests/ 

HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 866
Content-Type: application/json
Correlation-ID: da5e0bec637c4366b60a4819a72a0677
Date: Wed, 29 Jun 2022 14:38:58 GMT
Referrer-Policy: same-origin
Server: nginx/1.20.1
Strict-Transport-Security: max-age=15768000
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "artifact": "/pulp/api/v3/artifacts/6dfaf152-e309-4884-84e5-480f51c9bfc8/",
            "blobs": [
                "/pulp/api/v3/content/container/blobs/3cfe1dc2-d8a3-4e5e-a8ce-1eb577736844/",
                "/pulp/api/v3/content/container/blobs/b4b6b051-9b21-4f42-b67c-f377e9eb7d0e/",
                "/pulp/api/v3/content/container/blobs/79e6a9dc-472a-473a-8f6c-b32bde9fe198/",
                "/pulp/api/v3/content/container/blobs/a1d5528c-3077-4f88-b28a-4c769b6f9f06/"
            ],
            "config_blob": "/pulp/api/v3/content/container/blobs/a4a0b057-1ffd-4fea-8e2d-5e42198cf0b7/",
            "digest": "sha256:bad379fd10130963eeedc03f203ba3578bd1e54e3748d208e617f7ee31afdc63",
            "listed_manifests": [],
            "media_type": "application/vnd.docker.distribution.manifest.v1+json",
            "pulp_created": "2022-06-29T14:33:43.843062Z",
            "pulp_href": "/pulp/api/v3/content/container/manifests/c298963a-7cbe-4e32-b8e7-bfa3754f23e9/",
            "schema_version": 2
        }
    ]
}


(pulp) [vagrant@pulp3-source-fedora34 pulp_container]$ less /var/lib/pulp/media/artifact/ba/d379fd10130963eeedc03f203ba3578bd1e54e3748d208e617f7ee31afdc63 |python -m json.tool

{
    "schemaVersion": 2,
    "config": {
        "mediaType": "application/vnd.oci.image.config.v1+json",
        "digest": "sha256:c4fa3965dac905ff648c2d8e707899e2f6f0a10b48e6b728e97f32cb6500e9ed",
        "size": 3841
    },
    "layers": [
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:ac56bdc7f9934acede05653e9e01e73e961c31818b522c0732ad35350bb3a82b",
            "size": 85633977
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:22f677655049d4c2e6cd9e49ca9ed20f34ac175ef0c82f5c5eabc79031c1c29a",
            "size": 1876
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:b0faf8bf3c2afb8c9f9beb648acc08a68c5d205710b2f2828cc21d59e051ca3d",
            "size": 241681193
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:7113491e908546d75cc713354580cc9e9766be7ee033c4fcb8ea66446e23a959",
            "size": 173313
        }
    ]
}
(pulp) [vagrant@pulp3-source-fedora34 pulp_container]$ http   https://pulp3-source-fedora34.fluffy.example.com/pulp/api/v3/content/container/blobs/ 
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 1516
Content-Type: application/json
Correlation-ID: ab87de51f5f2447c9725078228ec8839
Date: Wed, 29 Jun 2022 14:40:18 GMT
Referrer-Policy: same-origin
Server: nginx/1.20.1
Strict-Transport-Security: max-age=15768000
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "count": 5,
    "next": null,
    "previous": null,
    "results": [
        {
            "artifact": "/pulp/api/v3/artifacts/53d9b609-aab5-4c55-9c20-f05d24a9d64d/",
            "digest": "sha256:b0faf8bf3c2afb8c9f9beb648acc08a68c5d205710b2f2828cc21d59e051ca3d",
            "pulp_created": "2022-06-29T14:33:43.821909Z",
            "pulp_href": "/pulp/api/v3/content/container/blobs/a1d5528c-3077-4f88-b28a-4c769b6f9f06/"
        },
        {
            "artifact": "/pulp/api/v3/artifacts/5f2ec40e-e488-48ea-a14c-564737ca8c99/",
            "digest": "sha256:ac56bdc7f9934acede05653e9e01e73e961c31818b522c0732ad35350bb3a82b",
            "pulp_created": "2022-06-29T14:33:38.499151Z",
            "pulp_href": "/pulp/api/v3/content/container/blobs/79e6a9dc-472a-473a-8f6c-b32bde9fe198/"
        },
        {
            "artifact": "/pulp/api/v3/artifacts/bdfbfbb4-3a58-460c-b50f-814563ca30aa/",
            "digest": "sha256:7113491e908546d75cc713354580cc9e9766be7ee033c4fcb8ea66446e23a959",
            "pulp_created": "2022-06-29T14:33:38.497614Z",
            "pulp_href": "/pulp/api/v3/content/container/blobs/b4b6b051-9b21-4f42-b67c-f377e9eb7d0e/"
        },
        {
            "artifact": "/pulp/api/v3/artifacts/7db0a3e3-5225-49f2-b6f9-b2a51ec5fc0a/",
            "digest": "sha256:c4fa3965dac905ff648c2d8e707899e2f6f0a10b48e6b728e97f32cb6500e9ed",
            "pulp_created": "2022-06-29T14:33:32.766393Z",
            "pulp_href": "/pulp/api/v3/content/container/blobs/a4a0b057-1ffd-4fea-8e2d-5e42198cf0b7/"
        },
        {
            "artifact": "/pulp/api/v3/artifacts/3250fe62-3345-49ad-8e43-ff3366bbfa04/",
            "digest": "sha256:22f677655049d4c2e6cd9e49ca9ed20f34ac175ef0c82f5c5eabc79031c1c29a",
            "pulp_created": "2022-06-29T14:33:32.763686Z",
            "pulp_href": "/pulp/api/v3/content/container/blobs/3cfe1dc2-d8a3-4e5e-a8ce-1eb577736844/"
        }
    ]
}

Note schema_version and media_type stored in DB vs that it is an OCI image in reality.

Root cause

This is the offending line https://github.com/pulp/pulp_container/blob/main/pulp_container/app/tasks/sync_stages.py#L352 and https://github.com/pulp/pulp_container/blob/main/pulp_container/app/tasks/sync_stages.py#L132
Based on the specs mediaType is not a required field for the oci image https://github.com/opencontainers/image-spec/blob/main/manifest.md#image-manifest-property-descriptions

[fao89]

https://bugzilla.redhat.com/buglist.cgi?quicksearch=2102078

[ianballou]

I'd consider this relatively important for Katello. @ipanova do you anticipate other container images being affected by this? So far we're only seeing the ceph-zabbix one.

[ipanova]

@ianballou yes, oci container images that do not contain mediatype as part of manifest.json. We'll get this fixed and backport where it's needed.

[ipanova]

@ianballou to what branch do you want this fix backported?

[ianballou]

@ipanova 2.10 and 2.9 as well if possible.