Podman push image to redhat quay with sigstore was failed caused by send malformed manifest to quay

[LiZhang19817]

Hi Guys,

When use podman 4.2.1 to push image to Redhat Quay 3.8.0, hit 500 error code, based on the log error message , seems like podman send malformed json to quay, pls review this issue and give suggestions.

[root@ip-10-0-1-76 fedora]# podman push quayregistry-quay-quay-enterprise-13240.apps.quaytest-13240.qe.azure.devcluster.openshift.com/quay/demo --tls-verify=false --sign-by-sigstore-private-key=./cosign.key
Key Passphrase: 
Getting image source signatures
Copying blob 288cf3a46e32 done  
Copying blob 75ba02937496 done  
Copying blob 0c7daf9a72c8 done  
Copying blob 955c9335e041 done  
Copying blob 8e079fee2186 done  
Copying blob 186da837555d done  
Copying blob d172a9e6f9e6 done  
Copying blob cf399be408ea done  
Copying blob 793b971ccb99 done  
Copying config da84e66c3a done  
Writing manifest to image destination
Signing manifest using a sigstore signature
Storing signatures
Error: writing signatures: uploading manifest sha256-2353c13421e07e3d3dd1bb181cf0b7ad5e6dce3e1bb363c33f48d12e0a0ada49.sig to quayregistry-quay-quay-enterprise-13240.apps.quaytest-13240.qe.azure.devcluster.openshift.com/quay/demo: received unexpected HTTP status: 500 Internal Server Error 

gunicorn-registry stdout | 2022-10-11 03:58:52,461 [214] [ERROR] [gunicorn.error] Error handling request /v2/quay/demo/manifests/sha256-2353c13421e07e3d3dd1bb181cf0b7ad5e6dce3e1bb363c33f48d12e0a0ada49.sig
gunicorn-registry stdout | Traceback (most recent call last):
gunicorn-registry stdout |   File "/quay-registry/image/oci/config.py", line 209, in __init__
gunicorn-registry stdout |     validate_schema(self._parsed, OCIConfig.METASCHEMA)
gunicorn-registry stdout |   File "/usr/local/lib/python3.9/site-packages/jsonschema/validators.py", line 934, in validate
gunicorn-registry stdout |     raise error
gunicorn-registry stdout | jsonschema.exceptions.ValidationError: '' is not one of ['layers']
gunicorn-registry stdout | Failed validating 'enum' in schema['properties']['rootfs']['properties']['type']:
gunicorn-registry stdout |     {'description': 'MUST be set to layers.',
gunicorn-registry stdout |      'enum': ['layers'],
gunicorn-registry stdout |      'type': 'string'}
gunicorn-registry stdout | On instance['rootfs']['type']:
gunicorn-registry stdout |     ''
gunicorn-registry stdout | During handling of the above exception, another exception occurred:
gunicorn-registry stdout | Traceback (most recent call last):
gunicorn-registry stdout |   File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 55, in handle
gunicorn-registry stdout |     self.handle_request(listener_name, req, client, addr)
gunicorn-registry stdout |   File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/ggevent.py", line 127, in handle_request
gunicorn-registry stdout |     super().handle_request(listener_name, req, sock, addr)
gunicorn-registry stdout |   File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 108, in handle_request
gunicorn-registry stdout |     respiter = self.wsgi(environ, resp.start_response)
gunicorn-registry stdout |   File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2463, in __call__
gunicorn-registry stdout |     return self.wsgi_app(environ, start_response)
gunicorn-registry stdout |   File "/usr/local/lib/python3.9/site-packages/werkzeug/middleware/proxy_fix.py", line 169, in __call__
gunicorn-registry stdout |     return self.app(environ, start_response)
gunicorn-registry stdout |   File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2449, in wsgi_app
gunicorn-registry stdout |     response = self.handle_exception(e)
gunicorn-registry stdout |   File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1866, in handle_exception
gunicorn-registry stdout |     reraise(exc_type, exc_value, tb)
gunicorn-registry stdout |   File "/usr/local/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
gunicorn-registry stdout |     raise value
gunicorn-registry stdout |   File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2446, in wsgi_app
gunicorn-registry stdout |     response = self.full_dispatch_request()
gunicorn-registry stdout |   File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1951, in full_dispatch_request
gunicorn-registry stdout |     rv = self.handle_user_exception(e)
gunicorn-registry stdout |   File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1820, in handle_user_exceptiongunicorn-registry stdout |     'x-ms-copy-source': 'REDACTED' 

[vrothberg]

Thanks for reaching out. Which version of Quay are you running? Does it work using cosign?

[vrothberg]

Cc: @mtrmac

[LiZhang19817]

[root@ip-10-0-1-169 fedora]# podman -v
podman version 4.2.1

[LiZhang19817]

Thanks for reaching out. Which version of Quay are you running? Does it work using cosign?

Yes, Quay Version is 3.7.* and 3.8.* and Cosign works well with Quay

[mtrmac]

Thanks for your report. containers/image#1684 seems to fix that, at least against the public quay.io instance, although I’d very much appreciate independent validation.

---

For Podman maintainers: This containers/image#1684 and independent containers/image#1683 are relevant for sigstore interoperability with registries. Would the be worth backporting to the 4.2 branch (and do we plan more releases from that branch at all)?

(It’s … interesting? that in the >2 months of existence of the code, both of these problems were only noticed this week.)

[LiZhang19817]

thx, which podman version will have this fix?

[mtrmac]

Some future one.

[vrothberg]

Thanks for your report. containers/image#1684 seems to fix that, at least against the public quay.io instance, although I’d very much appreciate independent validation.

For Podman maintainers: This containers/image#1684 and independent containers/image#1683 are relevant for sigstore interoperability with registries. Would the be worth backporting to the 4.2 branch (and do we plan more releases from that branch at all)?

(It’s … interesting? that in the >2 months of existence of the code, both of these problems were only noticed this week.)

@rhatdan WDYT?

I think it's probably worth backporting to RHEL.

[rhatdan]

It will be in podman 4.4 4.3 is just about to release. I think this would be worth back porting.

[TomSweeneyRedHat]

@vrothberg has this been backported to skopeo and Podman for RHEL 8.7/9.1 yet? We need to do so as soon as possible. Holler if you need version pointers.

[mtrmac]

@TomSweeneyRedHat These are c/image changes. Should that happen on a c/image branch that is then vendor-danced, or is this a Podman-only backport?

[vrothberg]

@vrothberg has this been backported to skopeo and Podman for RHEL 8.7/9.1 yet? We need to do so as soon as possible. Holler if you need version pointers.

@TomSweeneyRedHat, you said you'll open BZs but I haven't seen them yet. Did you open some?

[LiZhang19817]

@vrothberg @TomSweeneyRedHat want to know which podman version having this fix, so Quay team can double check with latest Quay 3.8.

[mtrmac]

@LiZhang19817 containers/image#1684 was included in Podman 4.3.1, 4.4.0, and all later versions. Search for

> ociConfig.RootFS.Type = "layers"

to confirm or to check for pre-release / branched versions.