500 during podman push of ML

[ipanova]

$ podman manifest create report-bug
c44970ca83a5687a2ae4ac0adb1876950e504d7bc4c3e7fa665ef748d4e14d90
(pulp) [vagrant@pulp3-source-fedora34 pulp_container]$ podman manifest push --all   report-bug  pulp3-source-fedora34.fluffy.example.com/foo:report-bug
Getting image list signatures
Copying 0 of 0 images in list
Writing manifest list to image destination
Storing list signatures
(pulp) [vagrant@pulp3-source-fedora34 pulp_container]$ podman manifest push --all   report-bug  pulp3-source-fedora34.fluffy.example.com/foo:report-bug
Getting image list signatures
Copying 0 of 0 images in list
Writing manifest list to image destination
Storing list signatures
(pulp) [vagrant@pulp3-source-fedora34 pulp_container]$ podman manifest push --all   report-bug  pulp3-source-fedora34.fluffy.example.com/foo:report-bug --format v2s2
Getting image list signatures
Copying 0 of 0 images in list
Writing manifest list to image destination
Storing list signatures
(pulp) [vagrant@pulp3-source-fedora34 pulp_container]$ podman manifest push --all   report-bug  pulp3-source-fedora34.fluffy.example.com/foo:report-bug --format oci
Getting image list signatures
Copying 0 of 0 images in list
Writing manifest list to image destination
Error: Uploading manifest list failed, attempted the following formats: application/vnd.oci.image.index.v1+json(uploading manifest report-bug to pulp3-source-fedora34.fluffy.example.com/foo: received unexpected HTTP status: 500 Internal Server Error)
(pulp) [vagrant@pulp3-source-fedora34 pulp_container]$ journalctl -r 
-- Journal begins at Sun 2022-05-15 16:28:22 UTC, ends at Mon 2022-06-20 12:35:42 UTC. --
Jun 20 12:35:42 pulp3-source-fedora34.fluffy.example.com audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pulpcore-resource-manager comm="systemd" exe="/usr/lib/systemd/systemd>
Jun 20 12:35:42 pulp3-source-fedora34.fluffy.example.com systemd[1]: Started Pulp Resource Manager.
Jun 20 12:35:42 pulp3-source-fedora34.fluffy.example.com systemd[1]: pulpcore-resource-manager.service: Consumed 3.322s CPU time.
Jun 20 12:35:42 pulp3-source-fedora34.fluffy.example.com systemd[1]: Stopped Pulp Resource Manager.
Jun 20 12:35:42 pulp3-source-fedora34.fluffy.example.com audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pulpcore-resource-manager comm="systemd" exe="/usr/lib/systemd/systemd">
Jun 20 12:35:42 pulp3-source-fedora34.fluffy.example.com audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pulpcore-resource-manager comm="systemd" exe="/usr/lib/systemd/systemd>
Jun 20 12:35:42 pulp3-source-fedora34.fluffy.example.com systemd[1]: pulpcore-resource-manager.service: Scheduled restart job, restart counter is at 409.
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com systemd[988]: podman-419168.scope: Deactivated successfully.
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]: pulp [0a6d3112eca24d6ca949569945abf4f6]:  - - [20/Jun/2022:12:35:40 +0000] "PUT /v2/foo/manifests/report-bug HTTP/1.0" 500 145 "-" "containers/5.16.0 (github.com/>
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]: AttributeError: 'NoneType' object has no attribute 'get'
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:     config_media_type = config_layer.get("mediaType")
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:   File "/home/vagrant/devel/pulp_container/pulp_container/app/registry_api.py", line 905, in put
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:     response = handler(request, *args, **kwargs)
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:     raise exc
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:     self.raise_uncaught_exception(exc)
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:     response = super().handle_exception(exc)
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:   File "/home/vagrant/devel/pulp_container/pulp_container/app/registry_api.py", line 252, in handle_exception
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:     response = self.handle_exception(exc)
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:     return self.dispatch(request, *args, **kwargs)
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:     return view_func(*args, **kwargs)
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:     response = wrapped_callback(request, *callback_args, **callback_kwargs)
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:     response = get_response(request)
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]: Traceback (most recent call last):
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]: pulp [0a6d3112eca24d6ca949569945abf4f6]: django.request:ERROR: Internal Server Error: /v2/foo/manifests/report-bug
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]: pulp [4d220dabcf514592b7619bcd7fcda1f6]:  - admin [20/Jun/2022:12:35:40 +0000] "GET /token/?account=admin&scope=repository%3Afoo%3Apull%2Cpush&service=pulp3-sourc>
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]: pulp [47aedb1205e045969bdd864017299e6b]:  - - [20/Jun/2022:12:35:40 +0000] "GET /v2/ HTTP/1.0" 401 106 "-" "containers/5.16.0 (github.com/containers/image)"
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com gunicorn[414290]: pulp [47aedb1205e045969bdd864017299e6b]: django.request:WARNING: Unauthorized: /v2/
Jun 20 12:35:40 pulp3-source-fedora34.fluffy.example.com systemd[988]: Started podman-419168.scope.

[lubosmj]

The issue is related to #854. We expect application/vnd.oci.image.index.v1+json to specified in the content_type header. With other types, we automatically fall back to standard manifest push workflow.

[lubosmj]

The issue can be resolved by using podman v4. In that version, podman uses the correct content_type.

[ipanova]

so this was a bug in podman? did you manage to find it in the podman release notes?

[ipanova]

regardsless, we should not raise 500 but handle it proprely
we rely on config being not empty

config_layer = content_data.get("config")
config_media_type = config_layer.get("mediaType")
config_digest = config_layer.get("digest")

If you add jsonchema validation then it would just need to raise 'manifest invalid' which is better then 500

[lubosmj]

so this was a bug in podman? did you manage to find it in the podman release notes?

Skimming through the changelogs in https://github.com/containers/image/releases and https://github.com/containers/podman/releases, I cannot find any changelogs specific to pushing empty manifest lists...

There are some bug fixes related to tagging manifest lists and pushing manifest lists, though. Maybe, this one is the change that had an impact on the reported behaviour: The Podman APIs for Manifest List and Network operations have been completely rewritten to address issues and inconsistencies in the previous APIs. Incompatible APIs should warn if they are used with an older Podman client..

[ipanova]

@lubosmj i see, let's just add jsonschema and raise manifest invalid, i believe that would address these 2 related issues