Upgrade from container 2.16 to 2.19 causes 404 on manifests

[ianballou]

Version

      "versions": {
        "deb": "3.2.0",
        "rpm": "3.25.3",
        "core": "3.49.4",
        "file": "3.49.4",
        "ostree": "2.3.0",
        "python": "3.11.1",
        "ansible": "0.21.3",
        "certguard": "3.49.4",
        "container": "2.19.3"
      }

Describe the bug
On a Katello smart proxy, after upgrading from container 2.16 to 2.19, the manifests endpoint now gives 404s.

Before:

Apr 26 15:05:48 centos8-proxy-devel pulpcore-api[1829]: pulp [1cfff6d9f16c46a5bcd7182726da4547]:  - - [26/Apr/2024:15:05:48 +0000] "GET /v2/default_organization-precipitation-container_view-buttermilk_pancakes-centos-bootc_fedora-bootc/manifests/latest HTTP/1.1" 302 0 "-" "Ruby"
Apr 26 15:05:48 centos8-proxy-devel pulpcore-content[1830]: [26/Apr/2024:15:05:48 +0000] "GET /pulp/container/default_organization-precipitation-container_view-buttermilk_pancakes-centos-bootc_fedora-bootc/manifests/latest?expires=1714147548&validate_token=f9506d66d3afd0a0c333d26316596f6defcb21b595e3d4d0ed7dec166e7447b0:5f3f08399424852ca3ecd384479b8bd0b8dd6d13b24f6c2047edf2fdb1668cfe HTTP/1.1" 200 428 "https://centos8-proxy-devel.manicotto.example.com/v2/default_organization-precipitation-container_view-buttermilk_pancakes-centos-bootc_fedora-bootc/manifests/latest" "containers/5.29.2 (github.com/containers/image)"

After:

Apr 26 16:13:06 centos8-proxy-devel pulpcore-api[59929]: pulp [26d972177daa463ba6da256a25c69e5f]: django.request:WARNING: Not Found: /v2/default_organization-precipitation-container_view-buttermilk_pancakes-centos-bootc_fedora-bootc/manifests/latest
Apr 26 16:13:06 centos8-proxy-devel pulpcore-api[59929]: pulp [26d972177daa463ba6da256a25c69e5f]:  - - [26/Apr/2024:16:13:06 +0000] "GET /v2/default_organization-precipitation-container_view-buttermilk_pancakes-centos-bootc_fedora-bootc/manifests/latest HTTP/1.1" 404 104 "-" "Ruby"

I don't think it's related to our apache config, it doesn't appear to have changed between upgrades.

To Reproduce
Steps to reproduce the behavior:

Create a Katello smart proxy and upgrade it to Pulpcore 3.49 (if not already on it).
Sync some content to it and try to pull after logging in.

I can help create a reproducer environment if necessary.

Here's an example distribution:

   {
    "pulp_href": "/pulp/api/v3/distributions/container/container/018f1b83-8bfc-7000-8c31-f73e46c6ec5c/",
    "content_guard": "/pulp/api/v3/contentguards/core/content_redirect/018f1b82-5828-7236-a171-0983f85dbc44/",
    "hidden": false,
    "name": "1-container_view-precipitation-b2950a65-8dd1-4009-b869-55a904a2ea1a",
    "base_path": "default_organization-precipitation-container_view-buttermilk_pancakes-centos-bootc_fedora-bootc",
    "pulp_last_updated": "2024-04-26T17:45:56.990539Z",
    "repository": null,
    "pulp_created": "2024-04-26T17:45:56.990519Z",
    "pulp_labels": {},
    "repository_version": "/pulp/api/v3/repositories/container/container/018f1b82-355f-768a-8a3d-39e9753baa2a/versions/1/",
    "registry_path": "pulpcore-api/default_organization-precipitation-container_view-buttermilk_pancakes-centos-bootc_fedora-bootc",
    "remote": null,
    "namespace": "/pulp/api/v3/pulp_container/namespaces/018f1b82-5859-7435-9848-9af653457ba4/",
    "private": false,
    "description": null
  }

Expected behavior
Pull works

Additional context

The only thing that's making me a little suspicious is:

 "registry_path": "pulpcore-api/default_organization-precipitation-container_view-buttermilk_pancakes-centos-bootc_fedora-bootc",

On a normal Katello machine, instead of pulpcore-api it has the machine's FQDN. And, pulling does work on normal Katello machines.

However, this registry path is the same when using pulp-container 2.16, and I can pull just fine.

[ianballou]

Here's an example of what happens when you pull:

[vagrant@centos8-proxy-devel ~]$ podman pull centos8-proxy-devel.manicotto.example.com/default_organization-precipitation-container_view-buttermilk_pancakes-centos-bootc_fedora-bootc
Trying to pull centos8-proxy-devel.manicotto.example.com/default_organization-precipitation-container_view-buttermilk_pancakes-centos-bootc_fedora-bootc:latest...
Error: initializing source docker://centos8-proxy-devel.manicotto.example.com/default_organization-precipitation-container_view-buttermilk_pancakes-centos-bootc_fedora-bootc:latest: reading manifest latest in centos8-proxy-devel.manicotto.example.com/default_organization-precipitation-container_view-buttermilk_pancakes-centos-bootc_fedora-bootc: StatusCode: 404, "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">..."

With how there is HTML coming back with the 404, it feels like the endpoint isn't being recognized at all by Pulp's content application.

I verified that doing nothing but upgrading Pulp triggers the issue, so we can rule out other configuration changes in the smart proxy being the cause.

[ianballou]

Reproducing this a second time, here are the logs in /var/log/messages:

Apr 26 18:23:32 centos8-proxy-devel pulpcore-api[5124]: pulp [a9d890ba91324f069d3d990671e46c11]: django.request:WARNING: Not Found: /v2/default_organization-precipitation-container_view-buttermilk_pancakes-centos-bootc_fedora-bootc/manifests/latest
Apr 26 18:23:32 centos8-proxy-devel pulpcore-api[5124]: pulp [a9d890ba91324f069d3d990671e46c11]:  - - [26/Apr/2024:18:23:32 +0000] "GET /v2/default_organization-precipitation-container_view-buttermilk_pancakes-centos-bootc_fedora-bootc/manifests/latest HTTP/1.1" 404 104 "-" "Ruby"

[lubosmj]

I am requesting the access to the affected smart proxy instance. This is very suspicious.

[lubosmj]

Can you share if there is a difference between apache configurations? Or, any configuration differences between the working and non-working instances?

[ianballou]

Can you share if there is a difference between apache configurations? Or, any configuration differences between the working and non-working instances?

I confirmed that the configuration stays completely the same between working / non working environments. All I did was update Pulpcore + every plugin and restart the services.

[ipanova]

RE: registry_path, it is wrong that it has pulpcore-api instead of normal host, we should look why this changed.

[ianballou]

This does seem to work fine with older pulp container 2.16, but perhaps it wasn't supposed to work before?

…

On Tue, Apr 30, 2024, 07:28 Ina Panova ***@***.***> wrote: RE: registry_path, it is wrong that it has pulpcore-api instead of normal host, we should look why this changed. — Reply to this email directly, view it on GitHub <#1604 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADQ4OFTL3HNR7AC6VZFIHJDY75563AVCNFSM6AAAAABG3G4UJGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBVGA2DIMJQGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ianballou]

Likely to be caused by #1509

As such, I think it's on our side to fix. Closing this out.