Pull-through caching may fail on a non-localfilesystem storage backend

[lubosmj]

Steps to reproduce:

1. Install Pulp with the S3 storage
2. Run podman pull to trigger the pull-through caching
3. Repeat 1-2 steps until you get the error

CI logs (pulp/S3):

Trying to pull [pulp.example.com:443/db63fe02-27e1-422e-95b4-6a6be3568c60/pulp/test-fixture-1:manifest_a.](http://pulp.example.com:443/db63fe02-27e1-422e-95b4-6a6be3568c60/pulp/test-fixture-1:manifest_a.)..
Getting image source signatures
Copying blob sha256:463b9bfafbaf88a2dee0aa77fd293a7cb91558af9664bc23d4ed00830463615d
Copying blob sha256:d7ab49ce0b755a907e914d148cc8c1f437f7688048aea18a15d965f60c5d0a43
time="2024-01-30T10:38:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: reading blob sha256:d7ab49ce0b755a907e914d148cc8c1f437f7688048aea18a15d965f60c5d0a43: fetching blob: blob unknown: Blob not found."
Getting image source signatures
Copying blob sha256:463b9bfafbaf88a2dee0aa77fd293a7cb91558af9664bc23d4ed00830463615d
Copying blob sha256:d7ab49ce0b755a907e914d148cc8c1f437f7688048aea18a15d965f60c5d0a43
Error: reading blob sha256:463b9bfafbaf88a2dee0aa77fd293a7cb91558af9664bc23d4ed00830463615d: fetching blob: received unexpected HTTP status: 500 Internal Server Error

Traceback:

Error: 1-30 10:38:53 +0000] [503] [ERROR] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib64/python3.8/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib64/python3.8/site-packages/aiohttp/web_app.py", line 543, in _handle
    resp = await handler(request)
  File "/usr/local/lib64/python3.8/site-packages/aiohttp/web_middlewares.py", line 114, in impl
    return await handler(request)
  File "/usr/local/lib/python3.8/site-packages/pulpcore/content/authentication.py", line 48, in authenticate
    return await handler(request)
  File "/usr/local/lib/python3.8/site-packages/pulpcore/content/instrumentation.py", line 230, in middleware
    resp = await handler(request)
  File "/usr/local/lib/python3.8/site-packages/pulpcore/cache/cache.py", line 352, in cached_function
    response = await self.make_entry(
  File "/usr/local/lib/python3.8/site-packages/pulpcore/cache/cache.py", line 395, in make_entry
    response = await handler(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pulp_container/app/registry.py", line 344, in get_by_digest
    return await Registry._dispatch(artifact, headers)
  File "/usr/local/lib/python3.8/site-packages/pulp_container/app/registry.py", line 95, in _dispatch
    raise NotImplementedError("Redirecting to this storage is not implemented.")dler(request)
  File "/usr/local/lib/python3.8/site-packages/pulpcore/cache/cache.py", line 352, in cached_function
    response = await self.make_entry(
  File "/usr/local/lib/python3.8/site-packages/pulpcore/cache/cache.py", line 395, in make_entry
    response = await handler(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pulp_container/app/registry.py", line 344, in get_by_digest
    return await Registry._dispatch(artifact, headers)
  File "/usr/local/lib/python3.8/site-packages/pulp_container/app/registry.py", line 95, in _dispatch
    raise NotImplementedError("Redirecting to this storage is not implemented.")

[lubosmj]

Intermittent failures in the CI were caused by

pulp_container/pulp_container/tests/functional/api/test_pull_through_cache.py

Line 100 in 637243b

def test_manifest_pull(delete_orphans_pre, pull_through_distribution, pull_and_verify):

.

[grzleadams]

I believe we're seeing this same behavior with NFS-backed storage, which is probably related to this. I'll be upgrading Docker on our K8s nodes this week so if we stop seeing 500's that would confirm it.

[lubosmj]

Same happens on Azure:

2024-04-21T03:24:10.1976888Z pulp [92796a8cd3d64c90af5f6434695afe5a]: django.request:WARNING: Unauthorized: /v2/
2024-04-21T03:24:10.1977146Z [2024-04-21 03:18:30 +0000] [524] [ERROR] Error handling request
2024-04-21T03:24:10.1977265Z Traceback (most recent call last):
2024-04-21T03:24:10.1977855Z   File "/usr/local/lib64/python3.8/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
2024-04-21T03:24:10.1977998Z     resp = await request_handler(request)
2024-04-21T03:24:10.1978690Z   File "/usr/local/lib64/python3.8/site-packages/aiohttp/web_app.py", line 543, in _handle
2024-04-21T03:24:10.1978813Z     resp = await handler(request)
2024-04-21T03:24:10.1979263Z   File "/usr/local/lib64/python3.8/site-packages/aiohttp/web_middlewares.py", line 114, in impl
2024-04-21T03:24:10.1979368Z     return await handler(request)
2024-04-21T03:24:10.1979903Z   File "/usr/local/lib/python3.8/site-packages/pulpcore/content/authentication.py", line 48, in authenticate
2024-04-21T03:24:10.1980009Z     return await handler(request)
2024-04-21T03:24:10.1980538Z   File "/usr/local/lib/python3.8/site-packages/pulpcore/content/instrumentation.py", line 230, in middleware
2024-04-21T03:24:10.1980643Z     resp = await handler(request)
2024-04-21T03:24:10.1981114Z   File "/usr/local/lib/python3.8/site-packages/pulpcore/cache/cache.py", line 352, in cached_function
2024-04-21T03:24:10.1981231Z     response = await self.make_entry(
2024-04-21T03:24:10.1981664Z   File "/usr/local/lib/python3.8/site-packages/pulpcore/cache/cache.py", line 395, in make_entry
2024-04-21T03:24:10.1981808Z     response = await handler(*args, **kwargs)
2024-04-21T03:24:10.1982286Z   File "/usr/local/lib/python3.8/site-packages/pulp_container/app/registry.py", line 309, in get_by_digest
2024-04-21T03:24:10.1982461Z     return await Registry._dispatch(artifact, headers)
2024-04-21T03:24:10.1982928Z   File "/usr/local/lib/python3.8/site-packages/pulp_container/app/registry.py", line 94, in _dispatch
2024-04-21T03:24:10.1983244Z     raise NotImplementedError("Redirecting to this storage is not implemented.")
2024-04-21T03:24:10.1983490Z NotImplementedError: Redirecting to this storage is not implemented.
2024-04-21T03:24:10.1984637Z pulp [None]: azure.core.pipeline.policies.http_logging_policy:INFO: Request URL: 'https://ci-azurite:10000/devstoreaccount1/pulp-test/pulp3/artifact/d7/ab49ce0b755a907e914d148cc8c1f437f7688048aea18a15d965f60c5d0a43?timeout=REDACTED'

[lubosmj]

The issue might lie here:

pulp_container/pulp_container/app/registry.py

Line 91 in 21e7564

elif not settings.REDIRECT_TO_OBJECT_STORAGE:

The highlighted code part is evaluated to False by default and we step into the else branch with raise NotImplementedError("Redirecting to this storage is not implemented.").

We should be able to handle a scenario where an artifact is stored on the object storage but we are in the process of returning the content initialized during the pull through caching.

registry_api.py -> registry.py -> get_by_tag -> Registry._dispatch -> return the content from the object storage