Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fetching a non-existing blob object results in an internal server error #513

Closed
pulpbot opened this issue Dec 15, 2021 · 1 comment · Fixed by #549
Closed

Fetching a non-existing blob object results in an internal server error #513

pulpbot opened this issue Dec 15, 2021 · 1 comment · Fixed by #549
Assignees
@ipanova
Labels
Issue
Milestone
2.11.0

Comments

@pulpbot
Copy link
Member

pulpbot commented Dec 15, 2021

Author: @lubosmj (lmjachky)

Redmine Issue: 9599, https://pulp.plan.io/issues/9599

The error is triggered only after removing the reference to a repository within a distribution. Steps to reproduce the error:

  1. Disable token authentication and sync a remote repository.
cat dockersync.bash 
http POST http://localhost:24817/pulp/api/v3/repositories/container/container/ name=foo

export REPO_HREF=$(http :24817/pulp/api/v3/repositories/container/container/ | jq -r '.results[] | select(.name == "foo") | .pulp_href')

http POST http://localhost:24817/pulp/api/v3/remotes/container/container/ name='lubosmj/mirror' upstream_name='lubosmj/mirror' url='https://registry.hub.docker.com'  policy='immediate' password="password" username="lubosmj"

export REMOTE_HREF=$(http :24817/pulp/api/v3/remotes/container/container/ | jq -r '.results[] | select(.name == "lubosmj/mirror") | .pulp_href')

pulp container repository sync --name foo --remote lubosmj/mirror

pulp container distribution create --name name --base-path base-path --repository foo

bash dockersync.bash
HTTP/1.1 201 Created
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 465
Content-Type: application/json
Correlation-ID: 01981a6e63f74f99bec6910b2a5b9d86
Date: Thu, 02 Dec 2021 14:29:40 GMT
Location: /pulp/api/v3/repositories/container/container/17dc4c9e-53e1-4ae5-82bc-c4d0632b9600/
Referrer-Policy: same-origin
Server: gunicorn
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "description": null,
    "latest_version_href": "/pulp/api/v3/repositories/container/container/17dc4c9e-53e1-4ae5-82bc-c4d0632b9600/versions/0/",
    "name": "foo",
    "pulp_created": "2021-12-02T14:29:39.928443Z",
    "pulp_href": "/pulp/api/v3/repositories/container/container/17dc4c9e-53e1-4ae5-82bc-c4d0632b9600/",
    "pulp_labels": {},
    "remote": null,
    "retain_repo_versions": null,
    "versions_href": "/pulp/api/v3/repositories/container/container/17dc4c9e-53e1-4ae5-82bc-c4d0632b9600/versions/"
}


HTTP/1.1 201 Created
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 614
Content-Type: application/json
Correlation-ID: 398410f0419246099448201306bb0842
Date: Thu, 02 Dec 2021 14:29:41 GMT
Location: /pulp/api/v3/remotes/container/container/9fc79087-758a-47c6-9832-04fd25191a62/
Referrer-Policy: same-origin
Server: gunicorn
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "ca_cert": null,
    "client_cert": null,
    "connect_timeout": null,
    "download_concurrency": null,
    "exclude_tags": null,
    "headers": null,
    "include_tags": null,
    "max_retries": null,
    "name": "lubosmj/mirror",
    "policy": "immediate",
    "proxy_url": null,
    "pulp_created": "2021-12-02T14:29:41.645134Z",
    "pulp_href": "/pulp/api/v3/remotes/container/container/9fc79087-758a-47c6-9832-04fd25191a62/",
    "pulp_labels": {},
    "pulp_last_updated": "2021-12-02T14:29:41.645148Z",
    "rate_limit": null,
    "sock_connect_timeout": null,
    "sock_read_timeout": null,
    "tls_validation": true,
    "total_timeout": null,
    "upstream_name": "lubosmj/mirror",
    "url": "https://registry.hub.docker.com"
}


Started background task /pulp/api/v3/tasks/7f9f6d2a-a51d-48b5-a919-61d8674f7d45/
...Done.
Started background task /pulp/api/v3/tasks/df191dd2-0a5f-4cd9-a5d8-0d2ba4c8a038/
.Done.
{
  "pulp_href": "/pulp/api/v3/distributions/container/container/34cfda8e-b5b8-4aa3-a656-ac5adba3dd76/",
  "repository": "/pulp/api/v3/repositories/container/container/17dc4c9e-53e1-4ae5-82bc-c4d0632b9600/",
  "content_guard": "/pulp/api/v3/contentguards/container/content_redirect/f6615734-fa37-463e-942b-05310b246ae2/",
  "base_path": "base-path",
  "name": "name",
  "pulp_labels": {},
  "pulp_created": "2021-12-02T14:29:48.358690Z",
  "repository_version": null,
  "registry_path": "pulp3-source-fedora34.localhost.example.com/base-path",
  "namespace": "/pulp/api/v3/pulp_container/namespaces/3cbed472-7734-43b7-8a31-6fbad955d2da/",
  "private": false,
  "description": null
}
  1. Fetch one of the existing blobs.
http :24817/pulp/api/v3/content/container/blobs/?limit=1
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 480
Content-Type: application/json
Correlation-ID: 23eedc8c421d4f858e99d8b1bc7cd025
Date: Thu, 02 Dec 2021 14:30:53 GMT
Referrer-Policy: same-origin
Server: gunicorn
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "count": 6,
    "next": "http://localhost:24817/pulp/api/v3/content/container/blobs/?limit=1&offset=1",
    "previous": null,
    "results": [
        {
            "artifact": "/pulp/api/v3/artifacts/36d71c4f-f0ca-4a25-9bc8-2d7818d21281/",
            "digest": "sha256:f7776189e4fa6a19c57087c0d1bbd413dbf6bc4c25547b2be2377a3b373b6676",
            "media_type": "application/vnd.docker.container.image.v1+json",
            "pulp_created": "2021-12-02T14:29:46.539410Z",
            "pulp_href": "/pulp/api/v3/content/container/blobs/2bc41c2a-eca4-44c9-aadd-9ffac91756e2/"
        }
    ]
}


http :24817/v2/base-path/blobs/sha256:f7776189e4fa6a19c57087c0d1bbd413dbf6bc4c25547b2be2377a3b373b6676 'Accept: application/vnd.docker.container.image.v1+json'
HTTP/1.1 302 Found
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 0
Content-Type: text/html; charset=utf-8
Correlation-ID: 461309c2d3474cdf886d0d23d19bc29a
Date: Thu, 02 Dec 2021 14:32:00 GMT
Docker-Distribution-Api-Version: registry/2.0
Location: http://pulp3-source-fedora34.localhost.example.com/pulp/container/base-path/blobs/sha256:f7776189e4fa6a19c57087c0d1bbd413dbf6bc4c25547b2be2377a3b373b6676?validate_token=d1ad8c80f9bbb71c2e682b658ec9097c9fd967a7f168c27ef998d1c43f3027a4:948980f81a848f4308098e51acbfb346d01bb7c71ee3ba1b43e7e07d9789afdf
Referrer-Policy: same-origin
Server: gunicorn
X-Content-Type-Options: nosniff
X-Frame-Options: DENY


http http://pulp3-source-fedora34.localhost.example.com/pulp/container/base-path/blobs/sha256:f7776189e4fa6a19c57087c0d1bbd413dbf6bc4c25547b2be2377a3b373b6676?validate_token=d1ad8c80f9bbb71c2e682b658ec9097c9fd967a7f168c27ef998d1c43f3027a4:948980f81a848f4308098e51acbfb346d01bb7c71ee3ba1b43e7e07d9789afdf 'Accept: application/vnd.docker.container.image.v1+json'
HTTP/1.1 200 OK
Accept-Ranges: bytes
Connection: keep-alive
Content-Disposition: attachment; filename=776189e4fa6a19c57087c0d1bbd413dbf6bc4c25547b2be2377a3b373b6676
Content-Length: 1820
Content-Type: application/octet-stream
Date: Thu, 02 Dec 2021 14:32:35 GMT
Docker-Content-Digest: sha256:f7776189e4fa6a19c57087c0d1bbd413dbf6bc4c25547b2be2377a3b373b6676
Docker-Distribution-API-Version: registry/2.0
Etag: "16bcf67e171d7042-71c"
Last-Modified: Thu, 02 Dec 2021 14:29:46 GMT
Server: nginx/1.20.1

{"architecture":"amd64","config":{"Hostname":"d0e5d053b661","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":null,"ArgsEscaped":true,"Image":"sha256:e004b88433f9737b37e2d66721effa20ff1263082c8152882a1976c72a494a34","Volumes":null,"WorkingDir":"","Entrypoint":["mysql"],"OnBuild":[],"Labels":{}},"container":"82bea353549a807a6bbcd4d9dfcd490c194691bf30c99b75abed0693c9d27072","container_config":{"Hostname":"d0e5d053b661","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["/bin/sh","-c","#(nop) ","ENTRYPOINT [\"mysql\"]"],"ArgsEscaped":true,"Image":"sha256:e004b88433f9737b37e2d66721effa20ff1263082c8152882a1976c72a494a34","Volumes":null,"WorkingDir":"","Entrypoint":["mysql"],"OnBuild":[],"Labels":{}},"created":"2020-01-27T14:35:45.362414698Z","docker_version":"1.13.1","history":[{"created":"2019-03-07T22:19:53.313789681Z","created_by":"/bin/sh -c #(nop) ADD file:aa17928040e31624cad9c7ed19ac277c5402c4b9ba39f834250affca40c4046e in / "},{"created":"2019-03-07T22:19:53.447205048Z","created_by":"/bin/sh -c #(nop)  CMD [\"/bin/sh\"]","empty_layer":true},{"created":"2020-01-27T14:35:45.032879266Z","created_by":"/bin/sh -c apk add --no-cache mysql-client"},{"created":"2020-01-27T14:35:45.362414698Z","created_by":"/bin/sh -c #(nop)  ENTRYPOINT [\"mysql\"]","empty_layer":true}],"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:3fc64803ca2de7279269048fe2b8b3c73d4536448c87c32375b2639ac168a48b","sha256:f8f387438ce6199d3c8dcd4e4daaaebea2eacb67c807090451805aa550fcc855"]}}
  1. Remove the repository from the distribution.
http PATCH :24817/pulp/api/v3/distributions/container/container/34cfda8e-b5b8-4aa3-a656-ac5adba3dd76/ repository=""
HTTP/1.1 202 Accepted
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, PUT, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Correlation-ID: fe0607bc5c0143ba8935b38dc52e5410
Date: Thu, 02 Dec 2021 14:33:00 GMT
Referrer-Policy: same-origin
Server: gunicorn
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "task": "/pulp/api/v3/tasks/905ae1c1-ac14-42af-8dcb-7350323fa5e5/"
}
  1. Try to fetch the same blob once again.
http http://pulp3-source-fedora34.localhost.example.com/pulp/container/base-path/blobs/sha256:f7776189e4fa6a19c57087c0d1bbd413dbf6bc4c25547b2be2377a3b373b6676?validate_token=d1ad8c80f9bbb71c2e682b658ec9097c9fd967a7f168c27ef998d1c43f3027a4:948980f81a848f4308098e51acbfb346d01bb7c71ee3ba1b43e7e07d9789afdf 'Accept: application/vnd.docker.container.image.v1+json'

HTTP/1.1 500 Internal Server Error
Connection: keep-alive
Content-Length: 55
Content-Type: text/plain; charset=utf-8
Date: Thu, 02 Dec 2021 14:33:16 GMT
Server: nginx/1.20.1

500 Internal Server Error

Server got itself in trouble

Traceback:

  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/home/vagrant/devel/pulpcore/pulpcore/content/authentication.py", line 41, in authenticate
    return await handler(request)
  File "/home/vagrant/devel/pulp_container/pulp_container/app/registry.py", line 220, in get_by_digest
    content__in=await sync_to_async(repository_version.get_content)(),
AttributeError: 'NoneType' object has no attribute 'get_content'
@pulpbot pulpbot added this to the 2.11.0 milestone Dec 15, 2021
@pulpbot
Copy link
Member Author

pulpbot commented Dec 15, 2021

From: @ipanova (ipanova@redhat.com)
Date: 2021-12-07T23:01:13Z

we need to check for None https://github.com/pulp/pulp_container/blob/main/pulp_container/app/models.py#L481

ipanova added a commit to ipanova/pulp_container that referenced this issue Jan 19, 2022
@ipanova
Fixed error for content retrieval from distribution when repo is remo…
befb50b 
…ved.

closes pulp#513
@ipanova ipanova self-assigned this Jan 19, 2022
@ipanova ipanova mentioned this issue Jan 19, 2022
Merged
ipanova added a commit to ipanova/pulp_container that referenced this issue Jan 21, 2022
@ipanova
Fixed content retrieval from distribution when repo is removed.
c5770b2 
closes pulp#513
ipanova added a commit that referenced this issue Jan 21, 2022
@ipanova
Fixed content retrieval from distribution when repo is removed.
a9f61cc 
closes #513
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ipanova ipanova

Labels
Issue
Projects
None yet
Milestone
2.11.0
Development

Successfully merging a pull request may close this issue.

Fixed error for content retrieval from distribution when repo is remo… ipanova/pulp_container
2 participants
@ipanova @pulpbot