Enable filtering by content #2992

lubosmj · 2022-07-20T14:01:48Z

closes #2952

Required PR: pulp/pulpcore#2992 [noissue]

gerrod3 · 2022-07-22T14:42:28Z

pulpcore/app/viewsets/custom_filters.py

+        for dist in qs.exclude(repository=None).prefetch_related("repository__versions"):
+            versions_distributions[dist.repository.latest_version().pk].append(dist.pk)


I think this doesn't handle the auto-distribute feature properly. For plugins that serve directly from a repository it'll work fine, but for plugins that use a publication it might be incorrect. When a repository is set on the distribution and that plugin uses publications, the distribution will serve the latest and highest repo-version publication. So technically you could have a repository with 10 versions, but only one publication on version 5 and that would be the version served. There is also a constant on the distribution class that is used to determine if serving from the publication is appropriate.

Aha, I was not aware of that. Good to know.

Required PR: pulp/pulpcore#2992 [noissue]

ipanova · 2022-07-28T11:38:13Z

pulpcore/app/viewsets/custom_filters.py

+
+
+class DistributionWithContentFilter(Filter):
+    """A Filter class enabling filtering by content units exposed by distributions."""


s/exposed/served

ipanova · 2022-07-28T11:38:54Z

pulpcore/app/viewsets/custom_filters.py

+    def __init__(self, *args, **kwargs):
+        """Initialize a help message for the filter."""
+        kwargs.setdefault(
+            "help_text", _("Filter distributions based on the content present inside them")


based on the content served by them

bmbouter · 2022-08-09T13:38:57Z

@newswangerd please let us know what you think of this PR. Thank you!

newswangerd · 2022-08-11T19:58:42Z

how do you use this? I assume it's GET /pulp/api/v3/distributions/?with_content=CONTENT_HREF, but that's returning an empty list for me.

Required PR: pulp/pulpcore#2992 [noissue]

lubosmj · 2022-08-12T19:11:17Z

I have just realized that the old approach did not take into account distributions that do not serve content from publications. @newswangerd, would you mind testing your workflow once again?

Now, it should work:

(pulp) [vagrant@pulp3-source-fedora35 backup]$ http :24817/pulp/api/v3/distributions/?with_content=/pulp/api/v3/content/container/tags/39877fac-7f5b-48db-9bc1-7a80c6828174/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 52
Content-Type: application/json
Correlation-ID: 4b6e37dbf345415695f0d2861621f2ee
Date: Fri, 12 Aug 2022 19:10:21 GMT
Referrer-Policy: same-origin
Server: gunicorn
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "count": 0,
    "next": null,
    "previous": null,
    "results": []
}


(pulp) [vagrant@pulp3-source-fedora35 backup]$ http :24817/pulp/api/v3/distributions/?with_content=/pulp/api/v3/content/container/tags/4186495f-4fa9-4d7d-bab5-7fc379df7a03/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 563
Content-Type: application/json
Correlation-ID: 72250cf544a346d683c2680e5741514c
Date: Fri, 12 Aug 2022 19:10:32 GMT
Referrer-Policy: same-origin
Server: gunicorn
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "base_path": "repo-version",
            "base_url": "https://pulp3-source-fedora35.localhost.example.com/pulp/content/repo-version/",
            "content_guard": "/pulp/api/v3/contentguards/core/content_redirect/0470f5f0-6b9b-46c2-a4db-2205859c11b7/",
            "name": "rhel7-repository",
            "pulp_created": "2022-08-12T19:08:59.458302Z",
            "pulp_href": "/pulp/api/v3/distributions/container/container/1df93922-416e-44f3-a835-6d9276bb8991/",
            "pulp_labels": {},
            "repository": "/pulp/api/v3/repositories/container/container/d607ffc3-e695-4543-80e0-7386637574db/"
        }
    ]
}

newswangerd

Looks like it's working! Are you going to write functional tests for this?

lubosmj · 2022-08-15T14:13:56Z

Looks like it's working! Are you going to write functional tests for this?

Thanks! 🪂 There are functional tests written for this feature in pulp_file (pulp/pulp_file#778).

I am still worried about the performance implications of doing so many queries under the hood. Would anyone from the @pulp/core team take a look at this PR?

bmbouter · 2022-08-15T14:44:41Z

I looked at this PR. Yes this could probably be optimized more, but I generally recommend to write it simple first and then come back later after the runtime is demonstrated to be an issue in practice for our users.

Required PR: pulp/pulpcore#2992 [noissue]

bmbouter

This looks good to me. We can optimize later. Thanks!

ipanova · 2022-08-18T11:38:46Z

@lubosmj what i am doing wrong?

$ podman push  pulp3-source-fedora36.puffy.example.com/ipanova/busybox 
Getting image source signatures
Copying blob 7ad00cd55506 done  
Copying config 62aedd01bd done  
Writing manifest to image destination
Storing signatures
(pulp) [vagrant@pulp3-source-fedora36 docs]$ http "https://PULP3-SOURCE-FEDORA36.puffy.example.com/pulp/api/v3/distributions/" --auth admin:password
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, POST, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 569
Content-Type: application/json
Correlation-ID: d67886553ab746e9b72cfb896170c24b
Date: Thu, 18 Aug 2022 11:35:05 GMT
Referrer-Policy: same-origin
Server: nginx
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "base_path": "ipanova/busybox",
            "base_url": "https://pulp3-source-fedora36.puffy.example.com/pulp/content/ipanova/busybox/",
            "content_guard": "/pulp/api/v3/contentguards/core/content_redirect/942a79d4-c1a0-4d0b-a81b-2446ec0a01c3/",
            "name": "ipanova/busybox",
            "pulp_created": "2022-08-18T11:34:48.284401Z",
            "pulp_href": "/pulp/api/v3/distributions/container/container/92fb9e64-a760-4be5-b2c9-6065db90a526/",
            "pulp_labels": {},
            "repository": "/pulp/api/v3/repositories/container/container-push/db20312f-4796-4b8f-81ec-189997fca558/"
        }
    ]
}


(pulp) [vagrant@pulp3-source-fedora36 docs]$ http "https://PULP3-SOURCE-FEDORA36.puffy.example.com/pulp/api/v3/content/container/manifests/?media_type=application/vnd.docker.distribution.manifest.v2%2Bjson" --auth admin:password
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 635
Content-Type: application/json
Correlation-ID: 2a2d89462a7f407aa9512be3c29c60d7
Date: Thu, 18 Aug 2022 11:35:13 GMT
Referrer-Policy: same-origin
Server: nginx
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "artifact": "/pulp/api/v3/artifacts/ded44334-44db-43be-a16f-1a34170ab109/",
            "blobs": [
                "/pulp/api/v3/content/container/blobs/2fa53240-0411-4e48-bc51-accb58232265/"
            ],
            "config_blob": "/pulp/api/v3/content/container/blobs/33985973-9678-4d3d-af18-ddf2fe5a7b42/",
            "digest": "sha256:7989ac99935ab357c08d5a2b10c732ad1724477a9db27a056fd341d99c16f59a",
            "listed_manifests": [],
            "media_type": "application/vnd.docker.distribution.manifest.v2+json",
            "pulp_created": "2022-08-18T11:34:51.223190Z",
            "pulp_href": "/pulp/api/v3/content/container/manifests/9c312afd-13ea-40c3-b030-c79f1235cf6a/",
            "schema_version": 2
        }
    ]
}


(pulp) [vagrant@pulp3-source-fedora36 docs]$ http "https://PULP3-SOURCE-FEDORA36.puffy.example.com/pulp/api/v3/distributions/?with_content=/pulp/api/v3/content/container/manifests/9c312afd-13ea-40c3-b030-c79f1235cf6a/" --auth admin:password
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, POST, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 52
Content-Type: application/json
Correlation-ID: 638fd27c6e38450fbfc761b99d0377e9
Date: Thu, 18 Aug 2022 11:35:24 GMT
Referrer-Policy: same-origin
Server: nginx
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "count": 0,
    "next": null,
    "previous": null,
    "results": []
}



(pulp) [vagrant@pulp3-source-fedora36 docs]$ http "https://PULP3-SOURCE-FEDORA36.puffy.example.com/pulp/api/v3/distributions/container/container/?with_content=/pulp/api/v3/content/container/manifests/9c312afd-13ea-40c3-b030-c79f1235cf6a/" --auth admin:password
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, POST, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 52
Content-Type: application/json
Correlation-ID: beaf7213f89646debcd41451f2ea4cbe
Date: Thu, 18 Aug 2022 11:35:58 GMT
Referrer-Policy: same-origin
Server: nginx
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "count": 0,
    "next": null,
    "previous": null,
    "results": []
}

ipanova · 2022-08-18T13:41:38Z

@lubosmj i had old version of the PR, false alarm :)

Required PR: pulp/pulpcore#2992 [noissue]

closes pulp#2952

Required PR: pulp/pulpcore#2992 [noissue]

lubosmj added a commit to lubosmj/pulp_file that referenced this pull request Jul 20, 2022

Add a test for filtering distributions by content

41589cb

Required PR: pulp/pulpcore#2992 [noissue]

lubosmj mentioned this pull request Jul 20, 2022

Add a test for filtering distributions by content pulp/pulp_file#778

Merged

lubosmj added a commit to lubosmj/pulp_file that referenced this pull request Jul 20, 2022

Add a test for filtering distributions by content

70b9234

Required PR: pulp/pulpcore#2992 [noissue]

lubosmj added a commit to lubosmj/pulp_file that referenced this pull request Jul 20, 2022

Add a test for filtering distributions by content

4ff2d8b

Required PR: pulp/pulpcore#2992 [noissue]

lubosmj force-pushed the distributions-content-in-2952 branch from 9288ada to eeeb3ae Compare July 20, 2022 14:53

lubosmj added a commit to lubosmj/pulp_file that referenced this pull request Jul 20, 2022

Add a test for filtering distributions by content

de0bf34

Required PR: pulp/pulpcore#2992 [noissue]

lubosmj added a commit to lubosmj/pulp_file that referenced this pull request Jul 20, 2022

Add a test for filtering distributions by content

0b88266

Required PR: pulp/pulpcore#2992 [noissue]

gerrod3 reviewed Jul 22, 2022

View reviewed changes

dkliban marked this pull request as draft July 26, 2022 13:08

lubosmj force-pushed the distributions-content-in-2952 branch from eeeb3ae to 8ee5667 Compare July 27, 2022 20:42

lubosmj added a commit to lubosmj/pulp_file that referenced this pull request Jul 27, 2022

Add a test for filtering distributions by content

70757a2

Required PR: pulp/pulpcore#2992 [noissue]

lubosmj marked this pull request as ready for review July 27, 2022 20:47

lubosmj requested a review from gerrod3 July 27, 2022 20:47

lubosmj added a commit to lubosmj/pulp_file that referenced this pull request Jul 27, 2022

Add a test for filtering distributions by content

2a0fef3

Required PR: pulp/pulpcore#2992 [noissue]

lubosmj added a commit to lubosmj/pulp_file that referenced this pull request Jul 27, 2022

Add a test for filtering distributions by content

421c2bd

Required PR: pulp/pulpcore#2992 [noissue]

lubosmj force-pushed the distributions-content-in-2952 branch from 8ee5667 to 315aeee Compare July 28, 2022 07:47

lubosmj added a commit to lubosmj/pulp_file that referenced this pull request Jul 28, 2022

Add a test for filtering distributions by content

77380d2

Required PR: pulp/pulpcore#2992 [noissue]

ipanova reviewed Jul 28, 2022

View reviewed changes

lubosmj force-pushed the distributions-content-in-2952 branch from 315aeee to 390720a Compare August 12, 2022 19:07

lubosmj added a commit to lubosmj/pulp_file that referenced this pull request Aug 12, 2022

Add a test for filtering distributions by content

6fdf03b

Required PR: pulp/pulpcore#2992 [noissue]

newswangerd approved these changes Aug 15, 2022

View reviewed changes

lubosmj added a commit to lubosmj/pulp_file that referenced this pull request Aug 16, 2022

Add a test for filtering distributions by content

f77f830

Required PR: pulp/pulpcore#2992 [noissue]

bmbouter approved these changes Aug 16, 2022

View reviewed changes

ipanova approved these changes Aug 18, 2022

View reviewed changes

mdellweg approved these changes Aug 22, 2022

View reviewed changes

lubosmj added a commit to lubosmj/pulp_file that referenced this pull request Aug 22, 2022

Add a test for filtering distributions by content

d39ff86

Required PR: pulp/pulpcore#2992 [noissue]

lubosmj added a commit to lubosmj/pulp_file that referenced this pull request Aug 22, 2022

Add a test for filtering distributions by content

92a87ac

Required PR: pulp/pulpcore#2992 [noissue]

lubosmj force-pushed the distributions-content-in-2952 branch from 390720a to 16b40d5 Compare August 22, 2022 10:44

lubosmj marked this pull request as draft August 22, 2022 10:44

lubosmj marked this pull request as ready for review August 22, 2022 10:45

lubosmj added a commit to lubosmj/pulp_file that referenced this pull request Aug 22, 2022

Add a test for filtering distributions by content

7f04996

Required PR: pulp/pulpcore#2992 [noissue]

Enable filtering by content

a5fd27a

closes pulp#2952

lubosmj force-pushed the distributions-content-in-2952 branch from 16b40d5 to a5fd27a Compare August 22, 2022 12:47

mdellweg merged commit e6a42a9 into pulp:main Aug 22, 2022

mdellweg pushed a commit to pulp/pulp_file that referenced this pull request Aug 22, 2022

Add a test for filtering distributions by content

8a55eaa

Required PR: pulp/pulpcore#2992 [noissue]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable filtering by content #2992

Enable filtering by content #2992

lubosmj commented Jul 20, 2022

gerrod3 Jul 22, 2022

lubosmj Jul 22, 2022

ipanova Jul 28, 2022

ipanova Jul 28, 2022

bmbouter commented Aug 9, 2022

newswangerd commented Aug 11, 2022

lubosmj commented Aug 12, 2022

newswangerd left a comment

lubosmj commented Aug 15, 2022 •

edited

Loading

bmbouter commented Aug 15, 2022

bmbouter left a comment

ipanova commented Aug 18, 2022

ipanova commented Aug 18, 2022

		for dist in qs.exclude(repository=None).prefetch_related("repository__versions"):
		versions_distributions[dist.repository.latest_version().pk].append(dist.pk)



		class DistributionWithContentFilter(Filter):
		"""A Filter class enabling filtering by content units exposed by distributions."""

Enable filtering by content #2992

Enable filtering by content #2992

Conversation

lubosmj commented Jul 20, 2022

gerrod3 Jul 22, 2022

Choose a reason for hiding this comment

lubosmj Jul 22, 2022

Choose a reason for hiding this comment

ipanova Jul 28, 2022

Choose a reason for hiding this comment

ipanova Jul 28, 2022

Choose a reason for hiding this comment

bmbouter commented Aug 9, 2022

newswangerd commented Aug 11, 2022

lubosmj commented Aug 12, 2022

newswangerd left a comment

Choose a reason for hiding this comment

lubosmj commented Aug 15, 2022 • edited Loading

bmbouter commented Aug 15, 2022

bmbouter left a comment

Choose a reason for hiding this comment

ipanova commented Aug 18, 2022

ipanova commented Aug 18, 2022

lubosmj commented Aug 15, 2022 •

edited

Loading