Decide what PEPs fall under Topic: Packaging and act accordingly

[CAM-Gerlach]

Right now, as originally discussed in #2096 , there seem to be two plausible interpretation of what qualifies for the Topic: Packaging label: PEPs that relate to packaging in some direct way, and PEPs that apply to packaging ecosystem (as opposed to the stdlib), and are thus discussed primarily on packaging community fora and accepted/rejected by the relevant packaging-community standing PEP delegate.

In particular, given PEP 632 (PEP-632), on deprecating and removing disutils, was removed from packaging at the request of @zooba for relating to the core stdlib, this would seem to suggest that PEP 453 (PEP-453) and PEP 477 (PEP-477), on ensurepip and its backport to Python 2.7, respectively, should also be removed by the same standard (or, alternatively, PEP 632 restored).

Presuming consensus can be reached prior to its merging, I can implement this on the existing #2690 , given it makes a number of related improvements/refinements to the packaging-tagged PEP headers.

Relevant discussion from #2096 (click to expand)

@pfmoore

this is a core Python PEP, not a packaging PEP, and like @dstufft noted, I don't think we tend to modify those after approval.

@CAM-Gerlach

In that case, as originally discussed on #2656 and done for PEP 632 (PEP-632), maybe we should drop Topic: Packaging from this PEP as well as the related PEP 477 (PEP-477) ? I can do it in #2690 since its closely related to the other changes in that PR (also, your input there on updating PEP 262 (PEP-262)'s status to reflect reality would be helpful).

@pfmoore

I don't know on that. It's packaging related in the sense that it is about packaging tools, so if people are looking for things that are "about" packaging, it does belong in the Packaging topic. But it's not a "PyPA specification" in the sense defined here, which is why I said that the normal PEP rules about modifications should apply, rather than the PyPA-specific process. Ultimately, I'm not actually sure what the rules are for when a PEP qualifies as being in the Packaging topic...

@CAM-Gerlach

Yeah, exactly. That basically comes down to whether a Topic is considered more of a "Category" (i.e. the former), or a "Track" (i.e. the latter). I presented both alternatives at the beginning of the discussion, and both were favored at different points in its evolution; the initial consensus in the Discourse thread was something more like a "Track", but on the PR and naming of the "Topic" header that was eventually implemented (that allows multiple topics per PR), it ended up basically the latter, at least nominally.

However, with PEP 632 (PEP-632) being removed at the behest of @zooba in #2636 (comment) , given these two PEPs basically just concern adding/backporting a utility module from the core standard library rather than PyPA/PyPI standards, it seems to me to not make sense to keep those PEPs but not another very similar one about the deprecation and removal of what was the core packaging module. So, I suggest either removing those two, or adding back PEP 632.

@pradyunsg

Alternatively, we can defer doing anything here until the authors have a concern with being listed under the packaging index as well or till we see/identify more instances of something like this? In any case, this discussion should move to a separate issue IMO.

@zooba

I think this is in the same boat as PEP 632, and shouldn't be tagged as a packaging PEP.

As a proposal of the rule I'd use, if we'd let the packaging delegate (i.e. Paul, right now) decide without going to python-dev or the SC, then it's a packaging PEP. Whether something is added to the standard library clearly falls outside of this scope, and so this PEP is standards track and not packaging.

[CAM-Gerlach]

Alternatively, we can defer doing anything here until the authors have a concern with being listed under the packaging index as well or till we see/identify more instances of something like this?

IMO, if it leave it up to authors to take the initiative and opt in or out, there likely won't be clear consistency in what PEPs get included, making the header much less useful than it should be. As this is (at least as of now) purely editorial metadata, rather than PEP content or changing the PEP approval process (that's still driven by the PEP-Delegate assignment), it is thus only an editorial concern.

As a proposal of the rule I'd use, if we'd let the packaging delegate (i.e. Paul, right now) decide without going to python-dev or the SC, then it's a packaging PEP. Whether something is added to the standard library clearly falls outside of this scope, and so this PEP is standards track and not packaging.

If we decide that's what we want Topic: Packaging to mean, then that (or perhaps a slightly more general form, i.e. it being something primarily debated and decided by the packaging community and its delegated representatives, rather than Python-Dev, the core devs and the SC directly, to encompass the multiple delegations and historical role of other bodies, such as Distutils-SIG, prior to the modern standing delegations), seems to be a fairly reasonably proxy to use.

[pfmoore]

My (purely personal!) understanding of the "Packaging" topic was for it to be helpful for readers to locate PEPs that were relevant to the Packaging ecosystem. I don't really see an advantage to having the topic simply reflect the PEP-delegate, given that there's already a PEP-delegate field in the PEP.

If we want a field that reflects what "rules" apply to the PEP (standard CPython PEP, as documented in PEP 1, or PyPA specification PEP, as documented on pypa.io) then maybe that's a different matter, but I don't see why a purely technical distinction based on management process warrants a split in how the PEPs are presented to the user.

So for me (again, personally, not as packaging PEP-delegate) I'd prefer the topic field to reflect the subject matter, not the process.

[CAM-Gerlach]

Indeed, that was also my understanding as well and what seemed to be the consensus of the discussion that added it, and why I proposed the name Topic over the initially suggested Track. As such, this seems the most reasonable interpretation to me given the designed syntax and semantics of the field (and indeed, we've so far added two more Topics, Release for release PEPs and Typing for typing PEPs, neither of which correspond specifically to the delegated authority for the PEP nor whether it affects the stdlib (or even the core language, for many typing PEPs).

We could introduce a new field Track for the special case of PEPs that follow a different process or don't affect the stdlib as opposed to other ecosystems, which right now would basically be Packaging, but that seems to be overcomplicating things when it isn't really what most people seem to be asking for, and the difference is small and getting smaller—plus, the PyPA specs site will, hopefully soon (with pypa/packaging.python.org#955 and pypa/packaging.python.org#1093 ) be the place for that anyway.

Therefore, I favor retaining those two PEPs and add back PEP-632. However, my overriding interest is that at least the basic editorial metadata and categorization is consistent such that readers can rely on it, so if the consensus favors the narrower definition, then those two PEPs should also be removed.

[zooba]

Therefore, I favor retaining those two PEPs and add back PEP-632.

I'm okay with this, as long as they are somehow marked as "frozen" and people don't propose updates to them anymore (or those are immediately shut down by everyone else and they don't keep pinging me until I shut it down :) ). These aren't typical packaging specs, they are decisions made about the standard library, and once implemented are not open to further tweaking in the way that package metadata and protocols are, and are not living documentation or reference material.

I think that's a distinction we want to make between PEPs and specs anyway, so maybe this is fine? If so, let's please accelerate that.

[pfmoore]

PyPA specs¹ have a different policy here. I agree it should be much clearer which set of rules apply to any given PEP, but I didn't think that was the intention of the "Packaging" topic. Both markings seem useful to me, though.

I don't know who can decide here, though - maybe it needs to go to the SC? I think that historically the modified process for PyPA specs was set up with SC approval, but I'm not sure - @ncoghlan may know.

Footnotes

1. And I think typing want to do something similar too. ↩

[CAM-Gerlach]

@zooba

I'm okay with this, as long as they are somehow marked as "frozen" and people don't propose updates to them anymore (or those are immediately shut down by everyone else and they don't keep pinging me until I shut it down :)

In #2378 , we have more clearly documented in PEP 1, the contributing guide, etc. that PEPs in general are historical documents rather than living documentation/specifications, and modifications to final PEPs are generally not accepted (unless in certain cases where they fix an obvious, unambiguous and non-trivial editorial or reST syntax defect); we've become more diligent about quickly resolving such issues/PRs one way or another. If notifications about this continue to be a non-trivial problem, you could remove your username from the CODEOWNERS such that it would fall back to the PEP editors team as the first responders, and we'd only pinging you in the unlikely case there was some legitimate reason.

In regards to retaining those specs on the "Packaging" PEP topic specifically, it seems pretty unlikely to bring much additional traffic to these PEPs, since it is only really aimed at a fairly narrow section of packaging folks experienced with the PEP process (and is not that easy to navigate to unless you're specifically looking for it), whereas the PyPA spec site has always been the intended go-to place for the packaging community at large, and the new admonitions in #2690 will help push that further. And in the more general case, the proposed directive in #2692 will be able to do the same for every Final PEP (including yours), adding an admonition banner at the top pointing to its canonical living documentation/specification and letting readers know that they generally aren't modified.

I think that's a distinction we want to make between PEPs and specs anyway, so maybe this is fine? If so, let's please accelerate that.

Yeah, the idea is (and nominally per PEP 1, always was) that PEPs should be change proposals, while the living specification or documentation is hosted elsewhere. but it is a long, slow, high-friction process involving a bunch of different communities. The effort for packaging is, of course, the furthest along, with recent further work on the aforementioned issues and I and others are continuing to push that forward. The aforementioned admonition banners will also help make this much more visible to those viewing the PEPs.

[CAM-Gerlach]

@pfmoore

PyPA specs have a different policy here. I agree it should be much clearer which set of rules apply to any given PEP, but I didn't think that was the intention of the "Packaging" topic. Both markings seem useful to me, though.

#2690 will do so for the existing PyPA spec PEPs, and #2692 will provide a standardized extension of that mechanism that can be applied to any PEP. My comment in pypa/packaging.python.org#1093 (comment) lists all the current PEPs that already correspond to PyPA specs (some fully migrated, some still stubs), as well as a few other candidates for migration.

To note, though, as I understand it the SC standing delegation principally applies to your and Donald's role as the default PEP delegate for packaging PEPs that concern the packaging formats/metadata and PyPI respectively, and that's already captured in the PEP-Delegate header field. It has always been normative PEP 1 policy that the PEPs are not the canonical location for living documentation or specifications after the PEPs are marked final, with the appropriate location determined on a PEP by PEP basis:

Formal documentation of the expected behavior should be maintained elsewhere, such as the Language Reference for core features, the Library Reference for standard library modules or the PyPA Specifications for packaging.

Therefore, given the Topic header is non-normative and was itself not a SC-level change, I'm not sure if there's anything in this issue the SC needs to get directly involved in, if @zooba is satisfied with the mitigations mentioned in my reply to him immediately above. There might be some edge cases in the above list where we're not sure what of the "possible" list should get added to the PyPA specs page if there's disagreement, but that's as much a matter for the PyPA and the page maintainers as the SC, and any non-trivial change that would involve delegated SC authority would have to go through a PEP anyway, at which point it could be decided whether the standing delegation applies or not.

1. And I think typing want to do something similar too. leftwards_arrow_with_hook

Yup, that seems to be the plan as well as I understand it.

[pfmoore]

To note, though, as I understand it the SC standing delegation principally applies to your and Donald's role as the default PEP delegate for packaging PEPs that concern the packaging formats/metadata and PyPI respectively, and that's already captured in the PEP-Delegate header field. It has always been normative PEP 1 policy that the PEPs are not the canonical location for living documentation or specifications after the PEPs are marked final, with the appropriate location determined on a PEP by PEP basis

That's my understanding, too - which is why I see the current attempts to "clarify" the process for packaging PEPs as more confusing than helpful. I think there was originally an intention when the PyPA process was formed, to try to keep the packaging process more flexible because we felt we were moving at a quicker pace than core Python, but in hindsight I don't think that's as important as we expected it to be.

It's also worth noting that some (core) PEPs do have the PEP as the canonical location for the documentation - PEP 8, and the DB-API, are two examples I can immediately think of. Packaging PEPs really aren't as different as people imagine.

Yup, that seems to be the plan as well as I understand it.

The typing community may well have their own constraints here, and I think we should be cautious about assuming that what works for packaging works for typing as well. Having a process for "non-core" PEPs (with input from both the packaging and the typing communities) is one thing. Having multiple processes, one for each community, all mostly similar but with subtle differences, is another thing entirely...

[CAM-Gerlach]

It's also worth noting that some (core) PEPs do have the PEP as the canonical location for the documentation - PEP 8, and the DB-API, are two examples I can immediately think of. Packaging PEPs really aren't as different as people imagine.

Yeah, the general idea seems to try to move away from that where practical, but in special cases like PEP 7, PEP 8 and the other process PEPs (1, 13, etc) that are living process documents that intrinsically relate to core Python, that seems very unlikely to ever change. But those cases aren't that common and they have a special Type and guidelines specifically for them.

The typing community may well have their own constraints here, and I think we should be cautious about assuming that what works for packaging works for typing as well.

Yeah, as I understand it they've been facing a lot of similar issues with the PEPs being the specs and the documentation, but also historical normative change documents with its own process, when it comes to a lot of people requesting updates, fixes and improvements as well as not having one set of coherent specs all in one place for people to reference. Their specific solution might be different in the details, but at the high-level and as far as the PEPs are concerned, that side can be handled similarly (with whatever appropriate custom text they want in the admonition).

@JelleZijlstra feel free to chime in here...

[pradyunsg]

So… what’s the next steps here?

Do we have a concrete answer / phrasing? If not, here’s one:

For new PEPs, only things that go through the PyPA processes should end up there. For existing PEPs, let’s not try to figure out some sort of a blanket rule and only migrate things where there’s a general consensus, and the delegates/authors aren’t opposed.

If that doesn’t work, let’s iterate. If we’re not able to figure something out, then this is an escalate-to-SC discussion, and then we’d need to figure out what’s the question + context to provide the SC so that they can decide. :)

[zooba]

Any PEPs that were approved while the PyPA processes existed but not by those processes should also be excluded.

[CAM-Gerlach]

I'm not sure if anyone here really disagrees on what PEPs should be covered by the PyPA processes and migrated to the PyPA specs page; the main concern (and the primary question posed by this issue) is whether the Topic is mere editorial metadata to help the reader find PEPs related to a certain topic (under which PEP 632, as well as PEP 453 and 477, should be included, since they are about packaging as a topic), or whether Topic: Packaging delineates a special class of PEPs that have different rules and processes (e.g. PEP delegate, canonical spec on packaging.python.org), under which those PEPs clearly don't belong. Originally, it was framed more as the latter, but how Topic has been used (Typing, Release PEPs, etc) and might be used in the future, it seems to favor the former interpretation.

Either way, the description message on the topic page should be updated, to make the intended scope more clear to everyone, as well as, per @zooba , adding the new banner directive I introduced in #2702 for this purpose to the relevant non-PyPA PEPs:

I'm okay with this, as long as they are somehow marked as "frozen" and people don't propose updates to them anymore (or those are immediately shut down by everyone else and they don't keep pinging me until I shut it down :)

[CAM-Gerlach]

I just talked with @brettcannon at the sprints and he favored the latter interpretation, that the Packaging topic should be equivalent to the PyPA specs / i.e. those that would be hosted on packaging.python.org (i.e. not the stdlib PEPs listed above) and that these cases weren't worth including here. If we agree on moving forward with that, I can either in #2690 or (after that's merged) make a new PR to drop the tag from the ensurepip PEPs, and perhaps further tweak the summary at the top to make this clear (though it already implies this).

[pradyunsg]

I also favour the latter interpretation and dropping ensurepip PEPs sounds like a good idea. :)

[ncoghlan]

For the record, belatedly answering @pfmoore's question from July: the formal record of the PyPA PEP process delegation is in the SC repo at https://github.com/python/steering-council/blob/main/process/standing-delegations.md

[CAM-Gerlach]

If anyone else wants to share their feedback on #2826 , now's the time! If no one objects, I was going to go ahead and merge it in a week or so, unless anyone wants to earlier.