Submission for SC consideration: PEP 646 -- Variadic Generics

[gvanrossum]

Please consider PEP 646 for inclusion in Python 3.10. This is primarily of interest to users of static type checkers, especially projects looking to implement shape checking for tensors (an important concept in the machine learning world, where Python's dominance is seeing competition from e.g. Swift for Tensorflow).

There are some small additions to Python's syntax in the proposal, namely the * operator in subscripts and for *args in function definitions. These are essentially just new uses of the existing "sequence unpacking" concept.

The PEP has been discussed at length in the typing sig, and the python-dev mailing list has been notified.

There are reference implementations of the checking part in Pyright and Pyre. There is a nascent patch for the syntactic changes to CPython where.

/CC: @mrahtz

https://www.python.org/dev/peps/pep-0646/

[mrahtz]

Thanks, Guido! There'll be one last section coming in the next week or so at the request of Stephan Hoyer, one of the core developers of NumPy and the JAX machine learning library. However, this section will just be clarifying how the PEP relates to other philosophies to tensor typing - the PEP is semantically complete.

…

On Wed, 31 Mar 2021 at 18:45, Guido van Rossum ***@***.***> wrote: Please consider PEP 646 for inclusion in Python 3.10. This is primarily of interest to users of static type checkers, especially projects looking to implement shape checking for tensors (an important concept in the machine learning world, where Python's dominance is seeing competition from e.g. Swift for Tensorflow). There are some small additions to Python's syntax in the proposal, namely the * operator in subscripts and for *args in function definitions. These are essentially just new uses of the existing "sequence unpacking" concept. The PEP has been discussed at length in the typing sig, and the python-dev mailing list has been notified ***@***.***/message/5GWS2GGVJ4PAMOWM6YVVKZVMR5BRFRGV/> . There are reference implementations of the checking part in Pyright and Pyre. There is a nascent patch for the syntactic changes to CPython where <python/cpython#24527>. /CC: @mrahtz <https://github.com/mrahtz> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#59>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBZ32BBV4VLTP4HPIE4FDDTGNNT5ANCNFSM42FFQEHA> .

[brettcannon]

Sorry for not responding sooner, but this PEP is in our review queue.

[gvanrossum]

The additional section is still being written at python/peps#1904.

[brettcannon]

All of the PEP authors should be getting an email I just sent to python-dev, but just in case the archived copy is https://mail.python.org/archives/list/python-dev@python.org/thread/5BVYPRJKQHKDLLN3T7H4WXP2NAD2ITFH/.

It basically says we unfortunately need to postpone considering this PEP for Python 3.11 instead of 3.10 due to time constraints between now and b1. We are also asking for a much more expanded discussion of the proposed syntax change as it is barely touched upon but is a rather critical part of the proposal.

[gvanrossum]

Thanks Brett, I understand.

@mrahtz regarding the proposed syntax change, I recommend that you reach out on python-dev and ask for a core dev to help you with that change. For example it would help if there was a working (draft mode) PR that implemented the syntax change and assigned it proper semantics (which should presumably that it works like *expr works in any other place). Unfortunately I have got my hands into too many different projects already and I can't help right now (this could change in a few months, but I recommend you don't hold your breath).

[mrahtz]

Thanks Guido. Probably what I'll do is give it a shot on my own first based on what I can learn from the PEP 637 PR, and reach out to a core dev for feedback once I've got something basically working. Stay tuned! :)

…

On Sat, 24 Apr 2021 at 05:00, Guido van Rossum ***@***.***> wrote: Thanks Brett, I understand. @mrahtz <https://github.com/mrahtz> regarding the proposed syntax change, I recommend that you reach out on python-dev and ask for a core dev to help you with that change. For example it would help if there was a working (draft mode) PR that implemented the syntax change and assigned it proper semantics (which should presumably that it works like *expr works in any other place). Unfortunately I have got my hands into too many different projects already and I can't help right now (this could change in a few months, but I recommend you don't hold your breath). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#59 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBZ32CB7LNG3ZWPHSHEMNLTKI65VANCNFSM42FFQEHA> .

[brettcannon]

As a reminder, we need more detail, justification, clarification, etc. around the proposed language change that this PEP is asking for is necessary in order for us to be able to review this PEP.

[mrahtz]

Hi Brett, Thanks for the reminder, and apologies for the slow reply - I was intending to reply last week, but got wrapped up in discussion about the relevant changes to the PEP. We've been working on understanding exactly what language changes would be needed over the past few months, and are finally in a position where we can add details of the changes to the PEP. We're in the process of doing this now at python/peps#2039. Before the steering council takes another look at the PEP, though, we'd also like to get explicit buy-in (and, as a prerequisite, feedback) from more libraries, beyond Stephan Hoyer's (core developer of NumPy/JAX) endorsement at ***@***.***/message/UDM7Y6HLHQBKXQEBIBD5ZLB5XNPDZDXV. In particular, we haven't been able to get in contact with folks from TensorFlow or PyTorch yet. We haven't had much luck posting to their mailing lists, so we'll be trying to reach out to individual core developers. What are the dates we should be aware of for the Python 3.11 release schedule? Best, Matthew

…

On Mon, 19 Jul 2021 at 20:21, Brett Cannon ***@***.***> wrote: As a reminder, we need more detail, justification, clarification, etc. around the proposed language change that this PEP is asking for is necessary in order for us to be able to review this PEP. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#59 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBZ32CWGCXQL7GQW3ARGXLTYRUKHANCNFSM42FFQEHA> .

[pablogsal]

What are the dates we should be aware of for the Python 3.11 release schedule?

Check out https://www.python.org/dev/peps/pep-0664/. The PEP would need to be accepted and implemented before 3.11.0b1, so you need to factor the whole process into it (including the discussion in python-dev and the SC deliberation).

[mrahtz]

Thanks, Pablo!

…

On Mon, 26 Jul 2021 at 12:57, Pablo Galindo Salgado < ***@***.***> wrote: What are the dates we should be aware of for the Python 3.11 release schedule? Check out https://www.python.org/dev/peps/pep-0664/. The PEP would need to be accepted and implemented before 3.11.0b1, so you need to factor the whole process into it (including the discussion in python-dev and the SC deliberation). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#59 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBZ32B4FY47TTIOTAGVBXLTZU5P3ANCNFSM42FFQEHA> .

[mrahtz]

@brettcannon We're now ready for the steering council to take another look. The final version of the PEP is online at https://www.python.org/dev/peps/pep-0646/.

(CC @fylux @pradeep90)

[brettcannon]

I have added it to the review queue.

[mrahtz]

Heads up that we've made a small change that gives more flexibility with annotations for *args: python/peps#2125 Apologies for this being a bit last-minute!

[warsaw]

Accepted: https://mail.python.org/archives/list/python-dev@python.org/thread/WIGGXRWVZJQHMQXD4MXKV2K4YQA3XDOW/

(Leaving this issue open because of @gvanrossum's follow up question)

[brettcannon]

This may need to be reconsidered due to python/peps#2125.

[brettcannon]

Actually, I'm going to close this for now as the PR has not been merged yet. Just re-open here and leave a comment and we can add the PEP back to the agenda to (most likely) reaffirm the acceptance.

[pradeep90]

Members of the Steering Council,

We apologize for updating the PEP at this late stage and causing you additional work. There was one aspect of the PEP that was needed to implement it in a compliant manner but that we had either forbidden or left unspecified. We have also made the representation of *args: *Ts easier to use based on feedback from tool authors who rely on the AST.

Summary of changes (overall PEP diff):

Tuple types (PR): We allow tuple types such as tuple[int, str] to be used wherever TypeVarTuples are used. All the other behavior is preserved. We have explained and updated the following parts of the PEP:

1. Remove section "Type Variable Tuples Must Have Known Length", which was artificially forbidding necessary behavior.
2. Describe how to unpack tuple types such as tuple[int, str].
3. Explain Array without type arguments in terms of tuple types.
4. Explain *args: *Tuple[<...>] for tuple types.
5. Explain Callable[[int, *Ts, str], None] in terms of tuple types.

Representation of *args: *Ts (PR): Based on feedback from tool authors, we made the AST for *Ts in *args: *Ts a Starred node, rather than a Starred node wrapped in a Tuple node. This uses the same grammar as the previous version of the PEP, but with a small compiler change. A CPython implementation is here.

Thank you for considering this PEP.

Best,
Matthew Rahtz
S Pradeep Kumar

[gvanrossum]

SC: As the PEP sponsor, I fully support the changes -- lifting the restriction on length simplifies the spec (removing an edge case), the many edits to the PEP clarify a lot of points, and the AST change simplifies the AST structure. Please reopen this issue. I also apologize for the confusion that happened right around the time the SC reviewed this PEP previously. It won't happen again, neither for this PEP for any other PEPs that I might sponsor in the future.

Matthew and Pradeep: The SC elections are currently going on, in less than two days we'll have a new SC. I'm sure the decision can wait until after the holiday season.

[Yhg1s]

I've added it back to the SC agenda, for the next SC.

[encukou]

Accepted again: https://mail.python.org/archives/list/python-dev@python.org/message/OR5RKV7GAVSGLVH3JAGQ6OXFAXIP5XDX/

I wonder if this is the new record for the number of times a PEP was accepted?