bpo-40334: PEP 617: New PEG parser for CPython #19503
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- The pegen parser generator and its tests are in Tools/peg_generator. This is mostly copied from the https://github.com/gvanrossum/pegen repo, with some adaptations. This is still targeting 3.8. - The new grammar (adapted to Python 3.9) is in Grammar/python.gram. - There's a new builtin module peg_parser, configured in Setup. - There's a new Make target regen-pegen which runs the parser generator with output directed to Modules/peg_parser/parse.c. There's some code duplication between Tools/peg_generator and Modules/peg_parser, due to the 3.8/3.9 split and some tweaks needed to link with the core. Eventually this probably should go away.
* Use trailer meta for a function that calls the start rule * Expose four C API functions that return either a mod_ty or a PyCodeObject * * Expose two Python-level functions that return an AST object * Extension module and C API does not get generated, but is hand-written * Rewrite of run_parser* functions Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
* The caller is responsible for transforming the mod_ty a code or an AST object * The C API functions don't expect the filename and the arena (where possible) objects as an argument, but construct them themselves
* Rewrite exported function:
* It now initializes the keywords field so that the keywords variable can be static
* It also accepts an argument, in order to be able to run the parser with different start rules
* Use the newly created function in fstring_compile_expr
…enerator itself (#23) * Remove C code from Tools/peg_generator/peg_parser and rely on Parser/pegen * Move all tests to Lib/test/test_peg_generator/* * Disable pegen tests on Github Actions
This is a straightforward port of we-like-parsers/pegen_experiments#236 . In addition, slightly touched up .gitignore, Makefile (black), mypy.ini.
* Remove start rule and add file, interactive, eval and fstring rules * Accept a mode parameter in C API functions
…active mode (#38) * Refactor raise_syntax_error to correctly handle interactive mode * Correctly handle Ctrl-D in interactive mode
Additionally, fix the script to calculate the nesting level using the generated extension.
* Fix audit tests * Try to fallback to get_error_line if we cannot get the error line using PyErr_ProgramTextObject * Add -E flag to run_tests.py only if PYTHONPARSER is not set Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
|
Fixing the buildbots here: #19666 |
|
|
Let me see how many buildbot workers you can break with a single commit :-D |
|
|
(not to be a downer but) this is breaking pyflakes's testsuite pretty hard on the |
|
Thanks for the heads up! We'll look into this once alpha6 is out (expected Monday). |
|
It seems like this PR has introduced the syntax: Since PEP 617 clearly states that "no new Python Grammar addition will be added that requires the PEG parser", I wonder whether this change was intentional? If it is I guess we would have to modify PEP 617. IMO I would strongly prefer having a grace period for third party parsers (like To be honest I'm just a big afraid of people complaining that parso doesn't support the latest Python syntax. PS: I'm also interested in other potential changes to the grammar, since I have never seen this communicated anywhere. |
|
It's an easter egg. In 3.9 people are not supposed to write this and you can point out that it's undocumented. In 3.10 we will come clean. |
|
Ok, thanks for clarifying. Will link to this comment in case people start complaining :). |
|
If this is actually unsupported, it should be documented as unsupported. @davidhalter brings up a good point about other Python parsers. As nice as this new syntax is, it would probably be better if it were held until 3.10. It's also very odd that setting I don't really see how this can be be an "easter egg". People are going to write this, because it's a natural thing to write (and it's good that it will indeed be supported by the new PEG parser). Then they will run into problems with tools that use other parsing libraries like parso, or if anyone sets |
|
I offered to take it out but @ambv wanted it. But if tools reject it, that's no different than tools rejecting other things that happen to work but are bad style, right? |
It does not change the documented behaviour of Python.
I do not agree with your view: the feature is not documented and therefore should not be relied upon (but there is no need to remove the easter egg because of that). Is sort of the same as when we made ordered dictionaries: it was considered an implementation detail and users should not rely on it (until we made it official). But without entering into a discussion about this, there is the much more important fact that we are going to release the first release candidate on Monday and removing this will be too risky for the stability of the release. The release manager is aware of this and it was ok with us maintaining it (#19704). |
Wait, so using parentheses in with statements is bad style? Why was it even added if you think this?
Something not being documented is not the same as being an easter egg. Easter eggs are supposed to be fun features that may or may not be documented, and don't actually affect real world usage of Python.
I recall the dictionary thing being well documented and highly discussed. It feels like you are trying to slip this in without anyone noticing. I can't really understand the motivation for that. But it should be quite clear that once this syntax works, people will use it.
If it affects compatibility in a negative way, it should be removed. If you can't make changes to a release before the release date, why not just do the final release now? If this isn't going to be removed, it should at least be documented and mentioned in the what's new section, and the incorrect bolded statement in the PEP should be updated. Not actually doing what it says it it does and calling it an "easter egg" is a lame cop out and rather insulting to people who actually try to maintain libraries that match the semantics of the Python parser. |
|
Unfortunately, I am not interested in starting a discussion on this topic |
|
@pablogsal if you don't want to participate in the discussion that is fine. No one asked you to. I hope you aren't suggesting that there shouldn't be a discussion at all. I don't think that's something that you have or should have the right to do. Who makes final decisions about new syntax features in Python? The steering council, or someone else? |
|
A better place to have a discussion would be bpo. |
I will clarify my words: I am just indicating that I am not interested in starting a discussion on this, but of course, you are free to do whatever you think necessary. |
|
@gvanrossum that's fair. Discussions like this shouldn't happen on closed PRs. I opened https://bugs.python.org/issue41506. I don't know how to use the nozy feature on bpo so anyone please nozy yourself if you are interested. |
|
Also I should note I didn't want to summarize other people's opinions so please restate them on the bpo if you want it to be heard. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The SC has approved PEP 617 so we'd like to land before alpha 6 goes out. We need reviews NOW.
https://bugs.python.org/issue40334