A new Python REPL

[pablogsal]

This issue will track all the different steps to bootstrap a new Python REPL with many new features. The target of this issue is that we can reach a point where new contributions can be made easily to the REPL once is written in pure Python. This issue only coveres the initial ground work to reach this status.

Tasks to be done:

• Redirect the parser and the main module to a new Python module.
• Fallback to the previous tokenizer-based REPL if the terminal is not a tty (and for backwards compat reasons)
• Bootstrap a new REPL module (based on a trimmed version pypy's pyrepl module).
• Ensure that the basic features work:
  • History
  • Completions
  • Ctrl-r and Ctrl-s

Linked PRs

• gh-111201: A new Python REPL #111567
• Add Lysandros Nikolaou to the News entry of gh-111201 #118616
• gh-111201: Fix event queue tests for pyrepl #118635
• gh-111201: Use a more common constant in completion tests in test_pyrepl #118638
• gh-111201: Add tests for unix console class in pyrepl #118653
• gh-111201: Allow bracketed paste to work #118700
• gh-111201: Add more tests to test_pyrepl to cover key translation #118705
• gh-111201: Allow pasted code to contain multiple statements in the REPL #118712
• gh-111201: Remove readline dependency from the PyREPL #119262
• gh-111201: Add append to screen method to avoid recalculation #119274
• gh-111201: Speed up paste mode in the REPL #119341
• gh-111201: auto-indentation in _pyrepl #119348
• [3.13] gh-111201: Remove readline dependency from the PyREPL (GH-119262) #119403
• [3.13] gh-111201: Add append to screen method to avoid recalculation (GH-119274) #119405
• [3.13] gh-111201: auto-indentation in _pyrepl (GH-119348) #119427
• gh-111201: Use calc_complete_screen after bracketed paste in PyREPL #119432
• [3.13] gh-111201: Use calc_complete_screen after bracketed paste in PyREPL (GH-119432) #119439
• gh-111201: Support pyrepl on Windows #119559
• gh-111201: Improve pyrepl auto indentation #119606
• [3.13] gh-111201: Improve pyrepl auto indentation (GH-119606) #119833
• gh-111201: [pyrepl] Ensure optional platform-specific imports are optional #119834
• gh-111201: Skip pyrepl Windows tests earlier #119848
• [3.13] gh-111201: Support pyrepl on Windows (GH-119559) #119850
• [3.13] gh-111201: Skip pyrepl Windows tests earlier (#119848) #119924
• gh-111201: fix auto-indent in pyrepl for muliple pound comments #123196
• [3.13] gh-111201: fix auto-indent in pyrepl for muliple pound comments (GH-123196) #123764

[pablogsal]

If you want to help with this endeavour, please mention it here so we can coordinate everyone!

[pablogsal]

I have a initial version that we can start building upon, I will create a PR shortly.

[tomasr8]

I learned about this from the "core.py" podcast and it sounds like a great idea! I'd be happy help with this effort :)

[vstinner]

Fallback to the previous tokenizer-based REPL if the terminal is not a tty (and for backwards compat reasons)

Do you mean not using _pyrepl? Can _pyrepl be used without readline nor tty?

[pablogsal]

Can _pyrepl be used without readline nor tty?

It needs at least a tty yes. Readline can be lifted but I am not sure yet

[novaTopFlex]

I would like to see a new REPL (read, evaluate, print, loop) for the Python interpreter if the interpreter goes by a different name. Two other major Python interpreters also exist, and they already use other names ("bpython" and "ipython"). I would recommend forking the current REPL and renaming the interpreter with the new REPL to "zpython" instead. Also, I would like to see a built-in "clear()" function in the REPL as I already encounter with the "exit()" function and other REPL-specific functions.

[theLastOfCats]

Big feature without PEP?

[encukou]

Most buildbots are broken with:

Traceback (most recent call last):
  File ".../Lib/unittest/mock.py", line 1420, in patched
    return func(*newargs, **newkeywargs)
  File ".../Lib/test/test_pyrepl.py", line 643, in test_push_without_key_in_keymap
    eq = EventQueue(sys.stdout.fileno(), "utf-8")
                    ~~~~~~~~~~~~~~~~~^^
io.UnsupportedOperation: fileno

e.g.: https://buildbot.python.org/all/#/builders/725/builds/7939/steps/5/logs/stdio

[pablogsal]

Checking, will make a PR this morning

[lysnikolaou]

I already have something, will push in a bit.

[kulikjak]

Hi, I am seeing the following error:

======================================================================
FAIL: test_completion_with_many_options (test.test_pyrepl.TestPyReplCompleter.test_completion_with_many_options)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/..../cpython-main/Lib/test/test_pyrepl.py", line 586, in test_completion_with_many_options
    self.assertEqual(output, "os.O_ASYNC")
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'os.O_AS' != 'os.O_ASYNC'
- os.O_AS
+ os.O_ASYNC
?        +++

The reason is most likely that os.O_ASYNC is not necessarily available on all platforms (I am seeing this on Solaris where it's missing). I tried replacing it with O_AP and O_APPEND and then it works as expected.

[hauntsaninja]

(hmm there is only one constant starting with os.O_AS, so not sure where the "with_many_options" comes in. should it just be testing os.O_? if anything test_simple_completion tests multiple options because of getenv and getenvb)

[lysnikolaou]

I think we can close this and handle the rest of the bugs in individual issues. Feel free to re-open if you feel otherwise.

[0dminnimda]

Thanks to all the participants, this is amazing! I really enjoy its increased portability!