[BUG] rich reverts to legacy Windows APIs when stdout is redirected

[ncoghlan]

• I've checked docs and closed issues for possible solutions.
• I can't find my issue in the FAQ.

Describe the bug

I have an application which invokes pdm python install ... in a subprocess. This works fine with Unicode installation paths when run from Windows terminal (pdm uses rich, and one of the items it prints is the full path to the Python executable being installed).

It fails when the same command is run under pytest, and the traceback shows that rich has autodetected the legacy windows console, and is trying to use the mbcs APIs instead of the fully Unicode capable Windows APIs ignoring all of PYTHONUTF8, PYTHONIOENCODING, PYTHONLEGACYWINDOWSFSENCODING and PYTHONLEGACYWINDOWSSTDIO which are set (or explicitly unset in the case of the latter two) to indicate that the legacy Windows APIs should NOT be used (and UTF-8 should be used instead).

Edit: when filling this issue, I forgot the affected process was running in isolated mode, so ignoring the Python level settings would actually have been the right thing to do. Explicitly enabling UTF-8 mode in the isolated subprocess eliminated the misbehaviour.

Presumably the source of the failed detection is something pytest is doing (since turning off console capturing also makes the error go away), but when Python has been explicitly configured to NOT use the legacy Windows IO APIs because I know they won't work, it would be preferable if rich failed on startup, rather than trying to continue and then failing with a Unicode error at some later time.

I also can't report the issue to pytest yet, because I couldn't work out from the documentation or the code exactly how rich is deciding which set of Windows APIs to use and how pytest might be affecting that detection process, so I decided to start here, and then hopefully have something to refer to from a pytest issue report later.

The last several lines in the failing traceback showing the use of the legacy Windows console:

... snip ...
  File "S:\work\.tox\py3.12\Lib\site-packages\pdm\cli\commands\python.py", line 157, in install_python
    ui.echo(f"[info]Executable:[/] {python_info.path}")
  File "S:\work\.tox\py3.12\Lib\site-packages\pdm\termui.py", line 204, in echo
    console.print(message, **kwargs)
  File "S:\work\.tox\py3.12\Lib\site-packages\rich\console.py", line 1673, in print
    with self:
  File "S:\work\.tox\py3.12\Lib\site-packages\rich\console.py", line 865, in __exit__
    self._exit_buffer()
  File "S:\work\.tox\py3.12\Lib\site-packages\rich\console.py", line 823, in _exit_buffer
    self._check_buffer()
  File "S:\work\.tox\py3.12\Lib\site-packages\rich\console.py", line 2027, in _check_buffer
    legacy_windows_render(buffer, LegacyWindowsTerm(self.file))
  File "S:\work\.tox\py3.12\Lib\site-packages\rich\_windows_renderer.py", line 19, in legacy_windows_render
    term.write_text(text)
  File "S:\work\.tox\py3.12\Lib\site-packages\rich\_win32_console.py", line 403, in write_text
    self.write(text)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.1264.0_x64__qbz5n2kfra8p0\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]

tox also provokes this failure, as when the test suite is run via tox -e py3.12 instead of directly, pdm install still reports the same exception, even though pytest is being run with -s

Setting NO_COLOR=1 in the environment does not fix the Unicode compatibility problem.

Click to expand...

Platform

Windows 10, running Windows Terminal.

If you're using Rich in a terminal:

> .\.tox\py3.12\Scripts\python.exe -m rich.diagnose
╭───────────────────────── <class 'rich.console.Console'> ─────────────────────────╮
│ A high level console interface.                                                  │
│                                                                                  │
│ ╭──────────────────────────────────────────────────────────────────────────────╮ │
│ │ <console width=192 ColorSystem.TRUECOLOR>                                    │ │
│ ╰──────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                  │
│     color_system = 'truecolor'                                                   │
│         encoding = 'utf-8'                                                       │
│             file = <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'> │
│           height = 63                                                            │
│    is_alt_screen = False                                                         │
│ is_dumb_terminal = False                                                         │
│   is_interactive = True                                                          │
│       is_jupyter = False                                                         │
│      is_terminal = True                                                          │
│   legacy_windows = False                                                         │
│         no_color = False                                                         │
│          options = ConsoleOptions(                                               │
│                        size=ConsoleDimensions(width=192, height=63),             │
│                        legacy_windows=False,                                     │
│                        min_width=1,                                              │
│                        max_width=192,                                            │
│                        is_terminal=True,                                         │
│                        encoding='utf-8',                                         │
│                        max_height=63,                                            │
│                        justify=None,                                             │
│                        overflow=None,                                            │
│                        no_wrap=False,                                            │
│                        highlight=None,                                           │
│                        markup=None,                                              │
│                        height=None                                               │
│                    )                                                             │
│            quiet = False                                                         │
│           record = False                                                         │
│         safe_box = True                                                          │
│             size = ConsoleDimensions(width=192, height=63)                       │
│        soft_wrap = False                                                         │
│           stderr = False                                                         │
│            style = None                                                          │
│         tab_size = 8                                                             │
│            width = 192                                                           │
╰──────────────────────────────────────────────────────────────────────────────────╯
╭── <class 'rich._windows.WindowsConsoleFeatures'> ───╮
│ Windows features available.                         │
│                                                     │
│ ╭─────────────────────────────────────────────────╮ │
│ │ WindowsConsoleFeatures(vt=True, truecolor=True) │ │
│ ╰─────────────────────────────────────────────────╯ │
│                                                     │
│ truecolor = True                                    │
│        vt = True                                    │
╰─────────────────────────────────────────────────────╯
╭────── Environment Variables ───────╮
│ {                                  │
│     'TERM': None,                  │
│     'COLORTERM': None,             │
│     'CLICOLOR': None,              │
│     'NO_COLOR': None,              │
│     'TERM_PROGRAM': None,          │
│     'COLUMNS': None,               │
│     'LINES': None,                 │
│     'JUPYTER_COLUMNS': None,       │
│     'JUPYTER_LINES': None,         │
│     'JPY_PARENT_PID': None,        │
│     'VSCODE_VERBOSE_LOGGING': None │
│ }                                  │
╰────────────────────────────────────╯
platform="Windows"

Writing a short test program to report those details from pytest (passing -s to turn off pytest's console capturing changes the diagnostic output to resemble the above):

> .\.tox\py3.12\Scripts\pytest.exe rich_test
===================================================================================== test session starts =====================================================================================
platform win32 -- Python 3.12.4, pytest-8.3.1, pluggy-1.5.0
rootdir: S:\work\
configfile: pyproject.toml
plugins: anyio-4.4.0
collected 1 item

rich_test\test_console.py F                                                                                                                                                              [100%]

========================================================================================== FAILURES ===========================================================================================
__________________________________________________________________________________ test_console_diagnostics ___________________________________________________________________________________

    def test_console_diagnostics():
        from rich.diagnose import report
        report()
>       assert False
E       assert False

rich_test\test_console.py:4: AssertionError
------------------------------------------------------------------------------------ Captured stdout call -------------------------------------------------------------------------------------
┌────────────────────── <class 'rich.console.Console'> ───────────────────────┐
│ A high level console interface.                                             │
│                                                                             │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ <console width=79 None>                                                 │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│                                                                             │
│     color_system = None                                                     │
│         encoding = 'utf-8'                                                  │
│             file = <_io.TextIOWrapper name='<tempfile._TemporaryFileWrapper │
│                    object at 0x0000019BE86E11C0>' mode='r+'                 │
│                    encoding='utf-8'>                                        │
│           height = 25                                                       │
│    is_alt_screen = False                                                    │
│ is_dumb_terminal = False                                                    │
│   is_interactive = False                                                    │
│       is_jupyter = False                                                    │
│      is_terminal = False                                                    │
│   legacy_windows = True                                                     │
│         no_color = False                                                    │
│          options = ConsoleOptions(                                          │
│                        size=ConsoleDimensions(width=79, height=25),         │
│                        legacy_windows=True,                                 │
│                        min_width=1,                                         │
│                        max_width=79,                                        │
│                        is_terminal=False,                                   │
│                        encoding='utf-8',                                    │
│                        max_height=25,                                       │
│                        justify=None,                                        │
│                        overflow=None,                                       │
│                        no_wrap=False,                                       │
│                        highlight=None,                                      │
│                        markup=None,                                         │
│                        height=None                                          │
│                    )                                                        │
│            quiet = False                                                    │
│           record = False                                                    │
│         safe_box = True                                                     │
│             size = ConsoleDimensions(width=79, height=25)                   │
│        soft_wrap = False                                                    │
│           stderr = False                                                    │
│            style = None                                                     │
│         tab_size = 8                                                        │
│            width = 79                                                       │
└─────────────────────────────────────────────────────────────────────────────┘
┌─── <class 'rich._windows.WindowsConsoleFeatures'> ────┐
│ Windows features available.                           │
│                                                       │
│ ┌───────────────────────────────────────────────────┐ │
│ │ WindowsConsoleFeatures(vt=False, truecolor=False) │ │
│ └───────────────────────────────────────────────────┘ │
│                                                       │
│ truecolor = False                                     │
│        vt = False                                     │
└───────────────────────────────────────────────────────┘
┌────── Environment Variables ───────┐
│ {                                  │
│     'TERM': None,                  │
│     'COLORTERM': None,             │
│     'CLICOLOR': None,              │
│     'NO_COLOR': None,              │
│     'TERM_PROGRAM': None,          │
│     'COLUMNS': None,               │
│     'LINES': None,                 │
│     'JUPYTER_COLUMNS': None,       │
│     'JUPYTER_LINES': None,         │
│     'JPY_PARENT_PID': None,        │
│     'VSCODE_VERBOSE_LOGGING': None │
│ }                                  │
└────────────────────────────────────┘
platform="Windows"

[github-actions]

Thank you for your issue. Give us a little time to review it.

PS. You might want to check the FAQ if you haven't done so already.

This is an automated reply, generated by FAQtory

[ncoghlan]

Given that changing the subprocess invocation to explicitly pass -X utf8 to the isolated Python child process eliminated the misebehaviour, it doesn't look like there is anything specific for rich to do here (unless it was to mention -X utf8 and PYTHONUTF8=1 as potential options for improving the handling of redirected standard streams on Windows when invoking rich-enabled applications somewhere in the docs or FAQ).

The long-term fix will come in Python 3.15, when https://peps.python.org/pep-0686/ makes UTF-8 mode an opt-out default, rather than its current opt-in behaviour.

[github-actions]

I hope we solved your problem.

If you like using Rich, you might also enjoy Textual