UTF-16 in tests output on Windows

[serhiy-storchaka]

When test crashes, the report about this is written to output encoded in UTF-16 (or UCS2). In other words, ASCII characters are intermixed with null bytes. The problem is that while the browser simply do not show null bytes, so you can read the message, you cannot copy it, because the content of clipboard is truncated at null bytes.

See for example https://github.com/python/cpython/actions/runs/6094370280/job/16535778004#step:5:1112

A�s�s�e�r�t�i�o�n� �f�a�i�l�e�d�:� �t�s�t�a�t�e�_�i�s�_�a�l�i�v�e�(�t�s�t�a�t�e�)� �&�&� �!�t�s�t�a�t�e�-�>�_�s�t�a�t�u�s�.�b�o�u�n�d�,� �f�i�l�e� �D�:�\�a�\�c�p�y�t�h�o�n�\�c�p�y�t�h�o�n�\�P�y�t�h�o�n�\�p�y�s�t�a�t�e�.�c�,� �l�i�n�e� �2�4�5�

It was copied from the raw logs view. In normal view you do not see any �, but you cannot copy this line past the first A.

[zooba]

This message is coming from the C runtime, so we probably don't have any control over how it displays. At a guess, it's writing directly to the console, which is a UTF-16 API, and that's being poorly handled by the GitHub Actions runner that is intercepting the output.

Unlikely we can do anything other than report it to Microsoft, and they will likely prioritise it very low as it's a debug-only error message that's usually going to appear in a message box.

(Unimportant, but it copies fine for me from the normal view. I guess all the control characters get stripped out, but it's probably a browser difference - I'm on Edge.)

[serhiy-storchaka]

It's probably an OS difference -- I'm on Linux. Clipboard is UTF-16 on Windows and UTF-8 on Linux.

If we cannot do anything with this, close this issue.

[zooba]

I don't know for sure that we can't do anything, I haven't checked. Remind me to check next week when I might have free time, or if someone else knows for sure they can chime in.

[vstinner]

To decrypt this string in Python, you can use [::2] :-)

>>> "A�s�s�e�r�t�i�o�n� �f�a�i�l�e�d�:� �t�s�t�a�t�e�_�i�s�_�a�l�i�v�e�(�t�s�t�a�t�e�)� �&�&� �!�t�s�t�a�t�e�-�>�_�s�t�a�t�u�s�.�b�o�u�n�d�,� �f�i�l�e� �D�:�\�a�\�c�p�y�t�h�o�n�\�c�p�y�t�h�o�n�\�P�y�t�h�o�n�\�p�y�s�t�a�t�e�.�c�,� �l�i�n�e� �2�4�5�"[::2]
'Assertion failed: tstate_is_alive(tstate) && !tstate->_status.bound, file D:\\a\\cpython\\cpython\\Python\\pystate.c, line 245'

[vstinner]

I don't know for sure that we can't do anything

Well. I would suggest closing this issue. The code which logs the assertion error is not under the control of Python.

@zooba: Or do you have an idea on how to control how CRT logs such error message?

[zooba]

I had a look and it tries to output using WriteConsoleW first. The fallback should use regular fwprintf, which knows what it's doing, but presumably the first call succeeds without changing the encoding (because it has no concept of what encoding to use). There's no way to forcibly bypass it, so all we can do here is close this.