IDLE 3.x on Windows exits when pasting non-BMP unicode

[jbvsmo]

BPO	13153
Nosy	@terryjreedy, @vstinner, @taleinat, @ned-deily, @ezio-melotti, @aivarannamaa, @serhiy-storchaka, @animalize, @miss-islington
PRs	bpo-13153: Test IDLE Windows no-color astral paste fix #16363 bpo-13153: _tkinter part of tkinter_pythoncmd_args_2 for paste fix #16365 bpo-13153: Use OS native encoding for converting between Python and Tcl. #16545 [3.8] bpo-13153: Use OS native encoding for converting between Python and Tcl. (GH-16545) #16580 [3.7] bpo-13153: Use OS native encoding for converting between Python and Tcl. (GH-16545) #16581
Files	tkinter_nobmp_error.patch tkinter_string_conv_3.patch tkinter_pythoncmd_args.patch tkinter_pythoncmd_args_2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2019-10-04.11:44:01.512>
created_at = <Date 2011-10-11.20:01:32.297>
labels = ['type-bug', 'expert-tkinter', '3.9', '3.8', 'expert-IDLE', '3.7', 'expert-unicode']
title = 'IDLE 3.x on Windows exits when pasting non-BMP unicode'
updated_at = <Date 2019-12-20.12:13:44.282>
user = 'https://github.com/jbvsmo'

bugs.python.org fields:

activity = <Date 2019-12-20.12:13:44.282>
actor = 'terry.reedy'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2019-10-04.11:44:01.512>
closer = 'serhiy.storchaka'
components = ['IDLE', 'Tkinter', 'Unicode']
creation = <Date 2011-10-11.20:01:32.297>
creator = 'JBernardo'
dependencies = []
files = ['28376', '31610', '33318', '34016']
hgrepos = []
issue_num = 13153
keywords = ['patch']
message_count = 77.0
messages = ['145363', '145366', '145369', '145573', '145580', '145581', '145584', '145585', '145605', '145607', '145611', '145616', '145635', '155799', '155801', '155810', '155814', '155857', '177750', '177778', '177812', '179276', '182106', '182130', '182141', '182166', '182172', '182180', '182181', '182184', '182207', '182305', '182312', '192240', '192243', '192260', '193566', '194564', '194586', '194603', '194623', '196958', '196993', '201457', '207370', '207371', '207381', '210786', '254165', '254170', '318862', '319247', '348153', '352661', '353083', '353123', '353152', '353761', '353765', '353766', '353767', '353769', '353812', '353833', '353848', '353897', '353898', '353901', '353903', '353910', '353916', '353918', '353919', '353937', '353945', '358701', '358704']
nosy_count = 9.0
nosy_names = ['terry.reedy', 'vstinner', 'taleinat', 'ned.deily', 'ezio.melotti', 'Aivar.Annamaa', 'serhiy.storchaka', 'malin', 'miss-islington']
pr_nums = ['16363', '16365', '16545', '16580', '16581']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13153'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[jbvsmo]

I was playing with some unicode chars on Python 3.2 (x64 on Windows 7), but when pasted a char bigger than 0xFFFF, IDLE crashes without any error message.

Example (works fine):
>>> '\U000104a2'
'𐒢'

But, if I try to paste the above char, the window will instantly close.

The interpreter uses 2-bytes per char (UTF-16) and I don't know if that's causing the problem (as side note, why don't the default Windows build uses 4-bytes char?).

I can't check now with my Ubuntu install (UTF-32) if the problem persists.

[ned-deily]

This is related to bpo-12342. The problem is that Tcl/Tk 8.5 (and earlier) do not support Unicode code points outside the BMP range as in this example. So IDLE will be unable to display such characters but it should not crash either.

[jbvsmo]

@ned

That looks like a bit different case. IDLE *can* print the char after you entered the '\Uxxxxxxxx' version of it.

It doesn't accept you to paste those caracters...

[terryjreedy]

The current Windows build used 2-byte unicode chars because that is what Windows does. In 3.3, all builds will use a new unicode implementation that uses 1,2,or4 bytes as needed. But I suspect we will still have the paste problem unless we can somehow bypass the tk limitation.

Printing a Python string to the screen does not seem to involve conversion to a tk string. Or else tk blindly copies surrogate pairs to Windows even though it cannot create them.

In any case, true window-closing crashes (as opposed to an error traceback) are obnoxious bugs that we try to fix if possible. I verified this on my 64-bit Win 7 system. Thanks for the report. Feel free to look into the code if you can.

[jbvsmo]

Just for comparison, on Python 2.7.1 (x32 on Windows 7) it's possible to paste the char (but can't use it) and a nice error is given.

>>> u'𐒢'
Unsupported characters in input

So the problem was partially solved but something might have happened with the 3.x port...

Searching on both source codes, I can see the following block was commented on Python3.2 but not on Python2.7 (Maybe someone removed someone else's bug fix?) and an assert was added.

#--- Lines 605 to 613 of PyShell.py

assert isinstance(source, str)
#                       v-- on Python2.7 it is types.UnicodeType instead
#if isinstance(source, str):
#    from idlelib import IOBinding
#    try:
#        source = source.encode(IOBinding.encoding)
#    except UnicodeError:
#        self.tkconsole.resetoutput()
#        self.write("Unsupported characters in input\n")
#        return

I uncommented those lines, removed the assert and deleted pycache for fresh bytecode but the error keeps happening.

This function runsource() is only called after the return key is pressed so the bug was introduced on another part of the program.

I'll search further but it's hard to do that without traceback of the error.

(Maybe runit() is the problem because it seems to build the line and call runsource(line))

------
PS: @terry Reedy
That looks nice to have different lengths for chars but what will be the impact on performance? Indexing will still be in constant time?

[jbvsmo]

Just to complete my monologue:
Here's the traceback from running IDLE in cmd line.

C:\Python32\Lib\idlelib>python -i idle.py
Traceback (most recent call last):
  File "idle.py", line 11, in <module>
    idlelib.PyShell.main()
  File "C:\Python32\Lib\idlelib\PyShell.py", line 1429, in main
    root.mainloop()
  File "C:\Python32\Lib\tkinter\__init__.py", line 1009, in mainloop
    self.tk.mainloop(n)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1-2: invalid continuation byte

Not much meaningful but is better than nothing... Looks like some traceback is missing, and this one points to tkinter.

[terryjreedy]

[Yes, indexing will still be O(1), though I personally consider that less important than most make it to be. Consistency across platforms and total time and space performance of typical apps should be the concern. There is ongoing work on improving the new implementation. Some operations already take less space and run faster.]

The traceback may very well be helpful. It implies that copying a supplemental char does not produce proper utf-8 encoded bytes. Or if it does, tkinter (or tk underneath it) does not recognize them. But then the problem should be the initial byte, not the continuation bytes, which are the same for all chars and which all have 10 for their two high order bits. See
https://secure.wikimedia.org/wikipedia/en/wiki/Utf-8
for a fuller explanation.

Line 1009 is the definition of Misc.mainloop(). I believe self.tk represents the embedded tcl interpreter, which is a black box from Python's viewpoint. Perhaps we should wrap the call with

try:
self.tk.mainloop(n)
except Exception as e:
<print error message with all info attached to e before exiting>

This should catch any miscellaneous crashes which are not otherwise caught and maybe turn the crash issues into bug reports -- the same way that running from the command line did. (It will still be good to catch what we can at error sites and give better, more specific messages.) (What I am not familiar with is how the command line interpreter might turn a tcl error into a python exception and why IDLE does not.)

When I copy '𐒢' and paste into the command line interpreter or Notepad++, I get '??'. I am guessing that ?? represent a surrogate pair and that Windows separately encodes each. The result would be 'illegal' utf-8 with an illegal continuation chars. An application can choose to decode the 'illegal' utf-8 -- or not. Python can when "errors='surrogate_escape" (or something like that) is specified. It might be possible to access the raw undecoded bytes of the clipboard with the third party pythonwin module. I do not know if there is anyway to do so with tk.

I wonder if tcl is calling back to Python for decoding and whether there was a change in the default for errors or the callback specification that would explain a change from 2.7 to 3.2.

Ezio, do you know anything about these speculations?

[ned-deily]

Thanks for the additional investigation. You don't see more in the traceback because the exception is occurring in the _tkinter C glue layer. I am able to reproduce the problem on some other platforms as well (e.g. Python 3.x on OS X with Carbon Tk 8.4). More later.