IDLE: Colorizer and output tests hang on macOS

[terryjreedy]

BPO	44330
Nosy	@terryjreedy, @taleinat, @ned-deily

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-06-07.01:41:11.947>
labels = ['3.9', '3.10', '3.11']
title = 'IDLE: Colorizer and output tests hang on macOS'
updated_at = <Date 2021-06-07.22:11:14.604>
user = 'https://github.com/terryjreedy'

bugs.python.org fields:

activity = <Date 2021-06-07.22:11:14.604>
actor = 'terry.reedy'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2021-06-07.01:41:11.947>
creator = 'terry.reedy'
dependencies = []
files = []
hgrepos = []
issue_num = 44330
keywords = []
message_count = 3.0
messages = ['395239', '395285', '395290']
nosy_count = 3.0
nosy_names = ['terry.reedy', 'taleinat', 'ned.deily']
pr_nums = []
priority = 'critical'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue44330'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-88496: Fix IDLE test hang on macOS #104025
• [3.11] gh-88496: Fix IDLE test hang on macOS (GH-104025) #104027
• gh-88496: IDLE - fix another test on macOS #104075
• [3.11] gh-88496: IDLE - fix another test on macOS (GH-104075) #104076

[terryjreedy]

Ned, have you seen or heard anything about tkinter/tk8.6.11 freezing on macOS? It is happening in many places.

On my MacBook, python3.9 -m test -ugui -v test_idle runs OK in about 10 seconds. With installed python3.10.0b2, it runs test_colorizer ColorCongigTest, ColorDelagatorInstantionTest, and ColorDelagatorTest.test_LoadTagDefs. It hangs on test_case_soft_keyword, which is new in 3.10 for the new match statement soft keywords. With added skips, it hangs on
test_def_statement
test_incremental_editing
test_match_soft_keyword
test_recolorize_main
test_removecolors
and then test_colorizer passes.

At this point, I thought the problem was the fancy new test methods and helpers. But running test_idle again, test_outwin.OutputWindowTest.test_goto_file_line (line 105) hangs displaying a blank window. The test writes something at 115 which did not appear. However, debug prints indicate that it is the 2nd write, about 121, that does not return. The write method just inserts into the text widget. Nothing apparently fancy. The failure is deterministic (over 10 times). In 3.9.5, the same test and all of test_idle ran OK 3 times.

Adding .update() and .update_idletasks() before the write did nothing. Making this test method fail before the write, let the test continue until it hung in test_write.

The problem seems to be in the test environment. Tests of match, case, and _ in Shell looked corrected. Grepping idlelib for 'tkinter' returned 222 hits and multiple gotos for different files and lines within a file worked normally.

[ned-deily]

I can reproduce test_idle hanging on all of the current python.org macOS universal2 variants (3.8.10, 3.9.5, 3.10.0b2) which use Tk 8.6.11 but not with the legacy 10.9 variants for 3.8.10 and 3.9.5 which use Tk 8.6.8. I've tried it on a few older systems, as far back as 10.9, and they all seem to hang the same way. For 3.9.5 (and 3.8.10 now in security-fix mode), the first hang seems to be in test_removecolors which I need to Ctrl-C out of:

/usr/local/bin/python3.9 -m test -v -uall test_idle
[...]
test_insert (idlelib.idle_test.test_colorizer.ColorDelegatorTest) ... ok
test_notify_range (idlelib.idle_test.test_colorizer.ColorDelegatorTest) ... ok
test_recolorize (idlelib.idle_test.test_colorizer.ColorDelegatorTest) ... ok
test_recolorize_main (idlelib.idle_test.test_colorizer.ColorDelegatorTest) ... ok
test_removecolors (idlelib.idle_test.test_colorizer.ColorDelegatorTest) ...

I also see the hangs with current MacPorts Pythons linked with their build of Tk 8.6.11.

[terryjreedy]

Yes, I have the 'old' 3.9.5. In IDLE, ^C does not break the hang, but Restart Shell does. As I expected, I had to skip test_outwin's test_write and test_writelines to stop that test hanging.

Next failure: test_sidebar.LineNumbersTest ends with test_toggle_linenumbering ... OK. That is the last test method and the hang is in tearDownClass "cls.root.update()". Changing to .update_idletasks() fixes this. I believe that the latter is what we usually use before destroying root. But still, update should not crash. At worst, a TclError.

With this fix, test_idle passes. So do test_tcl and test_ttk_textonly. Test_tk and test_ttk_guionly each have 1 failure. (For me) The test suite as a whole is a mess, with multiple Python-crashed message boxes.

[chrstphrchvz]

I also encountered the hang in test_case_soft_keyword. I believe it occurs during this update():

cpython/Lib/idlelib/colorizer.py

Line 313 in c986412

self.update()

Can this update() be replaced with update_idletasks()? I find that doing so allows the test to pass on Aqua. Tcl/Tk has generally discouraged using update, at least where update idletasks will suffice. And update on Tk Aqua (Cocoa) has been prone to issues.

[chrstphrchvz]

I also encounter the hang in test_outwin.OutputWindowTest.test_goto_file_line due to this update:

cpython/Lib/idlelib/outwin.py

Line 115 in c986412

self.text.update()

Changing it to update_idletasks() allows this test to pass here as well.

[terryjreedy]

Both of those .update() calls are from the original 2000 idlelib commit. There are 13 others (not all from 2000 ;-). (Possibly one or two from a locally defined update function.) I am not sure that I am completely clear on when to use which. Feel free to check others.

The output update is there to slow down the display to make each line visible and then scroll. When recursively grepping 'editor' the 295 lines take several seconds instead displaying the entire output in a fraction of a second. CommandPrompt displays 300 lines in about the same time. Maybe I will delete instead of replace, or consider for later.

The colorizer update() is preceded by this:

                if not ok:
                    # We're in an inconsistent state, and the call to
                    # update may tell us to stop.  It may also change
                    # the correct value for "next" (since this is a
                    # line.col string, not a true mark).  So leave a
                    # crumb telling the next invocation to resume here
                    # in case update tells us to leave.
                    self.tag_add("TODO", next)

How? No evidence of the claims, no test case to see if still true 23 years later. It turns out that either update causes color to be erased after a font face, size, or boldness changes. Comes back faster with update_idletasks. With neither, I don't see it disappear. I need to test behavior while typing or scrolling, but too late today.

(Note to myself: branch idletasks.)

[terryjreedy]

More is needed for test_idle to pass on my 'Air' with Catalina. I added another '_idletasks' in test_sidebar to fix another hang, but an 'is' test is failing in test_mousewheel. I may just skip on mac. (branch update2.)