Importing certain modules while debugging raises an exception

[piscvauyahoofr]

BPO	34609
Nosy	@brettcannon, @terryjreedy, @ncoghlan, @pitrou, @taleinat, @ericsnowcurrently, @pppery, @csabella, @tirkarthi
Superseder	bpo-33065: IDLE debugger: failure stepping through module loading
Files	test_main.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-09-23.18:14:03.033>
created_at = <Date 2018-09-07.18:30:27.399>
labels = ['type-bug', '3.8']
title = 'Importing certain modules while debugging raises an exception'
updated_at = <Date 2018-09-29.10:24:57.937>
user = 'https://bugs.python.org/piscvauyahoofr'

bugs.python.org fields:

activity = <Date 2018-09-29.10:24:57.937>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2018-09-23.18:14:03.033>
closer = 'terry.reedy'
components = []
creation = <Date 2018-09-07.18:30:27.399>
creator = 'piscvau@yahoo.fr'
dependencies = []
files = ['47790']
hgrepos = []
issue_num = 34609
keywords = []
message_count = 10.0
messages = ['324779', '324788', '324798', '324814', '324815', '324953', '326158', '326168', '326664', '326674']
nosy_count = 10.0
nosy_names = ['brett.cannon', 'terry.reedy', 'ncoghlan', 'pitrou', 'taleinat', 'eric.snow', 'ppperry', 'cheryl.sabella', 'xtreak', 'piscvau@yahoo.fr']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '33065'
type = 'behavior'
url = 'https://bugs.python.org/issue34609'
versions = ['Python 3.6', 'Python 3.8']

[piscvauyahoofr]

Idle is unable to import unittest. and raises an attributeError.
Output of the Python shell Idle is included in the file bug idle.

several files raise this exception
on of them is included in this report
the Trace for Idle is as follows :
Python 3.6.5 (default, Apr  1 2018, 05:46:30) 
[GCC 7.3.0] on linux
Type "copyright", "credits" or "license()" for more information.
>>> 
[DEBUG ON]
>>> 
 RESTART: /home/odile/Documents/dev_logiciels/pyhton/openclassrooms/labyrinthe/roboc2/test/test_carte.py 
Traceback (most recent call last):
  File "/home/odile/Documents/dev_logiciels/pyhton/openclassrooms/labyrinthe/roboc2/test/test_carte.py", line 5, in <module>
    import unittest
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 148, in __enter__
  File "<frozen importlib._bootstrap>", line 174, in _get_module_lock
  File "<frozen importlib._bootstrap>", line 59, in __init__
  File "<frozen importlib._bootstrap>", line 59, in __init__
  File "/usr/lib/python3.6/bdb.py", line 51, in trace_dispatch
    return self.dispatch_line(frame)
  File "/usr/lib/python3.6/bdb.py", line 69, in dispatch_line
    self.user_line(frame)
  File "/usr/lib/python3.6/idlelib/debugger.py", line 24, in user_line
    self.gui.interaction(message, frame)
AttributeError: '_ModuleLock' object has no attribute 'name'
[DEBUG ON]

Idle version is 3.6.5

[terryjreedy]

I cannot confirm this. I tested 3.6.6 on Windows 10 with tem3.py containing

import unittest
print unittest

I ran this twice, without and with debug on. The output:

======================== RESTART: F:\Python\a\tem3.py ========================

<module 'unittest' from 'C:\\Programs\\Python36\\lib\\unittest\\__init__.py'>
>>> 
[DEBUG ON]
>>> 
======================== RESTART: F:\Python\a\tem3.py 

========================
<module 'unittest' from 'C:\\Programs\\Python36\\lib\\unittest\\init.py'>
[DEBUG ON]

>>

On the second run, I hit the [Go] button on Debug Control. The minimal program above needs to be tested with 3.6.6 or 3.7.0 on Linux.

[taleinat]

Testing similarly to Terry, this doesn't reproduce with a recent build of master branch (3.8) on Ubuntu 16.04.

I'll be able to try a build of older versions with IDLE only in a few days.

[csabella]

I have Python 3.6.5 on Ubuntu 18.04. Initially, I was unable to recreate the issue. However, bpo-33065 from 4 months ago has the same trace (except on a different import), so I tried stepping through debug using the minimal example that Terry gave and successfully reproduced the trace.

I then tried a new build of 3.8 on Ubuntu and was also able to reproduce by using 'step' while in debug.

I also was able to recreate using 'step' under 3.6.3 on Windows 7.

Note: At one point while stepping, the debug window shows a KeyError on unittest. You'll see it yourself, but it is interesting.

[csabella]

Just to help track this down, 'import math' does not cause the trace, but 'import email' does. It seems to be modules that have a loader function. My guess (but I haven't tried it yet) is that this would happen if stepping through pdb outside of IDLE too.

[csabella]

Following up, I was able to recreate under 3.8 using pdb in powershell, so it's not just in IDLE.

Traceback (most recent call last):
  File "N:\projects\cpython\lib\pdb.py", line 1697, in main
    pdb._runscript(mainpyfile)
  File "N:\projects\cpython\lib\pdb.py", line 1566, in _runscript
    self.run(statement)
  File "N:\projects\cpython\lib\bdb.py", line 586, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "c:\users\onedrive\desktop\step.py", line 1, in <module>
    import unittest
  File "<frozen importlib._bootstrap>", line 980, in _find_and_load
  File "<frozen importlib._bootstrap>", line 149, in __enter__
  File "<frozen importlib._bootstrap>", line 101, in acquire
KeyError: 14104

[pppery]

The Pdb bug and the IDLE bug are unrelated to each other:

Pdb fails because it is attempting to import the readline module every time its trace_dispatch is called, and the import implementation is not reentrant in that way.

IDLE is crashing because the uninitialized _ModuleLock instance raised an error when repr is called, and the IDLE debugger lacks code to catch that exception.

[terryjreedy]

The IDLE issue here is a duplicate of bpo-33065, where the OP said that only step, not go, created the problem. ppperry, thank you for the solution.

Cheryl, please open a new issue for the pdb bug with a little more detail and ppperry's comment.

[brettcannon]

The failure line that Cheryl found in her traceback is at https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap.py#L101 .

Maybe Antoine has an idea since he did a bunch of work on import locking?

[pitrou]

I think pperry nailed it above:

Pdb fails because it is attempting to import the readline module every time its trace_dispatch is called, and the import implementation is not reentrant in that way.

More precisely, _ModuleLock.acquire() in https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap.py#L101 is not reentrant. If pdb steps into that function and tries to call it again by making an "import" call, _blocking_on[tid] will be overwritten and then deleted inside the nested call, so del _blocking_on in the enclosing call will raise a KeyError.

I think the solution would be either one of:

1. pdb avoids doing anything import-related as part of its step function
2. pdb avoids stepping inside importlib internals (e.g. by blacklisting importlib modules)

[exarkun]

Following up, I was able to recreate under 3.8 using pdb in powershell, so it's not just in IDLE.

Traceback (most recent call last):
  File "N:\projects\cpython\lib\pdb.py", line 1697, in main
    pdb._runscript(mainpyfile)
  File "N:\projects\cpython\lib\pdb.py", line 1566, in _runscript
    self.run(statement)
  File "N:\projects\cpython\lib\bdb.py", line 586, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "c:\users\onedrive\desktop\step.py", line 1, in <module>
    import unittest
  File "<frozen importlib._bootstrap>", line 980, in _find_and_load
  File "<frozen importlib._bootstrap>", line 149, in __enter__
  File "<frozen importlib._bootstrap>", line 101, in acquire
KeyError: 14104

This is the same as #91351