Crash: ModuleNotFoundError: No module named 'pydevd_file_utils'

[zhaoye]

Environment data

• VS Code version: 1.52.1
• Extension version (available under the Extensions sidebar): v2021.1.502429796
• OS and version: macOS 10.15.7
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.9.1 installed by homebrew
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): venv
• Relevant/affected Python packages and their versions: lyrebird
• Relevant/affected Python-related VS Code extensions and their versions: -
• Value of the python.languageServer setting: Pylance v2021.2.0

[NOTE: If you suspect that your issue is related to the Microsoft Python Language Server (python.languageServer: 'Microsoft'), please download our new language server Pylance from the VS Code marketplace to see if that fixes your issue]

Expected behaviour

I can debug my project

Actual behaviour

Got a crash: ModuleNotFoundError: No module named 'pydevd_file_utils'

Steps to reproduce:

1. Create python env and then set vscode to use it
   python3.9 -m venv venv

2. Coding

import venv
from pathlib import Path

venv.main(args=[
    '--clear',
    str(Path(f'~/work/tmp/lb').expanduser().absolute())
])

3. Run debug with .vscode/launch.json

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "py:File",
            "type": "python",
            "request": "launch",
            "program": "${file}"
        }
    ]
}

4. Got a exception
   

Terminal log

``` cd /Users/zhaoye/work/stash/playground ; /usr/bin/env /usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/bin/python3.9 /Users/zhaoye/.vscode/extensions/ms-python.python-2021.1.502429796/pythonFiles/lib/python/debugpy/launcher 55494 -- /Users/zhaoye/work/stash/playground/pydevbug/run.py Traceback (most recent call last): File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/zhaoye/.vscode/extensions/ms-python.python-2021.1.502429796/pythonFiles/lib/python/debugpy/__main__.py", line 45, in cli.main() File "/Users/zhaoye/.vscode/extensions/ms-python.python-2021.1.502429796/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main run() File "/Users/zhaoye/.vscode/extensions/ms-python.python-2021.1.502429796/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file runpy.run_path(target_as_str, run_name=compat.force_str("__main__")) File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 268, in run_path return _run_module_code(code, init_globals, run_name, File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/zhaoye/work/stash/playground/pydevbug/run.py", line 4, in venv.main(args=[ File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/venv/__init__.py", line 493, in main builder.create(d) File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/venv/__init__.py", line 75, in create self._setup_pip(context) File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/venv/__init__.py", line 300, in _setup_pip subprocess.check_output(cmd, stderr=subprocess.STDOUT) File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 420, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 524, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/Users/zhaoye/work/tmp/lb/bin/python3.9', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1. ```

5. Then change venv source code , let it print more info
   

Venv code

    def _setup_pip(self, context):
        """Installs or upgrades pip in a virtual environment"""
        # We run ensurepip in isolated mode to avoid side effects from
        # environment vars, the current directory and anything else
        # intended for the global Python environment
        cmd = [context.env_exe, '-Im', 'ensurepip', '--upgrade',
                                                    '--default-pip']
        # ----
        # Add print here for view more error info
        p = subprocess.run(cmd)
        print(p.returncode)
        # ----

        subprocess.check_output(cmd, stderr=subprocess.STDOUT)

6. Run debug again
   Got this error message : ModuleNotFoundError: No module named 'pydevd_file_utils'
   

Terminal log

cd /Users/zhaoye/work/stash/playground ; /usr/bin/env /usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/bin/python3.9 /Users/zhaoye/.vscode/extensions/ms-python.python-2021.1.502429796/pythonFiles/lib/python/debugpy/launcher 55665 -- /Users/zhaoye/work/stash/playground/pydevbug/run.py 
Traceback (most recent call last):
  File "/Users/zhaoye/.vscode/extensions/ms-python.python-2021.1.502429796/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 20, in <module>
    import pydevd_file_utils
ModuleNotFoundError: No module named 'pydevd_file_utils'
1
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/zhaoye/.vscode/extensions/ms-python.python-2021.1.502429796/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/Users/zhaoye/.vscode/extensions/ms-python.python-2021.1.502429796/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/Users/zhaoye/.vscode/extensions/ms-python.python-2021.1.502429796/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
    runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
  File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 268, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/zhaoye/work/stash/playground/pydevbug/run.py", line 4, in <module>
    venv.main(args=[
  File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/venv/__init__.py", line 497, in main
    builder.create(d)
  File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/venv/__init__.py", line 75, in create
    self._setup_pip(context)
  File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/venv/__init__.py", line 304, in _setup_pip
    subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/zhaoye/work/tmp/lb/bin/python3.9', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

[int19h]

This is an interesting edge case! What's happening here is that venv is creating a subprocess, and the debugger is trying to intercept process creation and inject itself in there. But it looks like there's a conflict between venv and pydevd both trying to adjust PYTHONPATH here, and venv "wins" and effectively erases pydevd from the path?

The immediate workaround is to disable subprocess debugging:

"subProcess": false

I'm not sure if there's anything we can do here otherwise, though. @fabioz?

[zhaoye]

Thanks for reply. @int19h
Is there any way to let pydevd work in venv's PYTHONPATH?

[fabioz]

I'm not sure if there's anything we can do here otherwise, though. @fabioz?

I think that in pydevd it should be possible to catch that ImportError and automatically do sys.path.append(os.path.dirname(os.path.abspath(__file__))) and do the import again.