Multiprocessing via debug adapter

[int19h]

The conclusion of the investigation in #1703 is that to deliver the best, most reliable implementation of multiprocessing, we need to switch from the existing design in which the debug server in the root process manages the child processes, to the design in which the debug adapter manages all processes.

In the new design, there is always a single adapter in a multiproc debug session. Debug servers in all processes - both the main process and subprocesses - all connect to that adapter, treating it as a DAP client. On the other side, the IDE establishes multiple connections to the debug adapter, each connection representing a separate process being debugged (at most one of these connections is over stdio, and that only in the "launch" scenario; the rest will be over sockets, using the DebugAdapterServer API in VSCode).

[calebickler]

Any updates? Excited for general release to see if I can use VSCode to debug my Pyramid app.

[karthiknadig]

@calebickler @rickstaa This is done, the python extension is rolling it out in waves. Currently about 20% of the users get the new debugger that supports multiprocessing.

If you want to try this out here is what you can do:

1. Get the insiders version of the extension from https://pvsc.blob.core.windows.net/extension-builds/ms-python-insiders.vsix .
2. Add this to your settings and reload. (we added the ability to opt-into experiments this week hence the need for insiders version)

    "python.experiments.optInto": [
        "DebugAdapterFactory - experiment",
        "PtvsdWheels37 - experiment"
    ]

If everything is setup right, when you debug you should see new_ptvsd in the path that gets printed out in the terminal.

[karthiknadig]

/cc @fabioz The above instructions might be helpful to you too :). Along with this you can add "debugAdapterPath": "C:\\git\\ptvsd\\src\\ptvsd\\adapter" to make the extension use the debug adapter that you are working on.

[rickstaa]

@karthiknadig Thanks a lot for the new debugger it is amazing. Appears to be working with processes spawned by the subprocess and multiprocess modules. Is it also supposed to work with processes spawned by the module and Qthread modules? I tried it out in one of my Pyqt5 scripts and it is not hitting the breakpoints. I will reopen the issue below if the new debugger still is supposed to work with Q threads.

• Related issue

[wpbrown]

@karthiknadig thanks for the update. I've been waiting for this update to debug my scripts that use the sh library which uses os.fork internally. I just tried out the new debugger (on Ubuntu 19.10, latest insiders code and insiders python extension, experiments opted in).

With no arguments to the subprocess, I'm able to step past the sh forks successfully. However with arguments to the subprocess it seems there is a failure in the forked process.

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/will/.vscode-insiders/extensions/ms-python.python-2020.1.57652-dev/pythonFiles/lib/python/new_ptvsd/wheels/ptvsd/__main__.py", line 45, in <module>
    cli.main()
  File "/home/will/.vscode-insiders/extensions/ms-python.python-2020.1.57652-dev/pythonFiles/lib/python/new_ptvsd/wheels/ptvsd/../ptvsd/server/cli.py", line 361, in main
    run()
  File "/home/will/.vscode-insiders/extensions/ms-python.python-2020.1.57652-dev/pythonFiles/lib/python/new_ptvsd/wheels/ptvsd/../ptvsd/server/cli.py", line 203, in run_file
    runpy.run_path(options.target, run_name="__main__")
  File "/usr/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/usr/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/mnt/tesla_warm/online/development/feed2pdf/main.py", line 144, in <module>
    result = str(lsusb('-d', '04f9:'))
  File "/home/will/.local/share/virtualenvs/feed2pdf-2y3uKzzJ/lib/python3.7/site-packages/sh.py", line 1427, in __call__
    return RunningCommand(cmd, call_args, stdin, stdout, stderr)
  File "/home/will/.local/share/virtualenvs/feed2pdf-2y3uKzzJ/lib/python3.7/site-packages/sh.py", line 774, in __init__
    self.wait()
  File "/home/will/.local/share/virtualenvs/feed2pdf-2y3uKzzJ/lib/python3.7/site-packages/sh.py", line 792, in wait
    self.handle_command_exit_code(exit_code)
  File "/home/will/.local/share/virtualenvs/feed2pdf-2y3uKzzJ/lib/python3.7/site-packages/sh.py", line 815, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_1: 

  RAN: /usr/bin/lsusb -d 04f9:

  STDOUT:


  STDERR:
Error patching args
Traceback (most recent call last):
  File "/home/will/.vscode-insiders/extensions/ms-python.python-2020.1.57652-dev/pythonFiles/lib/python/new_ptvsd/wheels/ptvsd/_vendored/pydevd/_pydev_bundle/pydev_monkey.py", line 206, in patch_args
    if is_python(args[0]):
  File "/home/will/.vscode-insiders/extensions/ms-python.python-2020.1.57652-dev/pythonFiles/lib/python/new_ptvsd/wheels/ptvsd/_vendored/pydevd/_pydev_bundle/pydev_monkey.py", line 117, in is_python
    if path.endswith("'") or path.endswith('"'):
TypeError: endswith first arg must be bytes or a tuple of bytes, not str

[fabioz]

Related to that error, on Python 3 the arguments to subprocess must be str, not bytes (it seems the debugger is making it fail inside the debugger -- which should be fixed -- but even if it didn't fail in the debugger it would fail later on subprocess).

i.e.: the code below fails in Python 3.7 due to passing bytes to the subprocess (with or without the debugger).

import subprocess
args = ['python', '-c', 'import sys;print(sys)']
args = [arg.encode('utf-8') for arg in args]
subprocess.call(args)

Error without the debugger:

Traceback (most recent call last):
  File "W:\runtime-pydev.debugger\check\mod2\snippet.py", line 4, in <module>
    subprocess.call(args)
  File "C:\bin\Miniconda\envs\py37_tests\lib\subprocess.py", line 323, in call
    with Popen(*popenargs, **kwargs) as p:
  File "C:\bin\Miniconda\envs\py37_tests\lib\subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "C:\bin\Miniconda\envs\py37_tests\lib\subprocess.py", line 1119, in _execute_child
    args = list2cmdline(args)
  File "C:\bin\Miniconda\envs\py37_tests\lib\subprocess.py", line 530, in list2cmdline
    needquote = (" " in arg) or ("\t" in arg) or not arg
TypeError: a bytes-like object is required, not 'str'

[fabioz]

On a 2nd thought, maybe there's some other function from the os module which does accept bytes and it should be handled... @wpbrown do you know which os function is being called where that error is raised?

You can change the logging call in the method def patch_args(args): in file /home/will/.vscode-insiders/extensions/ms-python.python-2020.1.57652-dev/pythonFiles/lib/python/new_ptvsd/wheels/ptvsd/_vendored/pydevd/_pydev_bundle/pydev_monkey.py (at the end of the patch_args function) to the code below if you're unsure about it:

pydev_log.exception('Error patching args. Caller: %s' % sys._getframe().f_back)

[wpbrown]

@fabioz Indeed. The sh library is using os.fork and os.execv to launch subprocesses. It doesn't use the subprocess library.

Error patching args. Caller: <frame at 0x7f78b415c590, file '/home/will/.vscode-insiders/extensions/ms-python.python-2020.1.57652-dev/pythonFiles/lib/python/new_ptvsd/wheels/ptvsd/_vendored/pydevd/_pydev_bundle/pydev_monkey.py', line 433, code new_execv>

[fabioz]

I'll take a look at that (right after I finish the issue I'm currently working on).

[int19h]

@rickstaa, it is supposed to work for Qt specifically. @fabioz, perhaps there's a regression in the monkey-patching code for PyQt? Or are we not setting pydevd up correctly for that anymore?

In the meantime - and for libraries other than PyQt that start their own threads - you can call ptvsd.debug_this_thread() to explicitly tell the debugger about a thread that it doesn't know about.

[calebickler]

hmm, I tried it with my pyramid app, I can debug main and see it config its self but if I put a breakpoint I in a view the request just hangs forever and I don't hit it

[int19h]

@calebickler Can you please file a separate issue for it, and add debugger logs there?

[brando90]

@rADikal8e7 Can you please file a separate issue for this, so that we can track the fix properly?

Does the version I downloaded have new bugs? Like I cannot set a specific file to run when I call the debugger (see stack overflow Q: https://stackoverflow.com/questions/60215436/how-to-correctly-set-specific-module-to-debug-in-vs-code). Is that related to the error mentioned by @rADikal8e7 ?

[int19h]

@brando90 I don't think it's related to the issue that we're discussing, but it sounds like a bug, and possibly a regression - please file a separate issue. and include your launch.json debug configuration.

[levrik]

@karthiknadig @int19h I installed the Insiders version of the extension, put the experiments into the VS Code settings.json but it's still using ptvsd_launcher.py. Did something change to opt in?
I'm on 2020.3.63446-dev and current stable of VS Code.
Also just tried with VS Code Insiders. It's also using the old ptvsd_launcher.py.

[brando90]

@brando90 I don't think it's related to the issue that we're discussing, but it sounds like a bug, and possibly a regression - please file a separate issue. and include your launch.json debug configuration.

ok done! #2088

[brando90]

is the debugger that supports multiprocessing done/finished?

I want to update all my vs code stuff without having to worry things break (e.g. my jupyter notebook stopped working and unsure how to fix it and thought updating vs code might be a good starting point but Im scared dubugger will break:

https://github.com/microsoft/vscode-python/issues/10168

)

[rADikal8e7]

@int19h After a recent auto update the extension can no longer access the new_ptvsd path. So i keep hitting the error where it accesses the old ptvsd path. I have tried to restore to a dev branch and make sure i opt into the experimental version mentioned in the post above #1706 (comment) with no luck. Reloaded vs-code insiders a bunch of times. Am i missing something basic?

Extension version: ms-python.python-2020.4.67210-dev

remote json settings:

{
    "python.experiments.optInto": [
    "DebugAdapterFactory - experiment",
    "PtvsdWheels37 - experiment"
    ]
}

Error:

            Traceback (most recent call last):
              File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.67210-dev/pythonFiles/lib/python/old_ptvsd/ptvsd/log.py", line 110, in g
                return f(*args, **kwargs)
              File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.67210-dev/pythonFiles/lib/python/old_ptvsd/ptvsd/pydevd_hooks.py", line 74, in start_client
                sock, start_session = daemon.start_client((host, port))
              File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.67210-dev/pythonFiles/lib/python/old_ptvsd/ptvsd/daemon.py", line 214, in start_client
                with self.started():
              File "/usr/local/lib/python2.7/contextlib.py", line 17, in __enter__
                return self.gen.next()
              File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.67210-dev/pythonFiles/lib/python/old_ptvsd/ptvsd/daemon.py", line 110, in started
                self.start()
              File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.67210-dev/pythonFiles/lib/python/old_ptvsd/ptvsd/daemon.py", line 145, in start
                raise RuntimeError('already started')
            RuntimeError: already started

[karthiknadig]

@rADikal8e7 Can you look at Python output panel? It kind of gives you an idea if the setting was picked up.

[rADikal8e7]

@karthiknadig Hmm.. i dont see those in the output. I have pretty minimalistic settings in the remote settings.json file.

Nothing shows up in the console output with regards to the exp group. This was taken after a reload.

[karthiknadig]

I think I know the problem, experiments are disabled if telemetry is disabled. Try turning these on:

[rADikal8e7]

@karthiknadig That did the trick.

Earlier telemetry was turned off in the user settings and enabled in the remote settings. I guess it needs to be enabled in both the settings for it to function as intended. Anyways thanks for the help.

[int19h]

This is done.

[brando90]

@calebickler @rickstaa This is done, the python extension is rolling it out in waves. Currently about 20% of the users get the new debugger that supports multiprocessing.

If you want to try this out here is what you can do:

1. Get the insiders version of the extension from https://pvsc.blob.core.windows.net/extension-builds/ms-python-insiders.vsix .
2. Add this to your settings and reload. (we added the ability to opt-into experiments this week hence the need for insiders version)

    "python.experiments.optInto": [
        "DebugAdapterFactory - experiment",
        "PtvsdWheels37 - experiment"
    ]

If everything is setup right, when you debug you should see new_ptvsd in the path that gets printed out in the terminal.

I want to have a version of the debugger that always works with pytorch (it's multitrheading thing). How do I do that?

[brando90]

@calebickler @rickstaa This is done, the python extension is rolling it out in waves. Currently about 20% of the users get the new debugger that supports multiprocessing.

If you want to try this out here is what you can do:

1. Get the insiders version of the extension from https://pvsc.blob.core.windows.net/extension-builds/ms-python-insiders.vsix .
2. Add this to your settings and reload. (we added the ability to opt-into experiments this week hence the need for insiders version)

    "python.experiments.optInto": [
        "DebugAdapterFactory - experiment",
        "PtvsdWheels37 - experiment"
    ]

If everything is setup right, when you debug you should see new_ptvsd in the path that gets printed out in the terminal.

where is this? How do I get into this window?

[brando90]

@calebickler @rickstaa This is done, the python extension is rolling it out in waves. Currently about 20% of the users get the new debugger that supports multiprocessing.

If you want to try this out here is what you can do:

1. Get the insiders version of the extension from https://pvsc.blob.core.windows.net/extension-builds/ms-python-insiders.vsix .
2. Add this to your settings and reload. (we added the ability to opt-into experiments this week hence the need for insiders version)

    "python.experiments.optInto": [
        "DebugAdapterFactory - experiment",
        "PtvsdWheels37 - experiment"
    ]

If everything is setup right, when you debug you should see new_ptvsd in the path that gets printed out in the terminal.

How does one reload? Your comments need to have more details for non developers to have them be useful.

[brando90]

@calebickler @rickstaa This is done, the python extension is rolling it out in waves. Currently about 20% of the users get the new debugger that supports multiprocessing.

If you want to try this out here is what you can do:

1. Get the insiders version of the extension from https://pvsc.blob.core.windows.net/extension-builds/ms-python-insiders.vsix .
2. Add this to your settings and reload. (we added the ability to opt-into experiments this week hence the need for insiders version)

    "python.experiments.optInto": [
        "DebugAdapterFactory - experiment",
        "PtvsdWheels37 - experiment"
    ]

If everything is setup right, when you debug you should see new_ptvsd in the path that gets printed out in the terminal.

Where is this new path thing that should be printed out? Can you be more precise? Is it in the terminal? Debug console, output problems....?

[brando90]

For now this works but I do not want to have to change settings of vscode. I want vscode to work and not have me think about the editor at all.

---

Current solution:

1. Open the setting.json file via command pellete with Cmd+shift+P and type settings.json and open it.
2. Then add this:

    "DebugAdapterFactory - experiment",
    "PtvsdWheels37 - experiment"
]

3. To reload close and open VSCODE

[cpvandehey]

Related:

I am trying to set up remote debugging with local k8s. I have the ptvsd debugger working for my gRPC apps that are inherently single process with multiple threads. I am wrestling with the python vscode attachment version v2020.7.91728-dev (internal build) and ptvsd==5.0.0a12 with my http apps that are built on top of the gunicorn framework that DOES use multiple processes. I should also mention that I have setup the ptvsdWheels47 and DebugAdapterFactory settings in my launch.json (see below for all of it).

Strangely, I can attach the debugger and all of my processes show up in vcode AND they even stop at the break point that I set:

Oddly, VScode does not show any of the variable information, does not show what line it stopped at in the IDE itself and does not offer any other data related to the stopping point. With that said, I can continue and see various print statement show up in my logs by stepping/continuing through the code (just a single print statement is all):

Would anyone have an idea of why I can actually debug my application without getting variable/stack information and having my IDE show where I am stopped? Very strange.

Launch.json:

{
    "version": "0.2.0",
    "python.experiments.optInto": [
        "DebugAdapterFactory - experiment",
        "PtvsdWheels37 - experiment"
    ],
    "configurations": [
        {
            "name": "Python: Remote Attach",
            "type": "python",
            "request": "attach",
            "port": 5678,
            "host": "localhost",
            "subProcess": true,
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}",
                    "remoteRoot": "/app"
                }
            ],
        },
    ]
}

[int19h]

@cpvandehey First and foremost, you need to switch to https://github.com/microsoft/debugpy.

If that still happens after doing so, please file a bug report in that repo.

[cpvandehey]

@int19hl, how come ptvsd isn't marked as deprecated on the github repo/readme? When is debugpy going to leave beta?

--

It did fix my issue (thanks) and I am now trying to push this update downstream to different projects e.g. I opened up a skaffold issue

[int19h]

There's an open PR to mark it deprecated, but we're holding off on that until we ship a version of vscode-python that no longer includes ptvsd. The most recent version has both in an A/B experiment, although it's at 100% now (i.e. you get debugpy unless you explicitly opt out). The next stable release of vscode-python will only have debugpy with no way to opt out, and the version that ships with it will be debugpy 1.0.0. For now, we keep it in beta to allow us to iterate rapidly, while making sure that pip install and pip install -U always fetch the most recent version.