Debugging a multiprocessing app #80

DonJayamanne · 2017-11-13T22:40:21Z

From @isvoboda on December 31, 2016 1:45

I have a problem debugging a multiprocess based app. A simple example is given, which, in the beginning, runs ok without any breakpoint. On the other hand, stepping to the line val = self.que.get(block=True, timeout=None) crashes the debugger, or just loses a focus and the process is no more attachable by VSCode.
I am enclosing a log from debug console and both launches I have tried.

Environment data

VS Code version: 1.8.1
Python Extension version: 0.5.5
Python Version: 2.7.6
OS and version: XUbuntu 14.04.5
My launch.json

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python",
            "type": "python",
            "request": "launch",
            "stopOnEntry": true,
            "pythonPath": "${config.python.pythonPath}",
            "program": "${file}",
            "cwd": "${workspaceRoot}",
            "debugOptions": [
                "WaitOnAbnormalExit",
                "WaitOnNormalExit",
                "RedirectOutput"
            ]
        },
        {
            "name": "External Terminal/Console",
            "type": "python",
            "request": "launch",
            "stopOnEntry": true,
            "pythonPath": "${config.python.pythonPath}",
            "program": "${file}",
            "cwd": "null",
            "console": "externalTerminal",
            "debugOptions": [
                "WaitOnAbnormalExit",
                "WaitOnNormalExit"
            ]
        }
      ]
}

Your settings.json:

Logs

Output from Python output panel

Traceback (most recent call last):
  File "/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.6/pythonFiles/PythonTools/visualstudio_py_debugger.py", line 1725, in loop
    cmd()
  File "/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.6/pythonFiles/PythonTools/visualstudio_py_debugger.py", line 1741, in command_step_into
    assert thread._is_blocked
AssertionError

Output from Console window (Help->Developer Tools menu)

TypeError: Cannot read property 'Threads' of null
    at PythonDebugger.threadsRequest (/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.6/out/client/debugger/Main.js:321:27)
    at PythonDebugger.DebugSession.dispatchRequest (/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.6/node_modules/vscode-debugadapter/lib/debugSession.js:427:22)
    at PythonDebugger.ProtocolServer._handleData (/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.6/node_modules/vscode-debugadapter/lib/protocol.js:104:38)
    at Socket.<anonymous> (/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.6/node_modules/vscode-debugadapter/lib/protocol.js:24:60)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at Socket.Readable.push (_stream_readable.js:134:10)
    at Pipe.onread (net.js:543:20): Error: TypeError: Cannot read property 'Threads' of null
    at PythonDebugger.threadsRequest (/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.6/out/client/debugger/Main.js:321:27)
    at PythonDebugger.DebugSession.dispatchRequest (/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.6/node_modules/vscode-debugadapter/lib/debugSession.js:427:22)
    at PythonDebugger.ProtocolServer._handleData (/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.6/node_modules/vscode-debugadapter/lib/protocol.js:104:38)
    at Socket.<anonymous> (/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.6/node_modules/vscode-debugadapter/lib/protocol.js:24:60)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at Socket.Readable.push (_stream_readable.js:134:10)
    at Pipe.onread (net.js:543:20)
    at file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:90:21136
    at Object.v [as _notify] (file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:32:13263)
    at Object.enter (file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:32:16768)
    at _run (file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:32:18089)
    at _error (file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:32:17606)
    at file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:85:1178
    at _.e.dispatch (file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:85:2523)
    at _.e.handleData (file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:85:2035)
    at Socket.<anonymous> (file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:85:1038)
    at emitOne (events.js:96:13)e.onUnexpectedError @ shell.ts:541(anonymous function) @ shell.ts:457e.onUnexpectedError @ errors.ts:68o @ errors.ts:88done @ winjs.base.raw.js:1378v @ winjs.base.raw.js:1224enter @ winjs.base.raw.js:901_run @ winjs.base.raw.js:1068_chainedError @ winjs.base.raw.js:1031n @ winjs.base.raw.js:736then @ winjs.base.raw.js:1436enter @ winjs.base.raw.js:747_run @ winjs.base.raw.js:1068_setCompleteValue @ winjs.base.raw.js:1052v @ winjs.base.raw.js:1209enter @ winjs.base.raw.js:901_run @ winjs.base.raw.js:1068_error @ winjs.base.raw.js:1041(anonymous function) @ v8Protocol.ts:53e.dispatch @ v8Protocol.ts:136e.handleData @ v8Protocol.ts:104(anonymous function) @ v8Protocol.ts:41emitOne @ events.js:96emit @ events.js:188readableAddChunk @ _stream_readable.js:176Readable.push @ _stream_readable.js:134onread @ net.js:543

Actual behavior

Stepping the debug process leads to a crash of debugger or loss of process focus.

Expected behavior

Stepping through a program and printing 0.

Steps to reproduce:

Debug the provided code, ie step into the process (reader or writer), the debugger stops working and nothing happens. After that, try to run the debugger once more without stepping through the code. Following these steps I got the error posted above in the developer tools console.

#!/usr/bin/env python

import multiprocessing
import random

class Writer(multiprocessing.Process):
    def __init__(self, que=None, out_pipe=None):
        super(Writer, self).__init__()
        self.daemon = True
        self.que = que

    def run(self):
        while True:
            data = random.uniform(0,1)
            self.que.put(data, block=True, timeout=None)

class Reader(multiprocessing.Process):
    def __init__(self, que=None, in_pipe=None):
        super(Reader, self).__init__()
        self.daemon = True
        self.que = que
        self.n_batch = 24
    
    def run(self):
        data = []
        for _ in xrange(self.n_batch):
            val = self.que.get(block=True, timeout=None)
            print("Received: {}".format(val))

def main():
    que = multiprocessing.Queue(maxsize=8)
    writer = Writer(que=que)
    reader = Reader(que=que)


    writer.start()
    reader.start()
    reader.join()

if __name__ == "__main__":
    main()

Copied from original issue: DonJayamanne/pythonVSCode#614

The text was updated successfully, but these errors were encountered:

DonJayamanne · 2017-11-13T22:40:22Z

From @rangersb on January 26, 2017 19:19

Hey, I have a related Issue.
I use python threading (import threading). VSCode stops at the beginning of my program, thats fine. After creating some threads (4 in total) the breakpoints in the main thread from which I created the other threads are just ignored. Debugging with other editors works...

DonJayamanne · 2017-11-13T22:40:22Z

From @isvoboda on February 2, 2017 21:25

Developer tools console with a new version of VS 1.9 and Python 0.5.7

Debug adapter process has terminated unexpectedly
TypeError: Cannot read property 'Threads' of null
    at PythonDebugger.threadsRequest (/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.7/out/client/debugger/Main.js:328:27)
    at PythonDebugger.DebugSession.dispatchRequest (/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.7/node_modules/vscode-debugadapter/lib/debugSession.js:427:22)
    at PythonDebugger.ProtocolServer._handleData (/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.7/node_modules/vscode-debugadapter/lib/protocol.js:104:38)
    at Socket.<anonymous> (/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.7/node_modules/vscode-debugadapter/lib/protocol.js:24:60)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at Socket.Readable.push (_stream_readable.js:134:10)
    at Pipe.onread (net.js:543:20): Error: TypeError: Cannot read property 'Threads' of null
    at PythonDebugger.threadsRequest (/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.7/out/client/debugger/Main.js:328:27)
    at PythonDebugger.DebugSession.dispatchRequest (/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.7/node_modules/vscode-debugadapter/lib/debugSession.js:427:22)
    at PythonDebugger.ProtocolServer._handleData (/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.7/node_modules/vscode-debugadapter/lib/protocol.js:104:38)
    at Socket.<anonymous> (/home/isvoboda/.vscode/extensions/donjayamanne.python-0.5.7/node_modules/vscode-debugadapter/lib/protocol.js:24:60)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at Socket.Readable.push (_stream_readable.js:134:10)
    at Pipe.onread (net.js:543:20)
    at file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:93:15836
    at Object.v [as _notify] (file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:32:12913)
    at Object.enter (file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:32:16418)
    at _run (file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:32:17739)
    at _error (file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:32:17256)
    at file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:86:24358
    at g.e.dispatch (file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:86:25709)
    at g.e.handleData (file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:86:25217)
    at Socket.<anonymous> (file:////usr/share/code/resources/app/out/vs/workbench/electron-browser/workbench.main.js:86:24218)
    at emitOne (events.js:96:13)

DonJayamanne · 2017-11-13T22:40:23Z

From @zjjott on July 20, 2017 8:17

+1 multiprocess cause like:

Traceback (most recent call last):
  File "/Users/zhutou/.vscode/extensions/donjayamanne.python-0.6.7/pythonFiles/PythonTools/visualstudio_py_debugger.py", line 1729, in loop
    cmd()
  File "/Users/zhutou/.vscode/extensions/donjayamanne.python-0.6.7/pythonFiles/PythonTools/visualstudio_py_debugger.py", line 1762, in command_step_over
    assert thread._is_blocked
AssertionError

and when I stop debugger ,vscode can't kill process group as it used to be

DonJayamanne · 2017-11-13T22:40:23Z

please confirm you are using PTVSD package version 3.0.0

DonJayamanne · 2017-11-13T22:40:24Z

From @isvoboda on July 20, 2017 12:48

@DonJayamanne Actually, I have not been using ptvsd at all. I guess it is not a remote debug related problem? Nevertheless, I have installed ptvsd==3.0 and the problem remains for me.

DonJayamanne · 2017-11-13T22:40:24Z

From @zjjott on July 21, 2017 2:46

@DonJayamanne I just install python extension in vscode,all things is default settings,and I debug a multiprocess tornado applications

DonJayamanne · 2017-11-13T22:40:25Z

From @MattMorgis on July 25, 2017 13:48

+1 - when debugging Snorkel everything works fine, until I run it in parallel. It stops at my first breakpoint, but then freezes and spits out the same output @zjjott posted.

I'm happy to help out in any way if you can point me in the right direction.

DonJayamanne · 2017-11-13T22:40:25Z

From @phucnt1992 on October 18, 2017 3:59

I have same issue, I try to debug a gunicorn + falcon app with attach mode and ptvsd==3.0.0 on Ubuntu OS. the output is the same and the app freezes after that. I know it is very old issue. Is anybody has solution for it?

DonJayamanne · 2017-11-13T22:40:26Z

From @egrgdam on October 30, 2017 15:17

having uwsgi + flask + ptvsd==3.0.0. Same issue.
@DonJayamanne, do you know release date for new ptvsd? (see somewhere stating that it's currently under some heavy change)

DonJayamanne · 2017-11-13T22:40:26Z

@egrgdam Sorry, no eta yet. Hopefully I'll have something within around two weeks.

broven · 2017-12-31T07:50:08Z

same issue

timbess · 2018-02-07T13:45:29Z

+1

diskompo · 2018-03-07T10:03:46Z

I have the same problem using threading.Thread()s.

dmitryzykov · 2018-03-28T13:33:34Z

I have the same issue with gunicorn + flask

nicolinux · 2018-03-28T13:56:36Z

Same issue with a Tornado application and basically everything that's behind/after a IOLoop.current().call_later() call.

DonJayamanne · 2018-09-27T16:53:29Z

Upstream issue microsoft/ptvsd#57

DonJayamanne · 2018-10-09T16:57:44Z

Update to upstream issue microsoft/ptvsd#885

For #80

DonJayamanne added bug labels Nov 13, 2017

DonJayamanne removed bug labels Nov 13, 2017

brettcannon added awaiting 2-PR area-debugging bug Issue identified by VS Code Team member as probable bug labels Nov 14, 2017

DonJayamanne added feature-request Request for new features or functionality and removed bug Issue identified by VS Code Team member as probable bug labels Mar 28, 2018

qubitron mentioned this issue Apr 12, 2018

Add support for multi-process debugging in experimental debugger #943

Closed

ericsnowcurrently added needs upstream fix and removed needs PR labels Sep 19, 2018

DonJayamanne mentioned this issue Sep 27, 2018

Multiprocess debugging support? #314

Closed

brettcannon added this to the 2018, week 40 - October Sprint 1 milestone Oct 3, 2018

DonJayamanne self-assigned this Oct 8, 2018

This was referenced Oct 9, 2018

Enable live reloading of flask and django debug configuration #2326

Closed

Refactor to move debugger files #2850

Merged

DonJayamanne mentioned this issue Oct 17, 2018

Add support for multi proc debugging #2912

Merged

6 tasks

brettcannon modified the milestones: 2018, week 40 - October Sprint 1, 2018, week 42 - October Sprint 2 Oct 17, 2018

DonJayamanne mentioned this issue Oct 18, 2018

Change default behavior for debugging of Flask #2954

Merged

5 tasks

DonJayamanne added the validate fix label Oct 24, 2018

DonJayamanne mentioned this issue Oct 25, 2018

Remove obsolete flags, stop terminating debug processes #3016

Merged

4 tasks

DonJayamanne added a commit that referenced this issue Oct 25, 2018

Remove obsolete flags, stop terminating debug processes (#3016)

55daecf

For #80

brettcannon closed this as completed Oct 30, 2018

lock bot locked as resolved and limited conversation to collaborators Nov 27, 2018

brettcannon removed validate fix labels Dec 13, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Debugging a multiprocessing app #80

Debugging a multiprocessing app #80

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

broven commented Dec 31, 2017

timbess commented Feb 7, 2018

diskompo commented Mar 7, 2018

dmitryzykov commented Mar 28, 2018

nicolinux commented Mar 28, 2018

DonJayamanne commented Sep 27, 2018

DonJayamanne commented Oct 9, 2018

Debugging a multiprocessing app #80

Debugging a multiprocessing app #80

Comments

DonJayamanne commented Nov 13, 2017

Environment data

Logs

Actual behavior

Expected behavior

Steps to reproduce:

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

DonJayamanne commented Nov 13, 2017

broven commented Dec 31, 2017

timbess commented Feb 7, 2018

diskompo commented Mar 7, 2018

dmitryzykov commented Mar 28, 2018

nicolinux commented Mar 28, 2018

DonJayamanne commented Sep 27, 2018

DonJayamanne commented Oct 9, 2018