debugpy.listen hangs in WSGI app

[mforkel]

Environment data

• debugpy version: 1.3.0
• OS and version: Raspbian 9 (stretch)
• Python version: 3.5.3 (provided by Raspbian)
• Using VS Code or Visual Studio: VS Code 1.56.0
• Web server: Apache/2.4.25 w/ mod_wsgi 4.5.11 (provided by Raspbian)

Actual behavior

WSGI app does not produce result

Expected behavior

WSGI app should produce response

Steps to reproduce:

I am trying to debug this WSGI application

#! /env/python3

def app(environ, start_response):
    import debugpy

    debugpy.log_to('/<myhome>/debugpy-test/logs')
    debugpy.listen(('0.0.0.0', 5678))
    #debugpy.wait_for_client()
    #debugpy.breakpoint()

    status = '200 OK'
    output = b'Hello World!\n'
    response_headers = [('Content-type', 'text/plain'),
                        ('Content-Length', str(len(output)))]
    start_response(status, response_headers)
    return [output]

application = app

with this Apache configuration

Listen 0.0.0.0:45678

<VirtualHost *:45678>
    WSGIScriptAlias / /<myhome>/debugpy-test/wsgi-test.py
    <Directory /<myhome>/debugpy-test>
        Require all granted
    </Directory>
</VirtualHost>

from a client running Windows 10 20H2.

When I comment out the call to debugpy.listen in the WSGI script, I can retrieve the page http://<server>:45678. But with the script as shown above, it hangs. The last entry to the server log is that of pydevd.settrace.

I can start debugging on the client, but then do nothing:

apache2    2934         www-data   20u  IPv4 8572793      0t0  TCP <server>:45678-><client>:65351 (ESTABLISHED)
python3    2994         www-data    4u  IPv4 8572846      0t0  TCP *:5678 (LISTEN)
python3    2994         www-data    5u  IPv4 8579261      0t0  TCP <server>:5678-><client>:51780 (ESTABLISHED)

[fabioz]

Can you attach the debugger logs you got from debugpy.log_to('/<myhome>/debugpy-test/logs')?

[mforkel]

Happily, here they are:
debugpy.adapter-2574.log
debugpy.server-2428.log

[fabioz]

It seems that the connection is not being really completed to connect to the 5678 port.

The way I'd do it is probably creating an ssh tunneling for the 5678 port (the debugger needs a raw socket connection, not really an http communication layer).

[mforkel]

I'm not sure about the semantics of debugpy.listen, but shouldn't it return? I thought that debugpy.wait_for_client might hang if there was a connection problem, but the script never gets to that point (plus, debugpy.wait_for_client is commented out). I surrounded debugpy.listen by printing to stderr, but only the first one produces a result.

In the identical server / client constellation, the following script works fine:

#! /env/python3

import debugpy

debugpy.log_to('/<myhome>/debugpy-test/logs')
debugpy.listen(('0.0.0.0', 5678))
debugpy.wait_for_client()

print("hello, world")

Execution is halted by debug.wait_for_client until I start debugging on the client and a breakpoint set by the client on the last line side is honored. I'll attach the logs.

To me it looks like debugpy.listen inside an WSGI app is behaving differently and might be causing the problem.

debugpy.adapter-3318.log
debugpy.server-3308.log

[fabioz]

You're right, the listen should not block...

Can you try to set the following environment variables:

PYDEVD_LOAD_NATIVE_LIB=0
PYDEVD_USE_CYTHON=0

in the server and then try to use it again to see if it does any difference?

The way that it works is that in: https://github.com/microsoft/debugpy/blob/v1.3.0/src/debugpy/server/api.py#L255 debugpy spawns a process locally (which is the debug adapter process) and then gets the debugger to communicate with it. Apparently, for some reason this internal connection is not being completed (and I'm not sure why -- some possible common causes here would be a firewall blocking the communication or a network card misconfiguration).

It's a bit strange that it doesn't throw any error though... if you're up to it, you could change your local version to add some prints in https://github.com/microsoft/debugpy/blob/v1.3.0/src/debugpy/_vendored/pydevd/pydevd.py#L2708 to know where exactly it's halting...

[mforkel]

The WSGI test app does not hang in listen if the environment variables are set to 0:

#! /env/python3

def app(environ, start_response):
    import os
    import debugpy

    os.environ['PYDEVD_LOAD_NATIVE_LIB'] = '0'
    os.environ['PYDEVD_USE_CYTHON'] = '0'

    debugpy.log_to('/<myhome>/debugpy-test/logs')
    debugpy.listen(('0.0.0.0', 5678))
    #debugpy.wait_for_client()
    #debugpy.breakpoint()

    status = '200 OK'
    output = b'Hello World!\n'
    response_headers = [('Content-Type', 'text/plain'),
                        ('Content-Length', str(len(output)))]
    start_response(status, response_headers)
    return [output]

application = app

Without setting the environment variables, the listen hangs in AttachDebuggerTracing of the successfully loaded library in https://github.com/microsoft/debugpy/blob/v1.3.0/src/debugpy/_vendored/pydevd/pydevd_tracing.py#L268.

This does not happen in the non-WSGI test:

#! /env/python3

#import os
import debugpy

#os.environ['PYDEVD_LOAD_NATIVE_LIB'] = '0'
#os.environ['PYDEVD_USE_CYTHON'] = '0'

debugpy.log_to('/<myhome>/debugpy-test/logs')
debugpy.listen(('0.0.0.0', 5678))
debugpy.wait_for_client()

print("hello, world")

At that point I should probably admit that I use DLLs that I compile for my Debian/Raspbian debugpy package. For Raspbian, the name of my library is attach_linux.cpython-35m-arm-linux-gnueabihf.so.

During my tests, I ran into another problem: After the WSGI test, the adapter process is not terminated. Running the test again thus results in a runtime error Can't listen for client connections: [Errno 98] Address already in use. This seems similar to #338, but I assume it is caused by the hanging listen?

[mforkel]

Sorry, wrong button...

[fabioz]

Given that Raspbian is ARM, you really need to disable the compiled bits with:

os.environ['PYDEVD_LOAD_NATIVE_LIB'] = '0'
os.environ['PYDEVD_USE_CYTHON'] = '0'

I just noticed that the code isn't handling the arm case properly (even if you compile yourself)... I'll fix this.

As for the Can't listen for client connections: [Errno 98] Address already in use, I think it's really probably related to the attach that didn't work.

If you kill all the python processes and start over with those environment variables, do you still have that issue?

[mforkel]

Given that Raspbian is ARM, you really need to disable the compiled bits with:

os.environ['PYDEVD_LOAD_NATIVE_LIB'] = '0'
os.environ['PYDEVD_USE_CYTHON'] = '0'

I think that's what I did in the WSGI test case, I just commented out those lines in the non-WSGI test.

I just noticed that the code isn't handling the arm case properly (even if you compile yourself)... I'll fix this.

If you are referring to the library naming, here is the patch I use (may be not pretty, but works for me):

--- a/src/debugpy/_vendored/pydevd/pydevd_tracing.py
+++ b/src/debugpy/_vendored/pydevd/pydevd_tracing.py
@@ -3,6 +3,7 @@
     ENV_FALSE_LOWER_VALUES, GlobalDebuggerHolder, ForkSafeLock
 from _pydev_imps._pydev_saved_modules import thread, threading
 from _pydev_bundle import pydev_log, pydev_monkey
+from fnmatch import fnmatch
 from os.path import os
 try:
     import ctypes
@@ -143,8 +144,10 @@
             suffix = 'amd64'
         else:
             suffix = 'x86'
-
-        filename = os.path.join(os.path.dirname(__file__), 'pydevd_attach_to_process', 'attach_linux_%s.so' % (suffix,))
+
+        libdir = os.path.join(os.path.dirname(__file__), 'pydevd_attach_to_process')
+        libs = [f for f in os.listdir(libdir) if fnmatch(f, 'attach_linux.*.so')]
+        filename = os.path.join(libdir, libs[0] if len(libs) == 1 else 'attach_linux.%s.so' % (suffix,))

     elif IS_MAC:
         if IS_64BIT_PROCESS:

This works in conjunction with two scripts used in the Debian packaging process. One clears the libraries and all intermediate files at the beginning of the build, and one builds the libraries for the target architecture. The resulting library files are _vendored/pydevd/_pydevd_bundle/pydevd_cython.cpython-35m-arm-linux-gnueabihf.so and _vendored/pydevd/pydevd_attach_to_process/attach_linux.cpython-35m-arm-linux-gnueabihf.so for Raspbian stretch with Python 3.5.

As for the Can't listen for client connections: [Errno 98] Address already in use, I think it's really probably related to the attach that didn't work.

If you kill all the python processes and start over with those environment variables, do you still have that issue?

If I kill the adapter process (/usr/bin/python3 /usr/lib/python3/dist-packages/debugpy/adapter ...) the port is available again.

[fabioz]

This smells like some incompatibility issue in the libraries (maybe it wasn't compiled as it should?).

Do you know which compiler flags were used to compile attach_linux.cpython-35m-arm-linux-gnueabihf.so?

(i.e.: it should follow what's in: compile_linux.sh: https://github.com/microsoft/debugpy/blob/main/src/debugpy/_vendored/pydevd/pydevd_attach_to_process/linux_and_mac/compile_linux.sh).

[fabioz]

Regarding the WSGI case, can you verify if the python script that called the listen the first time is actually exiting?

i.e.: if a python script calls listen, that port should be blocked and you should really only be able to connect again once it exits (so, that'd be expected and it'd be up to you to make sure it's killed before calling listen again in any other instance).

[mforkel]

Do you know which compiler flags were used to compile attach_linux.cpython-35m-arm-linux-gnueabihf.so?

This is how attach_linux.so is build within pbuilder using the pybuild build system:

cd /build/debugpy-1.3.0/.pybuild/pythonX.Y_3.5/build/debugpy/_vendored/pydevd/pydevd_attach_to_process
arm-linux-gnueabihf-g++ -shared -fPIC -nostartfiles -o attach_linux.so linux_and_mac/attach.cpp

It is later renamed to attach_linux.cpython-35m-arm-linux-gnueabihf.so by the build system

[mforkel]

Regarding the WSGI case, can you verify if the python script that called the listen the first time is actually exiting?

Do you mean the case that the extensions are disabled with

os.environ['PYDEVD_LOAD_NATIVE_LIB'] = '0'
os.environ['PYDEVD_USE_CYTHON'] = '0'

If the extensions are enabled and the listen does not return, the script that called it will not exit.