Investigate: Unable to find threadStateIndex for the current thread. curPyThread: 0x7fd48ec095c0

[fabioz]

@int19h @karthiknadig

I've seen this fail a couple of times already.

The effect is that we can't set the tracing to threads which aren't current when this happens.

The related code is in:
https://github.com/microsoft/ptvsd/blob/master/src/ptvsd/_vendored/pydevd/pydevd_attach_to_process/common/py_settrace.hpp#L115

and it's based on

https://github.com/Microsoft/PTVS/blob/master/Python/Product/PyDebugAttach/PyDebugAttach.cpp#L1152

Now, there's a difference from the implementation I chose vs the one in PyDebugAttach in that I actually try to validate that the threadStateIndex was found/valid, whereas PyDebugAttach doesn't validate (and "restores" the threadStateIndex to -1 in case it didn't match). so, I'd like to check if it's really ok to proceed in this case (if it is I'll just remove that validation from pydevd).

Test where this happened:

• test_error_in_condition[module-attach_socket_cmdline-NameError] (MacOS, Python 3.7)

[int19h]

Per pythread.h, this function has been deprecated in Python 3.7 – perhaps that is why?

/* Thread Local Storage (TLS) API
   TLS API is DEPRECATED.  Use Thread Specific Storage (TSS) API.

   The existing TLS API has used int to represent TLS keys across all
   platforms, but it is not POSIX-compliant.  Therefore, the new TSS API uses
   opaque data type to represent TSS keys to be compatible (see PEP 539).
*/

PyAPI_FUNC(int) PyThread_create_key(void) Py_DEPRECATED(3.7);
PyAPI_FUNC(void) PyThread_delete_key(int key) Py_DEPRECATED(3.7);
PyAPI_FUNC(int) PyThread_set_key_value(int key, void *value) Py_DEPRECATED(3.7);
PyAPI_FUNC(void *) PyThread_get_key_value(int key) Py_DEPRECATED(3.7);
PyAPI_FUNC(void) PyThread_delete_key_value(int key) Py_DEPRECATED(3.7);

It looks like on Linux and Mac specifically, it might return NULL now:

void *
PyThread_get_key_value(int key)
{
#ifdef PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT
    return pthread_getspecific(key);
#else
    return NULL;
#endif
}

Whereas on Windows it just goes to TLS:

void *
PyThread_get_key_value(int key)
{
    /* because TLS is used in the Py_END_ALLOW_THREAD macro,
     * it is necessary to preserve the windows error state, because
     * it is assumed to be preserved across the call to the macro.
     * Ideally, the macro should be fixed, but it is simpler to
     * do it here.
     */

    DWORD error = GetLastError();
    void *result = TlsGetValue(key);
    SetLastError(error);
    return result;
}

[DonJayamanne]

Suggestion:

• Please fix and release a point release so we can get our UI Tests running without any issues.
• If we do for any unforeseen reason decide to ship a point release of VS Code, we might release this new version of PTVSD (rationale - If debugger breaks consistently on CI, then it could fail for users as well).
  • I.e. its best to be ready with a fix

Information:

• We're running some UI Tests on Mac+Win+Linux, Python 2.7 and Python 3.7 on CI
• Debugging always fails on Mac Python 3.7
• Reverting to older version of PTVSD fixes this issue.
• Debugger seems to hang immediately after this message is displayed in terminal (see below image).
• Debugger hangs for over 20 seconds (i.e. it doesn't continue after the Unable to find threa... message is printed in the terminal.

Attachments:

• Image (evidence of debugger hanging after error printed in terminal)
• Logs (PTVSD logs attached)

ptvsd-1653.log

[fabioz]

I'm currently working on this issue.

[vibhuyadav]

getting the same error.

[minimAluminiumalism]

Meeting the same error today when I use the latest VSCode 1.37.0.

[lopugit]

woo python 3.7 woo, I went back to 3.5.2

[ygbr]

I'm also getting this on the latest Insiders build.
Python 3.7.4 on macOS using:

Version: 1.38.0-insider
Commit: ffa22b268f451b5211abbe2d2a47e6b753b55b1a
Date: 2019-08-13T07:23:55.783Z
Electron: 4.2.9
Chrome: 69.0.3497.128
Node.js: 10.11.0
V8: 6.9.427.31-electron.0
OS: Darwin x64 18.7.0

It also seems to be related with CherryPy WSGI applications being unable to auto-reload, the engine locks-down with:

ENGINE Waiting for thread pydevd.CheckAliveThread.

[Mithridates01]

I am also having the same issue. Are there any temporary workarounds for this?

[karthiknadig]

I just released https://pypi.org/project/ptvsd/4.3.2/ that has the fix for this and one another issue #1688 .

For attach scenarios:

If you are installing ptvsd in the environment for attach scenario then please update.

For launch scenarios:

This is a temporary workaround until the extension updates to the latest ptvsd.

1. Pip install to some directory. Install in a separate directory to avoid issues when Python extension updates to version of ptvsd with the fix.
   -m pip install --target=D:\debugger ptvsd
2. Make this change in your launch json. Note the customDebugger and PYTHONPATH env.

{
    "name": "Python: Current File",
    "type": "python",
    "request": "launch",
    "program": "${file}",
    "console": "integratedTerminal",
    "customDebugger": true,
    "env":{
        "PYTHONPATH": "D:\\debugger"
    }
},

The extension will show squiggles under the customDebugger setting. This is ok, it is by design.

/cc @DonJayamanne

[ygbr]

Hi @karthiknadig it does seem to have solved the Unable to find threadStateIndex, but I still get the hanging on ENGINE Waiting for thread pydevd.CheckAliveThread. which needs me to fully restart the debugger... do you think it is related to this issue or is it another thing?

tks!

[karthiknadig]

@ygbr That looks like a different issue. Filed a bug here with details, #1691.

[karthiknadig]

Python extension has released an update with a fix for "Unable to find threadStateIndex for the current thread" issue.