Issues Attaching Debugger to Python Application

[vsfeedback]

This issue has been moved from a ticket on Developer Community.

---

We have an Embedded Python specific application that we can’t use breakpoints when attaching with the debugger using Visual Studio 2022. Visual Studio 2017 does not have the issue.

We can successfully debug with Python 3.7.7 and Visual Studio 2017 with breakpoints using these steps:

1. Run the application
2. In Visual Studio 2017, go to Debug > Attach to Process
3. Set connection type: as local
4. Set Code type to Python code

We can successfully attach to the process, but not use breakpoints, with Visual Studio 2022 with these steps:

1. Run the application
2. In Visual Studio, go to Debug > Attach to Process
3. Set connection type: as local
4. Set Code type to Python (native) code

If we use Python code instead of Python (native), we get the following error or something similar:

“Failed to launch debug adapter. Timed out waiting for debug server to connect.”

If we compile the application and run the debugger through the sln, we can debug the C++ code. We cant debug py code this way either.

---

Original Comments

Tim Lakdawala on 10/30/2024, 08:56 AM:

(private comment, text removed)

Tim Lakdawala on 10/30/2024, 09:02 AM:

(private comment, text removed)

Feedback Bot on 10/30/2024, 09:39 AM:

We have directed your feedback to the appropriate engineering team for further evaluation. The team will review the feedback and notify you about the next steps.

---

Original Solutions

Feedback Bot solved on 10/30/2024, 04:57 PM, undefined votes:

Thanks for taking the time to report this issue! It seems to us that it’s describing the same thing as another one. We’ve closed this one as a duplicate and transferred all votes from this ticket to the other one. Please follow along on the linked ticket to communicate with the team, see updates on status, and help provide any needed diagnostic info. For more information, see our issue reporting guidelines.

Happy coding!

Tim Lakdawala on 10/31/2024, 03:06 AM:

Could we please get a link to the ticket this was moved over to? I don’t see one in the resolution.

Tim Lakdawala on 10/31/2024, 06:13 AM:

I never received the link to the other ticket so I can’t confirm if they are different.

Lisa Gao (CSI Interfusion Inc) [MSFT] on 11/1/2024, 01:14 PM:

Hi, this issue was traced by #6993.

Tim Lakdawala on 11/2/2024, 01:18 AM:

Hey Lisa,

Thanks for sharing the github issue. The issue isnt exactly our issue. In our case we are attempting to attach to an already compiled exe that has embedded python. Can we please have this request reopened?

[rchiodo]

Thanks for the issue. I'm going to need some more information to diagnose the problem.

My guess is that you're trying this with Python 3.10 or higher. Python (Native) debugging isn't currently supported on anything higher than 3.9.

[tlakdawala]

Hey rchiodo,

Thanks for the quick response. First question is there an expectation when Python 3.10 will be supported by Native debugging?

We have run into this issue with both Python 3.7.7 and 3.10. Below is some additional output. Please let me know what else I can provide.

Attaching to PID: 39428 (elapsed: 1.01s).
Attaching to PID: 39428 (elapsed: 2.02s).
Attaching to PID: 39428 (elapsed: 3.03s).
Attaching to PID: 39428 (elapsed: 4.05s).
Attaching to PID: 39428 (elapsed: 5.05s).
Attaching to PID: 39428 (elapsed: 6.06s).
Attaching to PID: 39428 (elapsed: 7.07s).
Attaching to PID: 39428 (elapsed: 8.09s).
Attaching to PID: 39428 (elapsed: 9.10s).
Attaching to PID: 39428 (elapsed: 10.11s).
--- Starting attach to pid: 39428 ---
--- Connecting to 64 bits target (current process is: 64) ---
--- Injecting attach dll: attach_amd64.dll into pid: 39428 ---
Running executable to inject dll.
OpenProcess with pid: 39428
VirtualAllocEx in pid: 39428
WriteProcessMemory in pid: 39428
loadLibraryAddress: 39428
Waiting for LoadLibraryA to complete.
Ok, finished dll injection.
--- Injecting run code dll: run_code_on_dllmain_amd64.dll into pid: 39428 ---
Running executable to inject dll.
OpenProcess with pid: 39428
VirtualAllocEx in pid: 39428
WriteProcessMemory in pid: 39428
loadLibraryAddress: 39428
Waiting for LoadLibraryA to complete.
Ok, finished dll injection.
Attaching to PID: 39428 (elapsed: 11.11s).
Attaching to PID: 39428 (elapsed: 12.12s).
Attaching to PID: 39428 (elapsed: 13.14s).
Attaching to PID: 39428 (elapsed: 14.15s).
From Embedded Python App..
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:program filesmicrosoft visual studio2022professionalcommon7deextensionsmicrosoftpythoncoredebugpy/..debugpyserverattach_pid_injected.py", line 90, in attach
    log.reraise_exception()
  File "c:program filesmicrosoft visual studio2022professionalcommon7deextensionsmicrosoftpythoncoredebugpy/..debugpyserverattach_pid_injected.py", line 77, in attach
    debugpy.connect(
  File "c:program filesmicrosoft visual studio2022professionalcommon7deextensionsmicrosoftpythoncoredebugpypublic_api.py", line 31, in wrapper
    return wrapped(*args, **kwargs)
  File "c:program filesmicrosoft visual studio2022professionalcommon7deextensionsmicrosoftpythoncoredebugpyserverapi.py", line 143, in debug
    log.reraise_exception("{0}() failed:", func.__name__, level="info")
  File "c:program filesmicrosoft visual studio2022professionalcommon7deextensionsmicrosoftpythoncoredebugpyserverapi.py", line 141, in debug
    return func(address, settrace_kwargs, **kwargs)
  File "c:program filesmicrosoft visual studio2022professionalcommon7deextensionsmicrosoftpythoncoredebugpyserverapi.py", line 297, in connect
    _settrace(host=host, port=port, client_access_token=access_token, **settrace_kwargs)
  File "c:program filesmicrosoft visual studio2022professionalcommon7deextensionsmicrosoftpythoncoredebugpyserverapi.py", line 45, in _settrace
    return pydevd.settrace(*args, **kwargs)
  File "c:program filesmicrosoft visual studio2022professionalcommon7deextensionsmicrosoftpythoncoredebugpy_vendoredpydevdpydevd.py", line 2821, in settrace
    _locked_settrace(
  File "c:program filesmicrosoft visual studio2022professionalcommon7deextensionsmicrosoftpythoncoredebugpy_vendoredpydevdpydevd.py", line 2902, in _locked_settrace
    py_db.connect(host, port)  # Note: connect can raise error.
  File "c:program filesmicrosoft visual studio2022professionalcommon7deextensionsmicrosoftpythoncoredebugpy_vendoredpydevdpydevd.py", line 1421, in connect
    s = start_client(host, port)
  File "c:program filesmicrosoft visual studio2022professionalcommon7deextensionsmicrosoftpythoncoredebugpy_vendoredpydevd_pydevd_bundlepydevd_comm.py", line 493, in start_client
    s.connect((host, port))
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it

[rchiodo]

That callstack looks like you're doing plain 'Python' debugging, not mixed mode. That would require python to actually be loaded in the embedded app before attaching.

In VS 2022 we switched how our debugging works.

For normal python debugging (non mixed mode), we use something called https://github.com/Microsoft/DEBUGPY. Debugpy is a soft-mode debugger, meaning it runs using APIs provided by the python runtime in order to debug. If the Python runtime isn't started yet, you'd get the error you show above.

For mixed mode debugging (also called Python Native), we use concord to implement a hard-mode debugger that uses the native C++ debugger in VS to listen to Windows debug events.

You might also be selecting both types of debugging at the same time. That isn't necessary. If you want to debug in mixed mode, just pick Python (Native) and Native in this drop down:

[rchiodo]

Support for 3.10/11/12/13 is a work in progress for mixed mode. Not sure when it will ship.

[tlakdawala]

We did mixed mode debugging (Native and Python (native)) using both 3.7.7 and 3.10 with VS2022. It connects but if you set a breakpoint in the Python code it will not stop at the breakpoint. We don't get any other output.

Is there any log we can check for more information?

[tlakdawala]

"That callstack looks like you're doing plain 'Python' debugging, not mixed mode. That would require python to actually be loaded in the embedded app before attaching."

The main developer wanted to also provide a note for this comment. The embedded Python is loaded before attaching. We utilize the PyObject_CallObject to first run our script. Further calls to PyObject_CallObject occur due to Window OS events such, as button presses. None of breakpoints are hit on these.

[rchiodo]

With mixed mode you can actually just debug the C++ app from the start. That might fix the breakpoints binding. There was a bug in 3.9 mixed mode where breakpoints wouldn't bind during attach. Launching would have fixed that problem. Maybe there's something wrong in 3.7.

3.10 shouldn't work at all. I'd be surprised if the debugger even started.

Ideally if you could share your app, I could just debug the problem.

You could also try the sample app we have here and see if attach works. I wrote it to debug another customer's attach situation.
https://github.com/microsoft/PTVS/tree/main/Examples/PythonNative

[rchiodo]

I'm not entirely sure PyObject_CallObject loads the full runtime in order for debugpy (non mixed mode) to attach. I'd need a piece of sample code to verify.

But mixed mode should work regardless how you load Python.

[tlakdawala]

Thanks for this info. The Main dev is going to look at your mention of the bug above and the sample you provided. He is also going to setup a Github account to be able to respond directly on the thread.

I am asking internally if we have permission to share the app. Do you have a private upload location that we can use if we can?

[rchiodo]

Do you have a private upload location that we can use if we can?

Just my e-mail address. It's my github id at microsoft dot com.

[tlakdawala]

I confirmed with our Infosec team we are unable to share the app. We can test the sample app instead.

[stevetm2w]

I took a look at the example code. We perform some different steps when we first run a python script from our C++ code, as we are importing a module rather than running a simple file...

if (m_pModule)// finalize the last ran module (if there was one) just before reload
{
Py_DECREF((PyObject )m_pModule);
Py_Finalize();
}
// other Py_SetPath and PyImport_AppendInittab code is here
Py_Initialize();
m_pModule = PyImport_ImportModule("Python_program");//name of the script to import
pFunc = PyObject_GetAttrString((PyObject)m_pModule, "main"); // method ptr
PyObject_CallObject(pFunc, NULL);

We do have a number of embedded methods we regester using PyImport_AppendInittab and its callbacks. Also, on exit we call the above Py_Finalize block.

[rchiodo]

None of that should matter if you're using mixed mode debugging. Mixed mode is actually all just a C++ debugger that displays C++ frames as their equivalent Python frames.

It's only the normal Python debugging that might not work with those calls.

[stevetm2w]

"Launching would have fixed that problem. Maybe there's something wrong in 3.7." - so if I do launch the C++ app with embedded python 3.10, how do I tell it to use mixed mode? I normally just press F5 to launch the C++ app btw... I've only noticed the "Python (native)" and "Native" options when using the Attach method.

[rchiodo]

3.10 won't work as mixed mode isn't working for that yet. It should work for 3.7 but 3.9 would be the safest bet as that's gotten the most bug fixes in VS 2022.

In the Debugger properties for the vcxproj there should be an option to pick Python/Native debugging

[tlakdawala]

We were able to get Python 3.7.7 working with mixed mode debugging with the direct debugger and attach to process. Thank you for all the helpful information!

Could we please get an update in this ticket once Python 3.10 is supported with mixed mode debugging?

[rchiodo]

I can try but it's unlikely that I'd remember. We have 1000s of tickets. Your better solution would be to watch for when we ship 3.13 support in VS 2022. There should be an announcement for that. I think we're shooting for early next year.

[rchiodo]

Or you can just ping me here because I do get an e-mail every time somebody updates an issue I'm assigned.

[tlakdawala]

Sounds good! Ill set a reminder to watch out for it early next year. I can ping you if we need to follow up.

[glechapelain]

Hi !

[stevetm2w]

Hey... just wanted to add a note this comment from Tim "We were able to get Python 3.7.7 working with mixed mode debugging with the direct debugger and attach to process"... this is true for VS2022, however I cannot change the execution line after hitting a breakpint, eg I cannot change the Py Program Counter via the "Set Next Statement" right click option. Just a heads up on that, as that works with VS2017.

[rchiodo]

The old debugger was hacking CPython to let it do that:

PTVS/Python/Product/PythonTools/visualstudio_py_debugger.py

Line 1919 in cfdced9

def on_legacySetLineNumber(self, request, args):

It seems it could be possible to do the same in the new debugger though, just it was never implemented:
https://docs.python.org/3/reference/datamodel.html#frame.f_lineno

[rchiodo]

I'll enter a new bug to see how popular set next statement is.
#8073