Windows: profiling inside a venv?

[madig]

I'm having trouble running py-spy inside a venv on Windows 10 x64 1909, Python 3.8.1. It errors out with Error: Failed to find python version from target process. Admin privileges don't help. I presume the python.exe inside venv\Scripts is not what py-spy expects, so I copied over the actual EXE and DLL from the global Python38 folder. Profiling then works, but runs extremely slowly with considerable sampling lag.

Is this a known unsupported use-case?

[benfred]

I've seen problems with venv on windows before #81 (comment) - in that case what was happening was that python.exe in the virtualenv creates a subprocess that is the actual python process. Connecting up to the parent process fails because it doesn't contain a python interpreter.

We have an option to profile subprocesses which should support this use case. Can you try that?

[madig]

> py-spy record -s -F -n -f speedscope --output pyspy.speedscope -- python .\bench.py
Can't get native stack traces with the ---subprocesses option on windows.

> py-spy record -s -F -f speedscope --output pyspy.speedscope -- python .\bench.py

Error: No python processes found in process 1580 or any of its subprocesses

Unless I'm doing it wrong?

[IceTDrinker]

Can confirm I observe a similar behavior, are there plans to change the way py-spy tries to profile python in a Windows venv ?

Cheers

[IceTDrinker]

Looking at how python sets things up on windows in the Python Home you have Libs/venv/scripts/nt which contains scripts that get copied during venv setup through "pyhton -m venv venv_dir"

Apparently the python.exe and pythonw.exe that are there are in fact venvlauncher.exe and venvwlauncher.exe that got renamed.

Not too sure on how to hook in venvlauncher quite yet though

Any pointers as to how one might poke in those executables to find the python interpreters ?

[IceTDrinker]

The symptom I observe is that no child process is found (I printed the length of the spies hashmap with --subprocesses, len was 1) it seems children retrieval does not work or python.exe from venv (i.e. venvlauncher.exe in disguise) does not manage to launch the actual python interpreter or it does not appear as a child process for some reason.

Cheers

[benfred]

Fixed by #370

[jankap]

The proposed fix does not work for me on Windows 10, Python 3.10, py-spy v0.3.11. No matter, if I run with pid or file arguments. I do provide the -s flag. The error is still Error: No python processes found in process xxxx or any of its subprocesses. I've tried to run it from inside the venv via pipenv run and pipenv shell as well as a global user install via pip.

Any advices? Thank you :)

[spacemanspiff2007]

I am experiencing the same issue under Win10 with Python 3.10.1.
After activating the venv I run

py-spy top -s -- python -m MyModule

and the output

Error: No python processes found in process 20872 or any of its subprocesses

@benfred
What kind of data can I provide that helps you to reproduce the issue?

[IceTDrinker]

I was using python 3.8 at the time, could you try 3.8 to see if it's a 3.10 issue?

I had the issue in the first place but having some time to fecth the subprocesses fixed my issue with my configuration.

Also as a curiosity how long does the call to MyModule last?

[spacemanspiff2007]

I can indeed confirm that this seems to be a 3.10. With 3.8 it works as expected.

[IceTDrinker]

Perhaps the way python 3.10 spawns threads is different, could be worth checking 3.9 as well if you get a chance!

[skewballfox]

ran into this issue myself with python3.10 on windows, is there any information I could collect locally which might help figure out the issue?

[evg656e]

Ran into same issue with python 3.11 on Windows 10. Here is some logs with RUST_LOG=info logger enabled (with --subprocesses and without it).

py-spy.log
py-spy-s.log

[tyler-suard-parker]

I have this same issue on Python 3.9