Profiler raises ValueError in Python 3.12

[anorthall]

Enabling the profiling panel in Python 3.12 and Django 5.0.1 causes the following exception:

   File "/usr/local/lib/python3.12/site-packages/debug_toolbar/panels/profiling.py", line 144, in process_request
     return self.profiler.runcall(super().process_request, request)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.12/cProfile.py", line 109, in runcall
     self.enable()
 ValueError: Another profiling tool is already active

This is due to a change in Python 3.12: python/cpython#110770

There is also some relevant discussion of the same issue over on the django-silk repo which may be helpful in finding a way forward: jazzband/django-silk#682

[elineda]

Hello,

Can you give me step for reproduce the bug. I try enable the profiling panel with django 5.0.1 and python 3.12.1, I have not the problem.

[matthiask]

It happens to me as well, but only intermittently. Maybe reloading a few times, enabling and disabling the profiler panel using the checkbox and/or restarting the development server works when trying to reproduce the problem?

[ddkasa]

This issue is happening for me as well, but I can't seem to determine what's causing the issue.

Tried to go through all the installed apps and no conflicts seem to resolve it and tried loading with 2 browsers as well.

Django 5.0.2 / Python 3.12.1

What's the progress on this or are there any updates on this?

Working the profile panel disabled by default in settings for now.

[mfosterw]

I don't have a solution, but I have found a way to consistently produce the bug. When running a NextJS app that depends on a django api with the vscode debugger, every hard refresh from the nextjs frontend results in this error on the backend. I have no idea if the vscode debugger is contributing to the error, but note that I am running both the frontend and the backend from vscode. I hope this can be useful for someone to eventually debug the problem! (In the meantime, I'll be disabling the panel).

[elineda]

Ok seem the problem occurs only the two requests are send on the runserver in the same times. Using a debuger like pycharm one is not necessary.
That's strange because runserver only do one request at the time. Need to find one profiler open even if the other is not finished

[elineda]

Found : you need to put --nothreading in runserver @mfosterw @ddkasa and @matthiask can you test this ?

[elineda]

I propose to add a note in installation.rst or tip.rst. I don't know if it's possible by adding something in the debug toolbar to disable the multitreading.

[ddkasa]

@elineda Adding --nothreading to the runserver command does seem to resolve the issue for me.

I am running my development server through Uvicorn though, so I am curious if you know a solution for that as well?

[elineda]

@elineda Adding --nothreading to the runserver command does seem to resolve the issue for me.

I am running my development server through Uvicorn though, so I am curious if you know know a solution for that as well?

Nan no way, uvicorn doesn't accept thread value

[tim-schilling]

I propose to add a note in installation.rst or tip.rst. I don't know if it's possible by adding something in the debug toolbar to disable the multitreading.

@elineda What do you think about putting it as an admonition/warning in the Profiling panel area of the docs? I think I have a slight preference to that, but would also be okay with it going in tip.rst. I'd prefer to keep the installation docs as short as possible.

[elineda]

I'm ok to add a note in profiling. If you have a problem with it you will check (I hope) the panel doc before complain.

[tim-schilling]

Agreed

[matthiask]

I wonder if it would be possible to add a counter variable which tracks the number of currently active profiling sessions; the profiler would only be activated when incrementing from zero and would only be deactivated when decrementing to zero?

[tim-schilling]

I think we'd have to count that across threads.

[elineda]

I wonder if it would be possible to add a counter variable which tracks the number of currently active profiling sessions; the profiler would only be activated when incrementing from zero and would only be deactivated when decrementing to zero?

in that case what we do when another request come, we do it without profiler or we wait ?

[elineda]

I think we'd have to count that across threads.

we can use cache for that.

[tim-schilling]

I think we'd have to count that across threads.

we can use cache for that.

Yes, that's an option. It would create a dependency on the CACHES setting being configured to run at least this particular panel. That's not ideal.

[elineda]

with async compatibilty we will need a cache, for the same reason isn't ?

[agusmakmun]

So the configs become:

DEBUG_TOOLBAR_CONFIG = {
    "DISABLE_PANELS": [
        "debug_toolbar.panels.profiling.ProfilingPanel",  # additional one
        "debug_toolbar.panels.redirects.RedirectsPanel",
    ],
    "SHOW_TEMPLATE_CONTEXT": True,
}

https://django-debug-toolbar.readthedocs.io/en/latest/configuration.html#toolbar-options

It's fixed for me as uvicorn doesn't have --nothreading parameter, especially for who use the django-cookie-cutter.

Meanwhile, to reproduce the profiler issue, we can just ctcl+s (save multiple times the python files in the project to trigger restarts).

[Mogost]

Perhaps I'm too naive. But why can't we just catch it and treat it as an exception?

https://github.com/jazzband/django-debug-toolbar/blob/b5cf78460d8a36d83ff8aadb97f01a6346527632/debug_toolbar/panels/profiling.py#L146

try:
	return self.profiler.runcall(super().process_request, request)
except ValueError:
	logger.warning(...)
	return super().process_request(request) 

Updated

I think we'd have to count that across threads.

we can use cache for that.

Yes, that's an option. It would create a dependency on the CACHES setting being configured to run at least this particular panel. That's not ideal.

Also, we could implement a check using .pid files. Before starting the profiler, check for an existing .pid file to ensure no other profiler is active. If the file exists, handle the conflict gracefully. This approach might prevent the error and ensure smooth profiling.

Or it's also possible (probably) to use lock just from threading (I am not sure)

import threading

profiler_lock = threading.Lock()

def start_profiler():
    if profiler_lock.acquire(blocking=False):
        try:
            # Start profiler
        finally:
            profiler_lock.release()
    else:
        print("Profiler already running")

I wonder if it would be possible to add a counter variable which tracks the number of currently active profiling sessions; the profiler would only be activated when incrementing from zero and would only be deactivated when decrementing to zero?

in that case what we do when another request come, we do it without profiler or we wait ?

wait or do without profile might be a setting.