-
Notifications
You must be signed in to change notification settings - Fork 516
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Segmentation fault in sentry profiler #2386
Comments
Thanks for the report. This looks like a problem with python but we'll take a look to see if there's anything we can do to mitigate it from our end. |
@Zylphrex it would be great if you could open a CPython issue and mention us if you conclude that there is nothing you are doing wrong at your end. I've pinged someone in python/cpython#99110 who worked on similar issues in CPython 3.11 / 3.12, but so far got no response. |
@zyv I haven't been able to identify anything that we can do from the SDK to avoid the segfault without disabling the profiler entirely. Also, it would be great if you could provide some more info about your environment that may be helpful in reproducing this problem.
Also some questions:
|
Python 3.11.5 (main, Sep 7 2023, 18:53:16) [GCC 12.2.0]
gunicorn -b 0.0.0.0:8000 --workers 2 --threads 4 --keep-alive 5 config.wsgi --log-file - --access-logfile - --log-level debug --enable-stdio-inheritance
add-trailing-comma==3.0.1 |
No, this happens after some time and so far we couldn't determine when & why exactly does it happen. As you can see, we run gunicorn with multiple threads (and apparently Sentry profiler is running in a separate thread on top of that). The segfaults happen under load, if the load is low, then they might not happen for days. We are also getting those segfaults in Celery background workers. This was not the case as we were using gevent, but due to extreme bugginess of gevent under Python 3.11 (gevent/gevent#1985) we also had to switch to threads there...
If you mean SIGINT to gunicorn, then no, it's not a factor. |
Hi, we are also experiencing a segfault in a multi-thread context that appears to be coming from the sentry profiler:
We're going to see if turning the profiler off stops it from happening. |
Hey @fnmut ! |
We are also facing similar issue. Even after disabling the profiler we are seeing this issue. python -VV pip freeze
|
We were forced to upgrade to Python 3.12 to get rid of this issue. Sadly found no solution with Python 3.11, but with Python 3.12 the issue seems to be fixed. |
@Zylphrex do you have any clue why this could happen in Python 3.11 but not in 3.12? |
@antonpirker I've linked a related CPython issue in Python 3.11 and they fixed more of those in Python 3.12... but apparently nobody was so impressed as to backport these fixes to Python 3.11. |
@iamyoudi have to tried upgradint to Python 3.12 to get rid of the problem? |
Thanks @zyv So I guess there is nothing we can do on our end. @zyv mentioned the problem in the CPython repo: python/cpython#99110 The work around for this is to upgrade to Python 3.12. I am closing this but will keep the comments open, so people can find this and maybe post future improvements in Python 3.11 or other workarounds. |
@antonpirker, I work with @fnmut. We've upgraded to Python 3.12, turned profiler back on... and segfaults again. Latest stacktrace is
Python version: 3.12.1 sentry-sdk 1.36.0 And the stacktrace according to
|
That's funny, it seems that |
What Python version are you running? I've been checking on this issue but I haven't found any new open issues in CPython that look related to this. |
How do you use Sentry?
Sentry Saas (sentry.io)
Version
1.30.0
Steps to Reproduce
The issue is hard to reproduce exactly as it occur sparingly.
The issue occurs in our Django application and whenever it happens there is a server crash with:
Example of an extract from log:
Worker (pid:33) was sent SIGSEGV!
The server start command is
gunicorn -b 0.0.0.0:8000 --workers 2 --threads 4 --keep-alive 5 config.wsgi --log-level info
From the logs attached below, it appears the server crashes when at a point sentry profiling resulted in
Segmentation fault
Expected Result
The sentry-sdk being able to do the expected profiling correctly and reporting them to Sentry.
Actual Result
Below are the logs from analysing the coredump with gdb
The text was updated successfully, but these errors were encountered: