Segmentation fault at exit for Python 3.8+ #1131
Description
Environment
How do you use Sentry? SaaS.
Which SDK and version? Python 0.19.2-1.1.0 (we've tried all)
Steps to Reproduce
- Upgrade our Docker image to Python 3.8 (ENH: Bump Dockerfile nipreps/fmriprep#2409)
- Get segfaults at sys.exit
Expected Result
No segfaults.
Actual Result
Quick note: This seems to be in some way related to the data that's being used. Other test data does not produce segfaults, but this is 100% reliable at this point. Unfortunately, a minimal reproduction is beyond our collective available effort. We're hoping that providing logs will make the issue obvious to y'all.
Here are log excerpts:
Python 3.8, sentry 1.1.0, no debugging
Fatal Python error: Segmentation fault
Thread 0x00007f7942cdc700 (most recent call first):
File "/usr/local/miniconda/lib/python3.8/threading.py", line 302 in wait
File "/usr/local/miniconda/lib/python3.8/site-packages/sentry_sdk/_queue.py", line 180 in get
File "/usr/local/miniconda/lib/python3.8/site-packages/sentry_sdk/worker.py", line 126 in _target
File "/usr/local/miniconda/lib/python3.8/threading.py", line 870 in run
File "/usr/local/miniconda/lib/python3.8/site-packages/sentry_sdk/integrations/threading.py", line 67 in run
File "/usr/local/miniconda/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/usr/local/miniconda/lib/python3.8/threading.py", line 890 in _bootstrap
Current thread 0x00007f7968bd7740 (most recent call first):
<no Python frame>
Python 3.8, sentry 0.19.5, no debugging
Fatal Python error: Segmentation fault
Thread 0x00007f8f121e7700 (most recent call first):
File "/usr/local/miniconda/lib/python3.8/threading.py", line 302 in wait
File "/usr/local/miniconda/lib/python3.8/site-packages/sentry_sdk/_queue.py", line 180 in get
File "/usr/local/miniconda/lib/python3.8/site-packages/sentry_sdk/worker.py", line 123 in _target
File "/usr/local/miniconda/lib/python3.8/threading.py", line 870 in run
File "/usr/local/miniconda/lib/python3.8/site-packages/sentry_sdk/integrations/threading.py", line 67 in run
File "/usr/local/miniconda/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/usr/local/miniconda/lib/python3.8/threading.py", line 890 in _bootstrap
Current thread 0x00007f8f380a3740 (most recent call first):
<no Python frame>
Python 3.8, sentry 1.1.0, removed threading integration, using sentry_sdk.Hub.current.client.close(timeout=2.0) before exit
[sentry] DEBUG: Setting up integrations (with default = False)
[sentry] DEBUG: Setting up previously not enabled integration atexit
[sentry] DEBUG: Setting up previously not enabled integration excepthook
[sentry] DEBUG: Setting up previously not enabled integration dedupe
[sentry] DEBUG: Setting up previously not enabled integration stdlib
[sentry] DEBUG: Setting up previously not enabled integration modules
[sentry] DEBUG: Setting up previously not enabled integration argv
[sentry] DEBUG: Setting up previously not enabled integration logging
[sentry] DEBUG: Enabling integration atexit
[sentry] DEBUG: Enabling integration excepthook
[sentry] DEBUG: Enabling integration dedupe
[sentry] DEBUG: Enabling integration stdlib
[sentry] DEBUG: Enabling integration modules
[sentry] DEBUG: Enabling integration argv
[sentry] DEBUG: Enabling integration logging
[...]
[sentry] DEBUG: Sending event, type:null level:info event_id:28838be7e52442cbb7b4a22f59233f8b project:1137693 host:sentry.io
[...]
[sentry] DEBUG: Sending event, type:null level:info event_id:1b85ce2784b148fe9d81f545cc54a856 project:1137693 host:sentry.io
210611-01:31:00,659 nipype.workflow IMPORTANT:
Generating reports...
210611-01:31:02,263 nipype.workflow IMPORTANT:
Closing Sentry client pre-shutdown...
[sentry] DEBUG: Flushing HTTP transport
[sentry] DEBUG: background worker got flush request
[sentry] DEBUG: background worker flushed
[sentry] DEBUG: Killing HTTP transport
[sentry] DEBUG: background worker got kill request
210611-01:31:02,264 nipype.workflow IMPORTANT:
Exiting...
[sentry] DEBUG: Killing HTTP transport
[sentry] DEBUG: background worker got kill request
[sentry] DEBUG: atexit: got shutdown signal
[sentry] DEBUG: atexit: shutting down client
Fatal Python error: Segmentation fault
Thread 0x00007f33f2e19700 (most recent call first):
File "/usr/local/miniconda/lib/python3.8/site-packages/sentry_sdk/worker.py", line 126 in _target
File "/usr/local/miniconda/lib/python3.8/threading.py", line 870 in run
File "/usr/local/miniconda/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/usr/local/miniconda/lib/python3.8/threading.py", line 890 in _bootstrap
Current thread 0x00007f3418bc6740 (most recent call first):
<no Python frame>
Python 3.8, sentry 1.1.0, removed threading integration, removed atexit integration, using sentry_sdk.Hub.current.client.close(timeout=2.0) before exit
[sentry] DEBUG: Setting up integrations (with default = False)
[sentry] DEBUG: Setting up previously not enabled integration excepthook
[sentry] DEBUG: Setting up previously not enabled integration dedupe
[sentry] DEBUG: Setting up previously not enabled integration stdlib
[sentry] DEBUG: Setting up previously not enabled integration modules
[sentry] DEBUG: Setting up previously not enabled integration argv
[sentry] DEBUG: Setting up previously not enabled integration logging
[sentry] DEBUG: Enabling integration excepthook
[sentry] DEBUG: Enabling integration dedupe
[sentry] DEBUG: Enabling integration stdlib
[sentry] DEBUG: Enabling integration modules
[sentry] DEBUG: Enabling integration argv
[sentry] DEBUG: Enabling integration logging
[...]
[sentry] DEBUG: Sending event, type:null level:info event_id:3049aa796bff4565ae4fdb3a509e916d project:1137693 host:sentry.io
[...]
[sentry] DEBUG: Sending event, type:null level:info event_id:96461d8a7a8a4ba4800e1c343420b102 project:1137693 host:sentry.io
210611-03:13:15,683 nipype.workflow IMPORTANT:
Generating reports...
210611-03:13:17,16 nipype.workflow IMPORTANT:
Closing Sentry client pre-shutdown...
[sentry] DEBUG: Flushing HTTP transport
[sentry] DEBUG: background worker got flush request
[sentry] DEBUG: background worker flushed
[sentry] DEBUG: Killing HTTP transport
[sentry] DEBUG: background worker got kill request
Fatal Python error: Segmentation fault
Thread 0x00007f143a413700 (most recent call first):
File "/usr/local/miniconda/lib/python3.8/threading.py", line 302 in wait
File "/usr/local/miniconda/lib/python3.8/site-packages/sentry_sdk/_queue.py", line 180 in get
File "/usr/local/miniconda/lib/python3.8/site-packages/sentry_sdk/worker.py", line 126 in _target
File "/usr/local/miniconda/lib/python3.8/threading.py", line 870 in run
File "/usr/local/miniconda/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/usr/local/miniconda/lib/python3.8/threading.py", line 890 in _bootstrap
Current thread 0x00007f14601c0740 (most recent call first):
<no Python frame>
Python 3.8, sentry 0.19.2, removed threading integration, removed atexit integration, using sentry_sdk.Hub.current.client.close(timeout=2.0) before exit
[sentry] DEBUG: Setting up integrations (with default = False)
[sentry] DEBUG: Setting up previously not enabled integration excepthook
[sentry] DEBUG: Setting up previously not enabled integration dedupe
[sentry] DEBUG: Setting up previously not enabled integration stdlib
[sentry] DEBUG: Setting up previously not enabled integration modules
[sentry] DEBUG: Setting up previously not enabled integration argv
[sentry] DEBUG: Setting up previously not enabled integration logging
[sentry] DEBUG: Enabling integration excepthook
[sentry] DEBUG: Enabling integration dedupe
[sentry] DEBUG: Enabling integration stdlib
[sentry] DEBUG: Enabling integration modules
[sentry] DEBUG: Enabling integration argv
[sentry] DEBUG: Enabling integration logging
[...]
[sentry] DEBUG: Sending event, type:null level:info event_id:f8750c08cbec4987831da5142514c18e project:1137693 host:sentry.io
[...]
[sentry] DEBUG: Sending event, type:null level:info event_id:1cbbc072b833448da494219e4bd9b6af project:1137693 host:sentry.io
210611-22:59:02,410 nipype.workflow IMPORTANT:
Generating reports...
210611-22:59:03,670 nipype.workflow IMPORTANT:
Closing Sentry client pre-shutdown...
[sentry] DEBUG: Flushing HTTP transport
[sentry] DEBUG: background worker got flush request
[sentry] DEBUG: background worker flushed
[sentry] DEBUG: Killing HTTP transport
[sentry] DEBUG: background worker got kill request
210611-22:59:03,670 nipype.workflow IMPORTANT:
Exiting...
Fatal Python error: Segmentation fault
Thread 0x00007fdef895a700 (most recent call first):
File "/usr/local/miniconda/lib/python3.8/threading.py", line 302 in wait
File "/usr/local/miniconda/lib/python3.8/site-packages/sentry_sdk/_queue.py", line 180 in get
File "/usr/local/miniconda/lib/python3.8/site-packages/sentry_sdk/worker.py", line 118 in _target
File "/usr/local/miniconda/lib/python3.8/threading.py", line 870 in run
File "/usr/local/miniconda/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/usr/local/miniconda/lib/python3.8/threading.py", line 890 in _bootstrap
Current thread 0x00007fdf1e707740 (most recent call first):
<no Python frame>
Python 3.9, sentry 0.19.2, removed threading integration, removed atexit integration, using sentry_sdk.Hub.current.client.close(timeout=2.0) before exit
[sentry] DEBUG: Setting up integrations (with default = False)
[sentry] DEBUG: Setting up previously not enabled integration excepthook
[sentry] DEBUG: Setting up previously not enabled integration dedupe
[sentry] DEBUG: Setting up previously not enabled integration stdlib
[sentry] DEBUG: Setting up previously not enabled integration modules
[sentry] DEBUG: Setting up previously not enabled integration argv
[sentry] DEBUG: Setting up previously not enabled integration logging
[sentry] DEBUG: Enabling integration excepthook
[sentry] DEBUG: Enabling integration dedupe
[sentry] DEBUG: Enabling integration stdlib
[sentry] DEBUG: Enabling integration modules
[sentry] DEBUG: Enabling integration argv
[sentry] DEBUG: Enabling integration logging
[...]
[sentry] DEBUG: Sending event, type:null level:info event_id:a4e5511d25d54241b00debfa7982a5b2 project:1137693 host:sentry.io
[...]
[sentry] DEBUG: Sending event, type:null level:info event_id:837eb0487fa9464ab0a02972f4a0aeb0 project:1137693 host:sentry.io
210612-02:56:44,343 nipype.workflow IMPORTANT:
Generating reports...
210612-02:56:45,912 nipype.workflow IMPORTANT:
Closing Sentry client pre-shutdown...
[sentry] DEBUG: Flushing HTTP transport
[sentry] DEBUG: background worker got flush request
[sentry] DEBUG: background worker flushed
[sentry] DEBUG: Killing HTTP transport
[sentry] DEBUG: background worker got kill request
210612-02:56:45,913 nipype.workflow IMPORTANT:
Exiting...
Fatal Python error: Segmentation fault
Current thread 0x00007f6b8a9c5740 (most recent call first):
<no Python frame>
Why 0.19.2? We were using that with Python 3.7 and had no issues. For a bit more context, we're running the Nipype workflow engine, which uses concurrent.futures to schedule jobs. Anyway, hope you all have seen something like this before, but I got limited hits for "segfault".