Skip to content

DBG: Add debug logs to track down segfault #2423

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 19 commits into from
Closed

DBG: Add debug logs to track down segfault #2423

wants to merge 19 commits into from

Conversation

@effigies
Copy link
Member

@effigies effigies commented Jun 10, 2021

Adjunct to #2409.

mgxd and others added 9 commits May 13, 2021 19:56
@mgxd
ENH: Update Dockerfile
2adcdae 
This PR bumps up the Docker image base to Ubuntu Focal LTS (20.04).

As a result, a number of packages were updated, including:
- AFNI
- FSL
- Connectome Workbench
- Graphviz
- Pandoc

Furthermore, Python was bumped to 3.8.x, and many of its packages were "modernized" accordingly.
@mgxd
ENH: Bump bids-validator, pin svgo to minor
cab1f20
@mgxd @effigies
Update Dockerfile
7f0bcd0 
Co-authored-by: Chris Markiewicz <effigies@gmail.com>
@mgxd
FIX: Robustness across argparse formatting change
d06ef8b
@mgxd
FIX: Add pandoc citeproc
7aa4826
@mgxd @effigies
Update Dockerfile
6babc76 
Co-authored-by: Chris Markiewicz <effigies@gmail.com>
@mgxd
TST: Refresh data cache
a7f16a0
@mgxd
ENH: Small improvements to bids validity checks
3caef2b
@mgxd
TST: Downgrade validator to avoid network requirement
d5f03a5
@effigies effigies mentioned this pull request Jun 10, 2021
Merged
@effigies effigies force-pushed the enh/update-dockerfile branch from bb61abb to e16bc4c Compare June 10, 2021 20:01
mgxd
mgxd reviewed Jun 10, 2021
View reviewed changes
@effigies
Copy link
Member Author

effigies commented Jun 11, 2021

With AtexitIntegration:

210611-01:31:02,263 nipype.workflow IMPORTANT:
	 Closing Sentry client pre-shutdown...
 [sentry] DEBUG: Flushing HTTP transport
 [sentry] DEBUG: background worker got flush request
 [sentry] DEBUG: background worker flushed
 [sentry] DEBUG: Killing HTTP transport
 [sentry] DEBUG: background worker got kill request
210611-01:31:02,264 nipype.workflow IMPORTANT:
	 Exiting...
 [sentry] DEBUG: Killing HTTP transport
 [sentry] DEBUG: background worker got kill request
 [sentry] DEBUG: atexit: got shutdown signal
 [sentry] DEBUG: atexit: shutting down client
Fatal Python error: Segmentation fault

Thread 0x00007f33f2e19700 (most recent call first):
  File "/usr/local/miniconda/lib/python3.8/site-packages/sentry_sdk/worker.py", line 126 in _target
  File "/usr/local/miniconda/lib/python3.8/threading.py", line 870 in run
  File "/usr/local/miniconda/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/usr/local/miniconda/lib/python3.8/threading.py", line 890 in _bootstrap

Current thread 0x00007f3418bc6740 (most recent call first):
<no Python frame>

With our own atexit teardown:

210611-03:13:17,16 nipype.workflow IMPORTANT:
	 Closing Sentry client pre-shutdown...
 [sentry] DEBUG: Flushing HTTP transport
 [sentry] DEBUG: background worker got flush request
 [sentry] DEBUG: background worker flushed
 [sentry] DEBUG: Killing HTTP transport
 [sentry] DEBUG: background worker got kill request
Fatal Python error: Segmentation fault

Thread 0x00007f143a413700 (most recent call first):
  File "/usr/local/miniconda/lib/python3.8/threading.py", line 302 in wait
  File "/usr/local/miniconda/lib/python3.8/site-packages/sentry_sdk/_queue.py", line 180 in get
  File "/usr/local/miniconda/lib/python3.8/site-packages/sentry_sdk/worker.py", line 126 in _target
  File "/usr/local/miniconda/lib/python3.8/threading.py", line 870 in run
  File "/usr/local/miniconda/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/usr/local/miniconda/lib/python3.8/threading.py", line 890 in _bootstrap

Current thread 0x00007f14601c0740 (most recent call first):
<no Python frame>

Seems it's a thread spawned by sentry that's causing problems, regardless of how it's killed. Have a hunch that it's a result of getsentry/sentry-python#994, so pinning <1.0.0 (see https://github.com/getsentry/sentry-python/releases/tag/1.0.0).

@effigies
Copy link
Member Author

effigies commented Jun 11, 2021

Giving up for now. Will leave open in case someone wants to try to pick it up, but feel free to close it.

@effigies effigies force-pushed the enh/update-dockerfile branch from 1488f04 to 827a920 Compare June 12, 2021 01:13
@effigies
Copy link
Member Author

effigies commented Jun 16, 2021

@mgxd It's not Sentry... I think it may be nipype.

@mgxd
Copy link
Collaborator

mgxd commented Jun 21, 2021

this can be closed since #2409 is in

@mgxd mgxd closed this Jun 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mgxd mgxd mgxd left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@effigies @mgxd