Exceptions related to weakref in pyhf 0.6.2 #1506

alexander-held · 2021-06-30T11:12:36Z

Description

With pyhf 0.6.2 I see a long trace of weakref-related exceptions at the end of a program using certain parts of pyhf. I have not extensively tested which parts cause this behavior, but have so far observed it with pyhf.simplemodels.correlated_background / pyhf.simplemodels.uncorrelated_background. I can also confirm that other functionality like pyhf.set_backend("numpy") does not trigger it.

Expected Behavior

No stack of weakref exceptions after code finishes running.

Actual Behavior

Large stack of weakref-related exceptions.

Steps to Reproduce

Run a simple file using pyhf.simplemodels.uncorrelated_background, the example runs in a python:3.8-slim container with Python 3.8.11. I see the same locally in a conda environment with Python 3.8.10 on Mac OS. With pyhf 0.6.2 the stack of exceptions appears. When installing pyhf==0.6.1 and using pyhf.simplemodels.hepdata_like, I do not see any exceptions. When running via python -m pdb test.py locally to debug, I do no longer see the exception. The current pyhf master also shows the same issue.

$ cat test.py
import pyhf

model = pyhf.simplemodels.uncorrelated_background(
    signal=[5.0], bkg=[10.0], bkg_uncertainty=[2.0]
)

$ docker run -it --rm -v$PWD:/test python:3.8-slim bash
root@8fd8a5f229c8:/# pip install pyhf
Collecting pyhf
  Downloading pyhf-0.6.2-py3-none-any.whl (140 kB)
     |████████████████████████████████| 140 kB 3.5 MB/s
Collecting pyyaml>=5.1
  Downloading PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB)
     |████████████████████████████████| 662 kB 5.7 MB/s
Collecting scipy>=1.4.1
  Downloading scipy-1.7.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (28.4 MB)
     |████████████████████████████████| 28.4 MB 16.4 MB/s
Collecting jsonpatch>=1.15
  Downloading jsonpatch-1.32-py2.py3-none-any.whl (12 kB)
Collecting tqdm>=4.56.0
  Downloading tqdm-4.61.1-py2.py3-none-any.whl (75 kB)
     |████████████████████████████████| 75 kB 2.9 MB/s
Collecting click>=7.0
  Downloading click-8.0.1-py3-none-any.whl (97 kB)
     |████████████████████████████████| 97 kB 6.2 MB/s
Collecting jsonschema>=3.0.0
  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
     |████████████████████████████████| 56 kB 6.3 MB/s
Collecting jsonpointer>=1.9
  Downloading jsonpointer-2.1-py2.py3-none-any.whl (7.4 kB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/site-packages (from jsonschema>=3.0.0->pyhf) (57.0.0)
Collecting six>=1.11.0
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting attrs>=17.4.0
  Downloading attrs-21.2.0-py2.py3-none-any.whl (53 kB)
     |████████████████████████████████| 53 kB 4.1 MB/s
Collecting pyrsistent>=0.14.0
  Downloading pyrsistent-0.18.0-cp38-cp38-manylinux1_x86_64.whl (118 kB)
     |████████████████████████████████| 118 kB 18.9 MB/s
Collecting numpy<1.23.0,>=1.16.5
  Downloading numpy-1.21.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
     |████████████████████████████████| 15.7 MB 18.2 MB/s
Installing collected packages: six, pyrsistent, numpy, jsonpointer, attrs, tqdm, scipy, pyyaml, jsonschema, jsonpatch, click, pyhf
Successfully installed attrs-21.2.0 click-8.0.1 jsonpatch-1.32 jsonpointer-2.1 jsonschema-3.2.0 numpy-1.21.0 pyhf-0.6.2 pyrsistent-0.18.0 pyyaml-5.4.1 scipy-1.7.0 six-1.16.0 tqdm-4.61.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@8fd8a5f229c8:/# python test/test.py
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf03a0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf01f0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0280>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0430>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf00d0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0160>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0670>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf05e0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf04c0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0550>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0820>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0790>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0700>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf09d0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0940>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf08b0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0c10>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0b80>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0a60>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0af0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0dc0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0d30>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0ca0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0f70>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0ee0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dcf0e50>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dc821f0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dc82160>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dc82040>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dc820d0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dc823a0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dc82280>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dc82310>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dc82430>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types
Exception ignored in: <function WeakMethod.__new__.<locals>._cb at 0x7f7e8dc824c0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/weakref.py", line 58, in _cb
  File "/usr/local/lib/python3.8/weakref.py", line 74, in __eq__
TypeError: isinstance() arg 2 must be a type or tuple of types

Checklist

Run git fetch to get the most up to date version of master
Searched through existing Issues to confirm this is not a duplicate issue
Filled out the Description, Expected Behavior, Actual Behavior, and Steps to Reproduce sections above or have edited/removed them in a way that fully describes the issue

The text was updated successfully, but these errors were encountered:

kratsg · 2021-06-30T12:41:51Z

@alexander-held what does test/test.py look like? The weakref error here famously occurs with matplotlib involved. The other one is actually a bug in python 3.7.

alexander-held · 2021-06-30T12:43:09Z

It's this one:

import pyhf

model = pyhf.simplemodels.uncorrelated_background(
    signal=[5.0], bkg=[10.0], bkg_uncertainty=[2.0]
)

The reason that it is test/test.py and not just test.py is that I mount it into /test in the container.

I am using Python 3.8 in these tests, and have not tried 3.7.

alexander-held · 2021-06-30T13:19:19Z

The issue seems to be related to click. When I add a pip install click~=7.0 after pip install pyhf, it disappears.

matthewfeickert · 2021-06-30T16:43:19Z

Thanks for this report @alexander-held. Yeah, I'm not sure why we hadn't seen this in tests before but this is definitley related to the click v8.X release series, as even

python -m pip install --upgrade "click<8.0.1"

giving v8.0.0 doesn't help. :(

matthewfeickert · 2021-06-30T16:47:23Z

Copying discussion from Slack, @kratsg points out:

I'll dump what I do know about weakref and why this comes up. The crash occurs on these lines (https://github.com/python/cpython/blob/ae5dcf588c39915b43e391c738cf99f56a1b7fe2/Lib/weakref.py#L73-L78). This is because the WeakMethod is NoneType'd (because it got deleted during garbage collect?). To test this, I added a nice print statement print(other, WeakMethod, type(other), type(WeakMethod)).

When it doesn't crash, I see this
<weakref at 0x15ecf1200; dead> <class 'weakref.WeakMethod'> <class 'weakref.WeakMethod'> <class 'type'>
When it does crash, I see this
<weakref at 0x15ecf19e0; dead> None <class 'weakref.WeakMethod'> <class 'NoneType'>
So indeed, weakref.WeakMethod is NoneType too early.
Why is it being deleted too early? Why is this combination of our pyhf and click version causing this issue?

matthewfeickert · 2021-07-21T14:05:55Z

To me this looks more like a bug in Click that happened to get added in version 8 than an intentional change. Once it's discovered I think you might want to do a bug report to Click.

@henryiii seems to think this is a bug with Click.

@kratsg, as you were the one to look into this the most do you have a minimal failing example? If so, I can open up an Issue with Click.

matthewfeickert · 2021-07-21T16:21:59Z

@kratsg just mentioned

problem is primarily both Click and pyhf rely on weakrefs on some level, and there's not much one can do to strictly avoid so easily. So I just rewrote it [on Giordon's test branch] to drop WeakList and and use the built-ins.

Once this gets fixed we should probably do a patch release to fix this sooner than later.

matthewfeickert · 2021-08-18T13:58:36Z

@kratsg I'd advocate that once we get PR #1530 in that we make a patch release (I'll handle the release checklist work) and bump the current v0.6.3 board to v0.6.4

matthewfeickert added user request Request coming form a pyhf user bug Something isn't working labels Jun 30, 2021

matthewfeickert mentioned this issue Jul 1, 2021

docs: Mention that downgrading dependencies might be needed at times #1513

Closed

This was referenced Jul 21, 2021

fix: Remove WeakList in favor of built-ins #1530

Merged

docs: Add FAQ on reasons for need to downgrade dependencies #1529

Merged

alexander-held mentioned this issue Aug 9, 2021

The example.ipynb not working cabinetry/cabinetry-tutorials#8

Closed

kratsg mentioned this issue Aug 18, 2021

annoying error messages when exiting script using pyhf #1553

Closed

matthewfeickert assigned kratsg and matthewfeickert Aug 18, 2021

matthewfeickert closed this as completed in #1530 Aug 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exceptions related to weakref in pyhf 0.6.2 #1506

Exceptions related to weakref in pyhf 0.6.2 #1506

alexander-held commented Jun 30, 2021 •

edited by matthewfeickert

Loading

kratsg commented Jun 30, 2021

alexander-held commented Jun 30, 2021 •

edited

Loading

alexander-held commented Jun 30, 2021

matthewfeickert commented Jun 30, 2021

matthewfeickert commented Jun 30, 2021

matthewfeickert commented Jul 21, 2021 •

edited

Loading

matthewfeickert commented Jul 21, 2021 •

edited

Loading

matthewfeickert commented Aug 18, 2021

Exceptions related to weakref in pyhf 0.6.2 #1506

Exceptions related to weakref in pyhf 0.6.2 #1506

Comments

alexander-held commented Jun 30, 2021 • edited by matthewfeickert Loading

Description

Expected Behavior

Actual Behavior

Steps to Reproduce

Checklist

kratsg commented Jun 30, 2021

alexander-held commented Jun 30, 2021 • edited Loading

alexander-held commented Jun 30, 2021

matthewfeickert commented Jun 30, 2021

matthewfeickert commented Jun 30, 2021

matthewfeickert commented Jul 21, 2021 • edited Loading

matthewfeickert commented Jul 21, 2021 • edited Loading

matthewfeickert commented Aug 18, 2021

alexander-held commented Jun 30, 2021 •

edited by matthewfeickert

Loading

alexander-held commented Jun 30, 2021 •

edited

Loading

matthewfeickert commented Jul 21, 2021 •

edited

Loading

matthewfeickert commented Jul 21, 2021 •

edited

Loading