mocker.patch fails to find tensorlib _setup attribute for Python 3.11

[matthewfeickert]

Now that TensorFlow v2.12.0 is out with Python 3.11 support I've been working to add Python 3.11 to the CI. Everything passes with the exception of the tests/test_tensor.py test_tensorlib_setup tests

pyhf/tests/test_tensor.py

Lines 607 to 618 in 89b3e40

	@pytest.mark.parametrize(
	'tensorlib',
	['numpy_backend', 'jax_backend', 'pytorch_backend', 'tensorflow_backend'],
	)
	@pytest.mark.parametrize('precision', ['64b', '32b'])
	def test_tensorlib_setup(tensorlib, precision, mocker):
	tb = getattr(pyhf.tensor, tensorlib)(precision=precision)
	func = mocker.patch(f'pyhf.tensor.{tensorlib}._setup')
	assert func.call_count == 0
	pyhf.set_backend(tb)
	assert func.call_count == 1

which all fail on

pyhf/tests/test_tensor.py

Line 615 in 89b3e40

func = mocker.patch(f'pyhf.tensor.{tensorlib}._setup')

with errors like

___________________ test_tensorlib_setup[64b-numpy_backend] ____________________

tensorlib = 'numpy_backend', precision = '64b'
mocker = <pytest_mock.plugin.MockerFixture object at 0x7fc9557f2e90>

    @pytest.mark.parametrize(
        'tensorlib',
        ['numpy_backend', 'jax_backend', 'pytorch_backend', 'tensorflow_backend'],
    )
    @pytest.mark.parametrize('precision', ['64b', '32b'])
    def test_tensorlib_setup(tensorlib, precision, mocker):
        tb = getattr(pyhf.tensor, tensorlib)(precision=precision)
    
>       func = mocker.patch(f'pyhf.tensor.{tensorlib}._setup')

mocker     = <pytest_mock.plugin.MockerFixture object at 0x7fc9557f2e90>
precision  = '64b'
tb         = <pyhf.tensor.numpy_backend.numpy_backend object at 0x7fca3444fc80>
tensorlib  = 'numpy_backend'

tests/test_tensor.py:605: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/hostedtoolcache/Python/3.11.2/x64/lib/python3.11/site-packages/pytest_mock/plugin.py:404: in __call__
    return self._start_patch(
        autospec   = None
        create     = False
        kwargs     = {}
        new        = sentinel.DEFAULT
        new_callable = None
        self       = <pytest_mock.plugin.MockerFixture._Patcher object at 0x7fc9557f00d0>
        spec       = None
        spec_set   = None
        target     = 'pyhf.tensor.numpy_backend._setup'
/opt/hostedtoolcache/Python/3.11.2/x64/lib/python3.11/site-packages/pytest_mock/plugin.py:214: in _start_patch
    mocked = p.start()  # type: unittest.mock.MagicMock
        args       = ('pyhf.tensor.numpy_backend._setup',)
        kwargs     = {'autospec': None, 'create': False, 'new': sentinel.DEFAULT, 'new_callable': None, ...}
        mock_func  = <function patch at 0x7fcb82251580>
        p          = <unittest.mock._patch object at 0x7fc9557f2790>
        self       = <pytest_mock.plugin.MockerFixture._Patcher object at 0x7fc9557f00d0>
        warn_on_mock_enter = True
/opt/hostedtoolcache/Python/3.11.2/x64/lib/python3.11/unittest/mock.py:1585: in start
    result = self.__enter__()
        self       = <unittest.mock._patch object at 0x7fc9557f2790>
/opt/hostedtoolcache/Python/3.11.2/x64/lib/python3.11/unittest/mock.py:1437: in __enter__
    original, local = self.get_original()
        autospec   = None
        kwargs     = {}
        new        = sentinel.DEFAULT
        new_callable = None
        self       = <unittest.mock._patch object at 0x7fc9557f2790>
        spec       = None
        spec_set   = None
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <unittest.mock._patch object at 0x7fc9557f2790>

    def get_original(self):
        target = self.getter()
        name = self.attribute
    
        original = DEFAULT
        local = False
    
        try:
            original = target.__dict__[name]
        except (AttributeError, KeyError):
            original = getattr(target, name, DEFAULT)
        else:
            local = True
    
        if name in _builtins and isinstance(target, ModuleType):
            self.create = True
    
        if not self.create and original is DEFAULT:
>           raise AttributeError(
                "%s does not have the attribute %r" % (target, name)
            )
E           AttributeError: <module 'pyhf.tensor.numpy_backend' from '/opt/hostedtoolcache/Python/3.11.2/x64/lib/python3.11/site-packages/pyhf/tensor/numpy_backend.py'> does not have the attribute '_setup'

local      = False
name       = '_setup'
original   = sentinel.DEFAULT
self       = <unittest.mock._patch object at 0x7fc9557f2790>
target     = <module 'pyhf.tensor.numpy_backend' from '/opt/hostedtoolcache/Python/3.11.2/x64/lib/python3.11/site-packages/pyhf/tensor/numpy_backend.py'>

/opt/hostedtoolcache/Python/3.11.2/x64/lib/python3.11/unittest/mock.py:1410: AttributeError

I'm not sure why as

import pyhf
import pyhf.tensor.numpy_backend
pyhf.tensor.numpy_backend._setup  # <function numpy_backend._setup at 0x7f8466564680>

and this is a Python 3.11 error only. :?

[matthewfeickert]

From chat @kratsg mentioned the following:

No, that's a change to inspect in python 3.11

https://docs.python.org/3/library/unittest.mock.html

The only exceptions are magic methods and attributes (those that have leading and trailing double underscores). Mock doesn’t create these but instead raises an AttributeError. This is because the interpreter will often implicitly request these methods, and gets very confused to get a new Mock object when it expects a magic method. If you need magic method support see magic methods.

I suspect you need to use name-mangling at that point.

[matthewfeickert]

Hm. What is confusing is that this isn't listed in the docs at all for the unittest.mock.Mock class and the same information exists for previous releases. So I'm not sure what has changed.

[matthewfeickert]

Yeah, it is definitely an issue between unittest.mock in Python 3.10 and Python 3.11 that isn't documented in the Python 3.11 changes, as the following script

# mock_debug.py
from unittest.mock import patch

import pyhf

tensor_backend = getattr(pyhf.tensor, "numpy_backend")(precision="64b")

with patch("pyhf.tensor.numpy_backend._setup") as mock_func:
    print(f"{mock_func.call_count=}")
    pyhf.set_backend(tensor_backend)
    print(f"{mock_func.call_count=}")

for Python 3.10 works

$ python --version --version && python /tmp/mock_debug.py 
Python 3.10.6 (main, Sep 20 2022, 17:04:13) [GCC 11.2.0]
mock_func.call_count=0
mock_func.call_count=1

but for Python 3.11 fails

$ python --version --version && python /tmp/mock_debug.py 
Python 3.11.1 (main, Feb 13 2023, 22:31:08) [GCC 11.3.0]
Traceback (most recent call last):
  File "/tmp/mock_debug.py", line 7, in <module>
    with patch("pyhf.tensor.numpy_backend._setup") as mock_func:
  File "/home/feickert/.pyenv/versions/3.11.1/lib/python3.11/unittest/mock.py", line 1437, in __enter__
    original, local = self.get_original()
                      ^^^^^^^^^^^^^^^^^^^
  File "/home/feickert/.pyenv/versions/3.11.1/lib/python3.11/unittest/mock.py", line 1410, in get_original
    raise AttributeError(
AttributeError: <module 'pyhf.tensor.numpy_backend' from '/home/feickert/Code/GitHub/pyhf/src/pyhf/tensor/numpy_backend.py'> does not have the attribute '_setup'

with the same AttributeError that we see in the failing test.

The only registered change in the What's New in Python 3.11 for unittest

Added methods enterContext() and enterClassContext() of class TestCase, method enterAsyncContext() of class IsolatedAsyncioTestCase and function unittest.enterModuleContext(). (Contributed by Serhiy Storchaka in bpo-45046.)

so I'm confused as to the change in behavior.

[matthewfeickert]

Okay the following works across both Python 3.10 and Python 3.11:

# mock_debug.py
from unittest.mock import patch

import pyhf

tensor_backend = getattr(pyhf.tensor, "numpy_backend")(precision="64b")

with patch.object(pyhf.tensor.numpy_backend, "_setup") as mock_func:
    print(f"{mock_func.call_count=}")  # mock_func.call_count=0
    pyhf.set_backend(tensor_backend)
    print(f"{mock_func.call_count=}")  # mock_func.call_count=1

[jiasli]

Thanks @matthewfeickert, your investigation is really helpful! We are facing a similar issue in our project, so I found this issue.

Finally, I discovered our issue is caused by python/cpython#18544 which changes the logic of unittest.mock._get_target to use pkgutil.resolve_name. This changes how the 'pyhf.tensor.numpy_backend' string is resolved:

import pkgutil

# _dot_lookup and _importer are copied from Python 3.10
def _dot_lookup(thing, comp, import_path):
    try:
        return getattr(thing, comp)
    except AttributeError:
        __import__(import_path)
        return getattr(thing, comp)


def _importer(target):
    components = target.split('.')
    import_path = components.pop(0)
    thing = __import__(import_path)

    for comp in components:
        import_path += ".%s" % comp
        thing = _dot_lookup(thing, comp, import_path)
    return thing


print(pkgutil.resolve_name('pyhf.tensor.numpy_backend'))
print(_importer('pyhf.tensor.numpy_backend'))

Output:

<module 'pyhf.tensor.numpy_backend' from 'D:\\cli\\py311\\Lib\\site-packages\\pyhf\\tensor\\numpy_backend.py'>
<class 'pyhf.tensor.numpy_backend.numpy_backend'>

Because pkgutil.resolve_name resolves 'pyhf.tensor.numpy_backend' to pyhf/tensor/numpy_backend.py module and that module doesn't have _setup, unittest.mock.patch raises that AttributeError. It seems pkgutil.resolve_name doesn't take the "re-binding" code into consideration:

pyhf/src/pyhf/__init__.py

Line 1 in bd8c21a

from pyhf.tensor import BackendRetriever as tensor

pyhf/src/pyhf/tensor/__init__.py

Line 24 in bd8c21a

self.numpy_backend = numpy_backend

[matthewfeickert]

Thanks very much for this great summary @jiasli! 🙌 I'll have to revisit this myself in the near future.