PyPy3.10 test failures due to datetime mocking: TypeError: isinstance() arg 2 must be a class, type, or tuple of classes and types, got MagicMock

[mgorny]

Description of issue or feature request:

The test suite currently fails 3 tests on PyPy3.10 7.3.17. This is due to a known limitation of pure Python datetime module from CPython. However, CPython normally covers it with a C extension, hiding the problem, but PyPy suffers from it.

Current behavior:

$ tox -e pypy310
pypy310: commands[0]> python3 --version
Python 3.10.14 (39dc8d3c85a7, Aug 29 2024, 13:06:01)
[PyPy 7.3.17 with GCC 13.3.1 20240614]
pypy310: commands[1]> python3 -m coverage run aggregate_tests.py
.................................................................................................................................................EEE.....................................
======================================================================
ERROR: test_expired_metadata (test_updater_top_level_update.TestRefresh)
Verifies that expired local timestamp/snapshot can be used for
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pypy3.10/unittest/mock.py", line 1379, in patched
    return func(*newargs, **newkeywargs)
  File "/tmp/python-tuf/tests/test_updater_top_level_update.py", line 752, in test_expired_metadata
    now = datetime.datetime.now(timezone.utc)
  File "/usr/lib/pypy3.10/unittest/mock.py", line 1114, in __call__
    return self._mock_call(*args, **kwargs)
  File "/usr/lib/pypy3.10/unittest/mock.py", line 1118, in _mock_call
    return self._execute_mock_call(*args, **kwargs)
  File "/usr/lib/pypy3.10/unittest/mock.py", line 1188, in _execute_mock_call
    return self._mock_wraps(*args, **kwargs)
  File "/usr/lib/pypy3.10/datetime.py", line 1720, in now
    return cls.fromtimestamp(t, tz)
  File "/usr/lib/pypy3.10/datetime.py", line 1709, in fromtimestamp
    return cls._fromtimestamp(t, tz is not None, tz)
  File "/usr/lib/pypy3.10/datetime.py", line 1698, in _fromtimestamp
    result = tz.fromutc(result)
  File "/usr/lib/pypy3.10/datetime.py", line 2362, in fromutc
    if isinstance(dt, datetime):
TypeError: isinstance() arg 2 must be a class, type, or tuple of classes and types, got MagicMock

======================================================================
ERROR: test_expired_timestamp_snapshot_rollback (test_updater_top_level_update.TestRefresh)
Verifies that rollback protection is done even if local timestamp has expired.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pypy3.10/unittest/mock.py", line 1379, in patched
    return func(*newargs, **newkeywargs)
  File "/tmp/python-tuf/tests/test_updater_top_level_update.py", line 358, in test_expired_timestamp_snapshot_rollback
    now = datetime.datetime.now(timezone.utc)
  File "/usr/lib/pypy3.10/unittest/mock.py", line 1114, in __call__
    return self._mock_call(*args, **kwargs)
  File "/usr/lib/pypy3.10/unittest/mock.py", line 1118, in _mock_call
    return self._execute_mock_call(*args, **kwargs)
  File "/usr/lib/pypy3.10/unittest/mock.py", line 1188, in _execute_mock_call
    return self._mock_wraps(*args, **kwargs)
  File "/usr/lib/pypy3.10/datetime.py", line 1720, in now
    return cls.fromtimestamp(t, tz)
  File "/usr/lib/pypy3.10/datetime.py", line 1709, in fromtimestamp
    return cls._fromtimestamp(t, tz is not None, tz)
  File "/usr/lib/pypy3.10/datetime.py", line 1698, in _fromtimestamp
    result = tz.fromutc(result)
  File "/usr/lib/pypy3.10/datetime.py", line 2362, in fromutc
    if isinstance(dt, datetime):
TypeError: isinstance() arg 2 must be a class, type, or tuple of classes and types, got MagicMock

======================================================================
ERROR: test_expired_timestamp_version_rollback (test_updater_top_level_update.TestRefresh)
Verifies that local timestamp is used in rollback checks even if it is expired.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pypy3.10/unittest/mock.py", line 1379, in patched
    return func(*newargs, **newkeywargs)
  File "/tmp/python-tuf/tests/test_updater_top_level_update.py", line 322, in test_expired_timestamp_version_rollback
    now = datetime.datetime.now(timezone.utc)
  File "/usr/lib/pypy3.10/unittest/mock.py", line 1114, in __call__
    return self._mock_call(*args, **kwargs)
  File "/usr/lib/pypy3.10/unittest/mock.py", line 1118, in _mock_call
    return self._execute_mock_call(*args, **kwargs)
  File "/usr/lib/pypy3.10/unittest/mock.py", line 1188, in _execute_mock_call
    return self._mock_wraps(*args, **kwargs)
  File "/usr/lib/pypy3.10/datetime.py", line 1720, in now
    return cls.fromtimestamp(t, tz)
  File "/usr/lib/pypy3.10/datetime.py", line 1709, in fromtimestamp
    return cls._fromtimestamp(t, tz is not None, tz)
  File "/usr/lib/pypy3.10/datetime.py", line 1698, in _fromtimestamp
    result = tz.fromutc(result)
  File "/usr/lib/pypy3.10/datetime.py", line 2362, in fromutc
    if isinstance(dt, datetime):
TypeError: isinstance() arg 2 must be a class, type, or tuple of classes and types, got MagicMock

----------------------------------------------------------------------
Ran 185 tests in 6.709s

FAILED (errors=3)
pypy310: exit 1 (7.69 seconds) /tmp/python-tuf/tests> python3 -m coverage run aggregate_tests.py pid=490151
  pypy310: FAIL code 1 (7.78=setup[0.07]+cmd[0.02,7.69] seconds)
  evaluation failed :( (7.84 seconds)

Expected behavior:

Tests passing :-).

One possible solution would be to use a more complete time mocking solution, e.g. freezegun. I've been able to quickly confirm that it solves the problem, using a quick patch:

 from tests.repository_simulator import RepositorySimulator
 from tuf.api.exceptions import (
@@ -306,8 +308,7 @@ def test_new_timestamp_unsigned(self) -> None:
 
         self._assert_files_exist([Root.type])
 
-    @patch.object(datetime, "datetime", wraps=datetime.datetime)
-    def test_expired_timestamp_version_rollback(self, mock_time: Mock) -> None:
+    def test_expired_timestamp_version_rollback(self) -> None:
         """Verifies that local timestamp is used in rollback checks even if it is expired.
 
         The timestamp updates and rollback checks are performed
@@ -331,14 +332,12 @@ def test_expired_timestamp_version_rollback(self, mock_time: Mock) -> None:
 
         self.sim.timestamp.version = 1
 
-        mock_time.now.return_value = datetime.datetime.now(
-            timezone.utc
-        ) + datetime.timedelta(days=18)
-        patcher = patch("datetime.datetime", mock_time)
-        # Check that a rollback protection is performed even if
-        # local timestamp has expired
-        with patcher, self.assertRaises(BadVersionNumberError):
-            self._run_refresh()
+        with freezegun.freeze_time(datetime.datetime.now( timezone.utc) +
+                                   datetime.timedelta(days=18)):
+            # Check that a rollback protection is performed even if
+            # local timestamp has expired
+            with self.assertRaises(BadVersionNumberError):
+                self._run_refresh()
 
         self._assert_version_equals(Timestamp.type, 2)
 

If you agree with this approach, I can make a proper pull request.