Use incremental mode in python evaluation test cases

[JukkaL]

The Python evaluation test cases (which don't actually run the code if there are type errors...) use full stubs for builtins and typing. Using incremental mode would likely speed them up significantly, as processing stubs probably takes the majority of time. These tests are the long pole in the full test suite, so this could give a nice speed boost.

[gvanrossum]

I looked into this for a few minutes and one problem is that currently each test is run in a clean directory. There currently is no way to change the name of the cache directory, and it's always .mypy_cache relative to the current directory. So we'd need something for this. (I've filed #1414 to track it.)

[emmatyping]

I just tried making the pythoneval tests use a shared cache directory via this diff:

diff --git a/mypy/test/testpythoneval.py b/mypy/test/testpythoneval.py
index cf4cbafd..9c207d5d 100644
--- a/mypy/test/testpythoneval.py
+++ b/mypy/test/testpythoneval.py
@@ -14,7 +14,7 @@ import os
 import os.path
 import re
 import sys
-
+from tempfile import TemporaryDirectory
 import pytest  # type: ignore  # no pytest in typeshed

 from typing import List
@@ -37,12 +37,12 @@ class PythonEvaluationSuite(DataSuite):
              'pythoneval-asyncio.test']
     base_path = test_temp_dir
     optional_out = True
-
+    cache_dir = TemporaryDirectory()
     def run_case(self, testcase: DataDrivenTestCase) -> None:
-        test_python_evaluation(testcase)
+        test_python_evaluation(testcase, os.path.join(self.cache_dir.name, '.mypy_cache'))


-def test_python_evaluation(testcase: DataDrivenTestCase) -> None:
+def test_python_evaluation(testcase: DataDrivenTestCase, cache_dir: str) -> None:
     """Runs Mypy in a subprocess.

     If this passes without errors, executes the script again with a given Python
@@ -71,6 +71,7 @@ def test_python_evaluation(testcase: DataDrivenTestCase) -> None:
     with open(program_path, 'w') as file:
         for s in testcase.input:
             file.write('{}\n'.format(s))
+    mypy_cmdline.append('--cache-dir={}'.format(cache_dir))
     output = []
     # Type check the program.
     out, err, returncode = api.run(mypy_cmdline)

However I get a failure (consistently) with:

_______________ testCanConvertTypedDictToAnySuperclassOfMapping _______________
[gw8] win32 -- Python 3.6.5 C:\Python36\python.exe
data: C:\Users\ethanhs\Documents\mypy\test-data\unit\pythoneval.test:1081:
..\..\..\..\Documents\mypy\mypy\test\testpythoneval.py:42: in run_case
    test_python_evaluation(testcase, os.path.join(self.cache_dir.name, '.mypy_cache'))
..\..\..\..\Documents\mypy\mypy\test\testpythoneval.py:92: in test_python_evaluation
    testcase.file, testcase.line))
..\..\..\..\Documents\mypy\mypy\test\helpers.py:98: in assert_string_arrays_equal
    raise AssertionError(msg)
E   AssertionError: Invalid output (C:\Users\ethanhs\Documents\mypy\test-data\unit\pythoneval.test, line 1081)
---------------------------- Captured stderr call -----------------------------
Expected:
  _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: error: Incompatible...
  _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note: Following mem...
  _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note:     Expected:
  _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note:         def __iter__(self) -> Iterator[int] (diff)
  _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note:     Got:
  _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note:         def __iter__(self) -> Iterator[str] (diff)
Actual:
  _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: error: Incompatible...
  _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note: Following mem...
  _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note:     Expected:
  _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note:         def () -> Iterator[int] (diff)
  _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note:     Got:
  _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note:         def () -> Iterator[str] (diff)

Alignment of first line difference:
  E: ...y:11: note:         def __iter__(self) -> Iterator[int]
  A: ...y:11: note:         def () -> Iterator[int]
                                ^

this appears to be an incremental bug?

[ilevkivskyi]

Yes, it looks like name of a method is lost during (de-)serialisation. If you have time and desire, you could fix this yourself, otherwise create a separate issue about this.

[ilevkivskyi]

@ethanhs I think the fail you see is the same as #4772, where also method name disappears after loading from cache

[emmatyping]

@ilevkivskyi that makes sense. I'm afraid I'm not familiar enough with the intricacies of incremental to debug this one. I'll bump the priority of that issue up however.

[ilevkivskyi]

@ethanhs #5109 should unblock this.