bug with loading dependencies when running mypy

[ryanking]

Pants version
2.16, but also was happening on 2.15

OS
Mac

I have been trying to hunt down a bug in pants for a few weeks and finally have a reproduction, but don't know what do do with it.

From time to time, a mypy run will fail due to missing dependencies. Dependencies which are definitely configured and findable. Often a re-run will succeed, which has made this very hard to reproduce.

I now found a case where running a single file will fail, but running with :: will succeed. I have both sandboxes and when I edit __files.txt in the latter it passes. However I don't know what to look for in these sandboxes to figure out what's going on.

Failure from the "bad" sandbox:

.cache/pex_root/venvs/e1862d6434ffd1c4230c239d7d17d05591664247/4e8b276cfa3c3338d49f8656ee098b9ee6162291/lib/python3.8/site-packages/botocore-stubs/compat.pyi:20:1: error: Library stubs not installed for "six.moves.http_client"  [import]
    from six.moves import http_client
    ^
.cache/pex_root/venvs/e1862d6434ffd1c4230c239d7d17d05591664247/4e8b276cfa3c3338d49f8656ee098b9ee6162291/lib/python3.8/site-packages/botocore-stubs/compat.pyi:20:1: note: Hint: "python3 -m pip install types-six"
.cache/pex_root/venvs/e1862d6434ffd1c4230c239d7d17d05591664247/4e8b276cfa3c3338d49f8656ee098b9ee6162291/lib/python3.8/site-packages/botocore-stubs/compat.pyi:20:1: note: (or run "mypy --install-types" to install all missing stub packages)
.cache/pex_root/venvs/e1862d6434ffd1c4230c239d7d17d05591664247/4e8b276cfa3c3338d49f8656ee098b9ee6162291/lib/python3.8/site-packages/botocore-stubs/compat.pyi:20:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
.cache/pex_root/venvs/e1862d6434ffd1c4230c239d7d17d05591664247/4e8b276cfa3c3338d49f8656ee098b9ee6162291/lib/python3.8/site-packages/botocore-stubs/compat.pyi:20:1: error: Library stubs not installed for "six.moves"  [import]
    from six.moves import http_client
    ^
src/django_apps/counseling/google_calendar.py:74:14: error: Module has no attribute "parser"  [attr-defined]
                (dateutil.parser.parse(e['start']), dateutil.parser.parse(e['end']))
                 ^~~~~~~~~~~~~~~
Found 3 errors in 2 files (checked 1 source file)

Success from the good sandbox, edited to run only the file in question:

Success: no issues found in 1 source file

A fun complication is that the mypy cache appears to get poisoned. If I have a failed run on the "bad" sandbox, the good one will then fail. If I reset the cache it will work.

[kaos]

Huh, interesting observation regarding the cache. I too have seen this issue on and off for months without being able to pin it down.

[ryanking]

Yeah I think there are potentially two problems here:

1. I somehow, sometimes get a sandbox that is missing some dependencies
2. incorrect results sometimes get in the mypy cache. this may not be a pants specific problem but might be obscuring problems in pants

[ryanking]

Spent some time debugging this with @stuhood in slack DM. A few things I have found–

1. It does not appear to be related to caching. I get the same behavior with pants --no-local-cache --no-remote-cache-read --no-remote-cache-write --no-pantsd check and incremental = False in mypy.ini
2. This might be very specific to dateutil, as changing from import dateutil to from dateutil import parser seems to have solved the problem :tableflip:

[achimnol]

I'm also experiencing this issue here:
https://github.com/lablup/backend.ai/actions/runs/5091896560/jobs/9152521418?pr=1022

[achimnol]

This may be related with the "--changed" subsystem.

[achimnol]

Patching the PR like

diff --git a/BUILD b/BUILD
index c9bda9306..8d74b8143 100644
--- a/BUILD
+++ b/BUILD
@@ -23,7 +23,7 @@ python_requirements(
         "types-PyYAML": ["yaml"],
         "types-python-dateutil": ["dateutil", "dateutil.parser", "dateutil.tz"],
         "types-redis": ["redis"],
-        "types-six": ["six"],
+        "types-six": ["six", "graphql", "promise"],
         "types-tabulate": ["tabulate"],
     },
 )

resolves the issue.

So this seems to be an issue of mismatch in transitive dependency resolution/requirement in mypy and pants.

[achimnol]

Still happens, especially with external package (graphql, promise, _pytest) imports where those have py.typed.

[achimnol]

I had to do it like lablup/backend.ai@c7bb82e

diff --git a/BUILD b/BUILD
index f85cfa7b..d9b923ba 100644
--- a/BUILD
+++ b/BUILD
@@ -23,8 +23,13 @@ python_requirements(
         "types-PyYAML": ["yaml"],
         "types-python-dateutil": ["dateutil", "dateutil.parser", "dateutil.tz"],
         "types-redis": ["redis"],
-        "types-setuptools": ["setuptools", "pkg_resources", "_pytest"],
-        "types-six": ["six", "graphql", "promise", "_pytest", "ai.backend"],
+        "types-setuptools": [
+            "setuptools",
+            "pkg_resources",
+            "ai.backend.testutils",
+            "ai.backend.test",
+        ],
+        "types-six": ["six", "graphql", "promise", "ai.backend.manager.models"],
         "types-tabulate": ["tabulate"],
     },
 )

[kuza55]

I'm running into some flakiness with mypy too. Blowing away the cache is unnecessary, merely re-running pants check is sufficient. I am running into this both in pre-commit as well as just running pants check from the command line.

I am getting this error with other modules than mentioned here:

.cache/pex_root/venvs/d17349559f692efab258d5f40c73f81d7382b44c/2540ac6a914a2a78bfd89ca0676ffdc7f0dfe44f/lib/python3.11/site-packages/pandas-stubs/io/formats/style_render.pyi:11: error: Cannot find implementation or library stub for module named "jinja2"  [import-not-found]
.cache/pex_root/venvs/d17349559f692efab258d5f40c73f81d7382b44c/2540ac6a914a2a78bfd89ca0676ffdc7f0dfe44f/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py:57: error: Library stubs not installed for "requests.models"  [import-untyped]

I have a separate lockfile for mypy due to some issue with the version of mypy that is bundled with pants, no idea if related.

The error seems relatively new, I don't recall seeing anything like this pre 2.19.0

[isra17]

Having a similar bug with opensearch-py and mypy complaining about types-simplejson or type-six. Happens only once, and then it succeed, even with --no-pantsd and --no-local-cache. I am unfortunately unable to reproduce reliably or find anything specific in the failure case.

[isra17]

Having a similar bug with opensearch-py and mypy complaining about types-simplejson or type-six. Happens only once, and then it succeed, even with --no-pantsd and --no-local-cache. I am unfortunately unable to reproduce reliably or find anything specific in the failure case.

Not sure if related, but looking at Pants log, it seems like the the requirements.pex get cancelled:

20:28:53.99 [INFO] Starting: Building 3 requirements for mypy.pex from the pants/3rdparty/python/mypy.lock resolve: mypy==1.7.0, pydantic==1.10.14, sqlalchemy[mypy]==1.4.51
20:28:55.40 [INFO] Starting: Building 66 requirements for requirements.pex from the pants/3rdparty/python/default.lock resolve: authlib==0.15.4; python_full_version >= "3.11.3" and python_full_version < "3.12.0", bcrypt==4.1.2; p... (6017 characters truncated)
20:28:55.62 [INFO] Canceled: Building 66 requirements for requirements.pex from the pants/3rdparty/python/default.lock resolve: authlib==0.15.4; python_full_version >= "3.11.3" and python_full_version < "3.12.0", bcrypt==4.1.2; p... (6017 characters truncated)
20:28:55.62 [INFO] Starting: Building 66 requirements for requirements.pex from the pants/3rdparty/python/default.lock resolve: authlib==0.15.4; python_full_version >= "3.11.3" and python_full_version < "3.12.0", bcrypt==4.1.2; p... (6017 characters truncated)
20:28:59.10 [INFO] Completed: Building 3 requirements for mypy.pex from the pants/3rdparty/python/mypy.lock resolve: mypy==1.7.0, pydantic==1.10.14, sqlalchemy[mypy]==1.4.51
20:29:10.23 [INFO] Completed: Building 66 requirements for requirements.pex from the pants/3rdparty/python/default.lock resolve: authlib==0.15.4; python_full_version >= "3.11.3" and python_full_version < "3.12.0", bcrypt==4.1.2; p... (6017 characters truncated)

Now, even successful run has the Cancelled: Building ..., but the one run that failed has a [DEBUG] Canceled: setup_sandbox right before. Am I getting warmer?

[isra17]

I'm actually thinking again, and doesn't it look like mypy is parsing these third-parties file directly?

Playing in a venv with a similar configuration, I can run mypy without any of my dependency's dependencies and mypy will set them as Any. I don't think there should be error from missing import in third-party libraries.

This might be related to python/mypy#12162

[isra17]

Now, even successful run has the Cancelled: Building ..., but the one run that failed has a [DEBUG] Canceled: setup_sandbox right before. Am I getting warmer?

Disregard this, I was able to get another failure and it didn't have this log.

[isra17]

I found it. This is an issue with mypy cache. If it runs again without some 4th-party dependencies, mypy will raise an error instead of settings these import as Any as usual. I filed a bug and easy repro here: python/mypy#17049

From pants side of things, I don't know if there could be a way to make the mypy cache specific to the venv and avoid reusing the same cache with different dependencies.

[isra17]

FYI,

# pants.toml
[mypy]
args = ["--no-incremental"]

Should fix it until this bug get fixed in mypy.

[hauntsaninja]

python/mypy#17420 should fix a thing here

[cburroughs]

Thank you for the work on python/mypy#17420 !

With both 1.10.1 and 1.11.0 I'm still unable to enable the incremental cache on our repo due to errors like:

foon.py:1:1: error: Cannot find implementation or library stub for module named "cvxpy.atoms.sum_squares"  [import-not-found]
    from dataclasses import dataclass
    ^
Found 126 errors in 41 files (checked 12 source files)

I have not been able to find a reproduction case yet. Note that here cvxpy.atoms.sum_squares is obviously not dataclasses so this seems like a different error. Like the original case, a re-run fixes it. All of the examples so far are with the same package (cvxpy) which I find puzzling.

I'm interested if others have had more luck.

[isra17]

Yes, sorry didn't have much time to come back, but I had similar issues...

Unfortunately, unlike the previous instance of the bug, I'm unable to make a minimal repro. Like before, the cache end up in a state that will yield an error, but succeed on the next run. If I take a copy and reuse it I can reproduce the failure. (I can't share it since it's from our internal project though).