Regression: "protobuf type stubs obscure other imports from google.*" is a problem again

[simon-liebehenschel]

Here we go again #5800

Failing code

from google.cloud.firestore import AsyncClient

Log

mypy.....................................................................Failed
- hook id: mypy
- exit code: 1
Installing missing stub packages:
/opt/pysetup/.venv/bin/python -m pip install types-Pillow types-PyYAML types-Pygments types-google-cloud-ndb types-protobuf types-pyOpenSSL types-python-dateutil types-redis types-regex types-requests types-stripe types-tqdm types-ujson
Collecting types-Pillow
  Downloading types_Pillow-9.4.0.17-py3-none-any.whl (46 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.3/46.3 kB 3.6 MB/s eta 0:00:00
Collecting types-PyYAML
  Downloading types_PyYAML-6.0.12.8-py3-none-any.whl (14 kB)
Collecting types-Pygments
  Downloading types_Pygments-2.14.0.6-py3-none-any.whl (16 kB)
Collecting types-google-cloud-ndb
  Downloading types_google_cloud_ndb-2.1.0.5-py3-none-any.whl (16 kB)
Collecting types-protobuf
  Downloading types_protobuf-4.22.0.0-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.3/61.3 kB 36.4 MB/s eta 0:00:00
Collecting types-pyOpenSSL
  Downloading types_pyOpenSSL-23.0.0.4-py3-none-any.whl (6.9 kB)
Collecting types-python-dateutil
  Downloading types_python_dateutil-2.8.19.10-py3-none-any.whl (9.1 kB)
Collecting types-redis
  Downloading types_redis-4.5.1.4-py3-none-any.whl (55 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.4/55.4 kB 184.9 MB/s eta 0:00:00
Collecting types-regex
  Downloading types_regex-2022.10.31.6-py3-none-any.whl (5.0 kB)
Collecting types-requests
  Downloading types_requests-2.28.11.15-py3-none-any.whl (14 kB)
Collecting types-stripe
  Downloading types_stripe-3.5.2.8-py3-none-any.whl (52 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.4/52.4 kB 191.3 MB/s eta 0:00:00
Collecting types-tqdm
  Downloading types_tqdm-4.65.0.0-py3-none-any.whl (18 kB)
Collecting types-ujson
  Downloading types_ujson-5.7.0.1-py3-none-any.whl (2.2 kB)
Collecting types-docutils
  Downloading types_docutils-0.19.1.6-py3-none-any.whl (16 kB)
Collecting types-setuptools
  Downloading types_setuptools-67.6.0.0-py3-none-any.whl (33 kB)
Requirement already satisfied: cryptography>=35.0.0 in /opt/pysetup/.venv/lib/python3.10/site-packages (from types-pyOpenSSL) (39.0.2)
Collecting types-urllib3<1.27
  Downloading types_urllib3-1.26.25.8-py3-none-any.whl (15 kB)
Requirement already satisfied: cffi>=1.12 in /opt/pysetup/.venv/lib/python3.10/site-packages (from cryptography>=35.0.0->types-pyOpenSSL) (1.15.1)
Requirement already satisfied: pycparser in /opt/pysetup/.venv/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=35.0.0->types-pyOpenSSL) (2.21)
Installing collected packages: types-urllib3, types-ujson, types-tqdm, types-stripe, types-setuptools, types-regex, types-PyYAML, types-python-dateutil, types-protobuf, types-Pillow, types-google-cloud-ndb, types-docutils, types-requests, types-Pygments, types-pyOpenSSL, types-redis
Successfully installed types-Pillow-9.4.0.17 types-PyYAML-6.0.12.8 types-Pygments-2.14.0.6 types-docutils-0.19.1.6 types-google-cloud-ndb-2.1.0.5 types-protobuf-4.22.0.0 types-pyOpenSSL-23.0.0.4 types-python-dateutil-2.8.19.10 types-redis-4.5.1.4 types-regex-2022.10.31.6 types-requests-2.28.11.15 types-setuptools-67.6.0.0 types-stripe-3.5.2.8 types-tqdm-4.65.0.0 types-ujson-5.7.0.1 types-urllib3-1.26.25.8
foobar.py:16:1: error: Library stubs not installed for
"google.cloud.firestore"  [import]
    from google.cloud.firestore import AsyncClient
    ^
foobar.py:16:1: note: Hint: "python3 -m pip install types-google-cloud-ndb"
foobar.py:16:1: note: (or run "mypy --install-types" to install all missing stub packages)
foobar.py:16:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)

Versions

• Mypy 1.1.1
• Python 3.10.10

mypy.ini

[mypy]
plugins = pydantic.mypy, numpy.typing.mypy_plugin

python_version = 3.10
show_error_codes = True
show_column_numbers = True
# Disallow untyped and partially typed functions
disallow_incomplete_defs = True
disallow_untyped_defs = True
# Ignore implicit 'return None'
no_warn_no_return = True
# Allow instantiating variables with untyped None value
local_partial_types = True
# Warns about casting an expression to its inferred type.
warn_redundant_casts = True
# Warns about unneeded # type: ignore comments.
warn_unused_ignores = True
# Shows a warning when encountering any code inferred to be unreachable or redundant after performing type analysis.
warn_unreachable=True
# Use visually nicer output in error messages: use soft word wrap, show source code snippets, and show error location markers.
pretty=True
# Disallows usage of generic types that do not specify explicit type parameters.
disallow_any_generics=True
# This flag causes mypy to stop treating arguments with a None default value as having an implicit Optional type.
no_implicit_optional=True
# Type-checks the interior of functions without type annotations.
check_untyped_defs=True

# Without these settings Mypy fails to run
namespace_packages=True
explicit_package_bases=True

[pydantic-mypy]
init_forbid_extra = True
warn_untyped_fields = True

[mypy-cashews.*,opencensus.*,google.auth.*,motor.motor_asyncio.*,stacklogging.*,google.*,bs4.*]
ignore_missing_imports = True

[mypy-astral.sun,google.*]
ignore_errors = True

.pre-commit-config.yaml

  # Source: https://stackoverflow.com/a/75003826/9608133
  - repo: local
    hooks:
      - id: mypy
        name: mypy
        entry: mypy --non-interactive --install-types
        require_serial: true
        language: system
        types: [python]

[AlexWaygood]

We haven't changed anything in how we deal with these packages over at typeshed. My guess is that this is a mypy bug in how it handles namespace packages. It's also possible that it could be a bug in the logic over at https://github.com/typeshed-internal/stub_uploader. Or something might have changed over at setuptools, I guess, since the stub_uploader uses setuptools to package typeshed's third-party stubs.

[AlexWaygood]

FYI, as an aside, I generally wouldn't recommend using mypy's --install-types option, as it means mypy has to run twice over your code instead of just once (so is pretty slow!), and may also lead to results that are hard to reproduce. Specify your types-* packages using the additional_dependencies field for the mypy hook in your .pre-commit-config.yaml file instead.

[simon-liebehenschel]

Specify your types-* packages using the additional_dependencies field for the mypy hook in your .pre-commit-config.yaml file instead.

I know, and I used to use this approach in the past. Filling additional_dependencies manually is impossible for thousands of repositories with unique dependencies. Unless I will write a bash script to parse dependencies and compose .pre-commit-config.yaml file dynamically from a template.

Another approach I did: I installed all packaged from typeshed. This helps to avoid running --install-types, but installing all existing types-* packages makes Docker images bloated with redundant pip packages.

---

For the current issue, the only temporary workaround is to install all types-* packages, but without types-google-cloud-ndb

For any future reader: feel free to reopen the issue.