types-protobuf fails pre-commit cache

[Itay4]

when configuring pre-commit mypy to run with types-protobuf, for example:

  -   repo: https://github.com/pre-commit/mirrors-mypy
      rev: v1.8.0
      hooks:
        -   id: mypy
            additional_dependencies:
              - types-protobuf

it fails with the following:

mypy.....................................................................Failed
- hook id: mypy
- exit code: 2

mypy: can't read file '/home/runner/.cache/pre-commit/repoem44ysms/py_env-python3.10/lib/python3.10/site-packages//google': No such file or directory

i suspect its related to #11249, but not sure - @AlexWaygood @hauntsaninja i would appreciate your insights here

[andylamp]

Encountering the same issue as well, this was previously working....

edit: happy to provide logs etc if that helps, but the resulting message I get when I run the pre-commits is the same as @Itay4 's output shown above.

edit2: if it helps, I tried using both Python 3.10 and Python 3.11 virtual envs for running the pre-commit hooks but the same error occurs.

edit3: I cannot find a way to use a "downgraded" of an additional_dependencies entry in mypy as a pre-commit hook so cannot check that previous versions worked, but we certainly did not have this problem previously...

[srittau]

Neither #11249, not #11245, the two PRs included in the latest release, should have an influence here. I'm not sure what exactly is going on here, but types-protobuf doesn't add a google directory, but a google-stubs directory, so it's no wonder that mypy can't find it. Does downgrading to types-protobuf 4.24.0.4 help?

[andylamp]

Hi @srittau, as I mentioned previously I do know a way to specific specific version within mypy in a pre-commit environment.

Using the default though, exhibits the behavior mentioned by the original author.

If you know a way to "pin" an additional depency version, let me know and I can test it.

[srittau]

This problem was introduced by typeshed-internal/stub_uploader#109. Previously, the google-stubs directory was missing the py.typed file that marked the stubs as partial. I think this was an error according to the typing spec:

For namespace packages (see PEP 420), stub-only packages should use the -stubs suffix on only the root namespace package. All stub-only namespace packages should omit __init__.pyi files. py.typed marker files are not necessary for stub-only packages, but similarly to packages with inline types, if used, they should be in submodules of the namespace to avoid conflicts and for clarity.

Cc @Avasam

[Avasam]

If I understand correctly, "google" is a namespace package and the py.typed marker should be found in google-stubs/protobuf/ rather than google-stubs/ ? (that's what I'm getting from the part of the spec srittau is quotig)

In the short term, simply setting partial_stub to false in https://github.com/python/typeshed/blob/main/stubs/protobuf/METADATA.toml (and namespace packages in general) with an appropriate comment should be enough to fix the current issue (no py.typed file should be generated for non-partial stubs, as it is optional and redundant as per the spec).

In the longer term, we may want a way to identify those so stub_uploader adds the py.typed file to the submodules. Like the lack of any file at the top-level.

[andylamp]

@Avasam thanks for the explanation, any workarounds for this issue for the interim :-)?

[srittau]

If I understand correctly, "google" is a namespace package and the py.typed marker should be found in google-stubs/protobuf/ rather than google-stubs/ ? (that's what I'm getting from the part of the spec srittau is quotig)

That's my understanding as well. I think making the necessary changes to the stub_uploader should be fairly simple, although I haven't investigated deeply.

[Avasam]

@Avasam thanks for the explanation, any workarounds for this issue for the interim :-)?

@andylamp Check the very bottom of https://github.com/pre-commit/mirrors-mypy 's readme. I'm pretty sure it's just using pip, so the same markers should work to pin to an older version: types-protobuf==X.X.X

[tonkolviktor]

Hi, would it be possible to delete this version from pypi?