Add Python type-hinting throughout Pulumi codebase

[jemrobinson]

✅ Checklist

• You have given your pull request a meaningful title (e.g. Enable foobar integration rather than 515 foobar).
• You are targeting the develop python-migration branch.
• Your branch is up-to-date with the develop python-migration branch (you probably started your branch from develop but it may have changed since then).
• If-and-only-if your changes are not yet ready to merge, you have marked this pull request as a draft pull request and added '[WIP]' to the title.
• If-and-only-if you have changed any Powershell code, you have run the code formatter. You can do this with ./tests/AutoFormat_Powershell.ps1 -TargetPath <path to file or directory>.

⤴️ Summary

• Add type hints to codebase
• Add typing stubs where needed

🌂 Related issues

Fix unresolved issues from #1375

🔬 Tests

Tested by running mypy locally.

[jemrobinson]

@JimMadge : this is mostly ready to review (just two files that aren't passing mypy: strict at the moment). I've removed the casts that you thought were ugly/unnecessary and added type-hinting throughout. This has also uncovered a couple of type-related bugs so these are fixed here too.

[JimMadge]

I think that is looking better. Painful to add now but I hope in the future this will,

• Help avoid errors/bugs due to types mismatching
• Ensure that input is properly parsed/sanitised

Are the .pyi files necessary? They are for modules we import?

[jemrobinson]

Are the .pyi files necessary? They are for modules we import?

These are effectively header files for libraries we import that don't have native type hinting (or don't advertise that they do). For common libraries you can normally install a library-name-types package but these don't always exist. Additionally, for some of the azure libraries I've changed the .pyi file to match the return value the function actually gives, not the one advertised in the docs.

[JimMadge]

These are effectively header files for libraries we import that don't have native type hinting (or don't advertise that they do). For common libraries you can normally install a library-name-types package but these don't always exist. Additionally, for some of the azure libraries I've changed the .pyi file to match the return value the function actually gives, not the one advertised in the docs.

I hope the starting point was automatically generated somehow 😱.

Sorry, I asked the wrong question. I think what I'm really trying to ask is,

• Does mypy fail without these?
• If it does,
  • Do we really want to maintain these ourselves?
  • Isn't that an upstream bug?
  • Can we use # type: ignore instead?

Also, did you get mypy to play nicely with Pulumi? I remember having problems which I think boiled down to Pulumi being so big it was impractical to do a thorough type check.

[jemrobinson]

* Does mypy fail without these?

It complains about missing stubs (eg. Skipping analyzing "azure.mgmt.dns": module is installed, but missing library stubs or py.typed marker)

* If it does,
  * Do we really want to maintain these ourselves?

Not really.

  * Isn't that an upstream bug?

Yes, but it's not a bug since type-hinting is still very optional.

  * Can we use `#  type: ignore` instead?

Yes.

Also, did you get mypy to play nicely with Pulumi? I remember having problems which I think boiled down to Pulumi being so big it was impractical to do a thorough type check.

Yes, the trick is actually to use a .pyi stub that replaces import everything with import the things we need.

I now have everything working with the stub files. I can try removing them if you prefer this?

[JimMadge]

I now have everything working with the stub files. I can try removing them if you prefer this?

I think we should try and make that work, especially as we agree there is no desire to maintain these stub files.

As type hinting is, and presumably will remain, optional I think it makes most sense to not type untyped libraries ourselves.

[jemrobinson]

@JimMadge : Now ready for review with type-hinting stubs removed and # type: ignore used instead.

[JimMadge]

LGTM 👍

I think the Pulumi stub file is a good idea to help type checking finish in a reasonable time without overwriting Pulumi's own typing hints.