[FYI] mypy_primer

[hauntsaninja]

With apologies to black_primer, I jotted down a script that lets us run mypy over a large amount of open source Python code. Hopefully this helps democratise the process of identifying regressions and evaluating risky changes.

Something like this was brought up in #4332
This could also serve as a way of decentralising something like #1339

This has informed:
#4579
#4574
#4582
#4583
And you can see it in action at:
python/mypy#9275

More details at: https://github.com/hauntsaninja/mypy_primer

Feel free to close or discuss!

[CraftSpider]

Like this idea, also reminds me of Rust's crater tool. Helps compare the (possibly biased) view of the typeshed collaborators with the practical usage of the types in the ecosystem.

[JelleZijlstra]

This is great, thank you! I wonder if we can incorporate it into typeshed CI.

[hauntsaninja]

Yeah, it would be nice to get it into CI in some form.

One concern is speed. A Github runner has two cores. rm -rf $(python3 -m pip cache dir); time python -m primer --clear --concurrency 2 --debug takes 17 minutes on my machine (which probably has better per core perf). Plus hopefully the time taken will increase over time (although we could always restrict to a subset of projects).
However, mypy_primer currently installs mypy from source. For typeshed we don't care too much about mypy version, so things would be faster if we installed from pip and got a mypyc-compiled mypy.

It's also worth noting that there'll be a class balance of errors: since most of the projects are well established, I assume most errors will be false positives. I do intend on starting a repository where we can put the things we'd want from #1339.

[srittau]

Thanks, that is very useful indeed. Maybe at first we could have a daily or weekly run that just informs us when there are problems, but doesn't cause CI to fail.

[JukkaL]

For typeshed we don't care too much about mypy version, so things would be faster if we installed from pip and got a mypyc-compiled mypy.

I'd recommend using a compiled mypy -- it's often 3-4x faster than interpreted.

[hauntsaninja]

Just measured, it's 3.5x over the corpus. I've made the change to mypy_primer: if you explicitly specify a version, it'll first look for it on PyPI.

[hauntsaninja]

mypy_primer now runs in CI