Add coverage benchmark

[kumaraditya303]

Add a coverage benchmark to help benchmarking and avoid regressions such as python/cpython#93516.
Benchmark result 3.10 vs 3.12 main branch.

Benchmark	3.10	3.12
coverage	96.8 ms	204 ms: 2.11x slower

[gvanrossum]

LGTM, thanks for doing something about this.

Question for @kumaraditya303 -- Have you actually tried this before and after the regression you mentioned as example?

Question for @ericsnowcurrently -- Do you agree this is useful?

[kumaraditya303]

Have you actually tried this before and after the regression you mentioned as example?

@gvanrossum: The regression I linked above ¹ is not caused by a single change but rather accumulation of multiple slowdown caused by multiple changes of which the most significant ones being python/cpython#91666 and python/cpython#27077. Since there is no benchmark in pyperformance to exercise coverage performance, the slowdown was unnoticed. Once this benchmark is added, it will help us pinpoint which commits caused the most slowdown with a simple git bisect between 3.10 and main branches.

Footnotes

1. https://github.com/python/pyperformance/pull/213#issue-1261503016 ↩

[gvanrossum]

Nevertheless it would be interesting to see the numbers of running this BM on the state before and after said commits to demonstrate its usefulness.

[kumaraditya303]

I am running the benchmarks now, will post soon.

[kumaraditya303]

@gvanrossum Benchmarking before and after python/cpython@944fffe commit

Results:

Benchmark	before	after
coverage	161 ms	213 ms: 1.32x slower

[ericsnowcurrently]

Question for @ericsnowcurrently -- Do you agree this is useful?

Yeah, it's a helpful indicator of impact.