Doesn't seem to work with pytest's --pdb mode

[risicle]

With pytest 5.4.1 & pytest-subtests 0.3.0, assertion failures in subtests don't invoke pdb for me. Assertion failures in the parent test do though.

[nicoddemus]

Thanks @risicle for the report!

[reaperhulk]

As pyca/cryptography switches over to subtests (which significantly improve our test performance!) it'd be great if this issue could be resolved. 😄 I don't understand the internals of this system well enough to fix it, but I'd be happy to put up some money to get this fixed, either directly to the author or a charitable donation to a 501(c)3 of their choosing.

[nicoddemus]

Hi @reaperhulk,

Thanks for the ping. I've not had much time to work on this in general, that's why this has staled a bit.

Thanks for the donation offer, I appreciate it, but it's not necessary for now. I will try to get to this soon to see how hard it is to fix it.

Btw could you share more about pyca/cryptography using this? It might be worth sharing your experience in pytest-dev/pytest#1367, where we discuss merging the subtests support in the core.

[alex]

pyca/cryptography@b3540ec is basically how we're using it -- replacing very large (1000s of tests) parameterizes with subtests. Our motivation here is performance.

[nicoddemus]

And did you see any improvements? If so, in collection, runtime, or both?

[reaperhulk]

There's a lot of noise in our CI measurements, but we're seeing improvements in total wall-clock time of ~20-30% across our full test suite. Collection time is vastly improved, but much of that is just because it's done lazily now. This might make pytest-xdist feasible for us though since previously collection was very expensive.

[nicoddemus]

Cool, thanks for sharing. 👍

[nicoddemus]

0.4.0 is out. 🎉