Should we expand our test suite by running some downstream projects tests in our CI?

[njsmith]

Apparently trio v0.11.0 broke trio-websocket, do to a subtle interaction between our deprecation logic and the import statement: python-trio/trio-websocket#102

I can't think of any way we could have anticipated this particular issue. I mean, besides "be smarter", but "be smarter" is not a plan. It never occurred to us that supporting the exact statement import trio.ssl was part of our API, or if it was, that it was a part that could break.

One possible hedge against issues like this would be to have a CI job that runs the test suites for some downstream projects like trio-websocket, just to better align what we test versus what people use in the real world. No guarantees, but it'd give us a better chance of catching stuff like this in the future.

trio-websocket is an obvious candidate for this. What else? trio-asyncio, pytest-trio? hypercorn/quart-trio? (CC @pgjones) Of course we can pick a list now and then adjust it over time.

[Zac-HD]

Related to python-trio/pytest-trio#79, I guess.

Personally I think this is a great idea in principle, and probably still a good idea in practice despite the volume of edge cases we're likely to turn up. In particular coordinating fixes for upstream tests is a nightmare.

My own preference is similar to Hypothesis, where all the extra modules have independent dependencies and are tested separately, but distributed together - if you don't need support for X, you just don't import hypothesis.extra.X. However a plain ol' monorepo where we have multiple directories for projects with independent PyPI packages would also make coordinated changes much easier, especially with some release automation!

[njsmith]

A monorepo is an interesting idea actually. But I think it's a different interesting idea than this one :-). (Maybe open a separate issue?)

For this, we should ideally be testing dev versions of trio against released versions of downstream packages. That's how we answer the question: "if we release this change right now, how much breakage will our users experience?" It's great if downstreams have a fix in their dev branches, but if they aren't released then it doesn't help users much.

And anyway we aren't going to keep the entire trio ecosystem inside a monorepo :-)

(This comment sponsored by the prominent numpy downstream that would deal with deprecations promptly in their dev branch, but then wait years between releases.)

[Zac-HD]

Sorry, that's unclear on a re-read: I think we absolutely should pull in and test against downstream packages. I additionally propose a few things that would make that easier for our own downstreams.

It's great if downstreams have a fix in their dev branches, but if they aren't released then it doesn't help users much.

Release automation! I've proposed a talk about it for the PyCon maintainer's summit 😄

(This comment sponsored by the prominent numpy downstream that would deal with deprecations promptly in their dev branch, but then wait years between releases.)

😱

[nicoddemus]

FWIW we do exactly that on pytest: we test it with downstream pytest-xdist (released version) and with upstream pluggy (master, because released versions are already tested in the other environments as pluggy is a pytest dependency).

It has caught some regressions for us in the past, and it is a little annoying to setup.

I have thought of a mono-repo using git submodules just for CI different packages together, but never actually implemented it. It would be interesting to see how something like that would work in practice. 😁

[pgjones]

I'm not sure I've much to add here; I would be happy if you tested Hypercorn and Quart-Trio. I'd also like to add something to the Hypercorn & Quart-Trio CI that would alert if there is a new issue (if you have ideas...).