Reintroduce error on xpass

[RonnyPfannschmidt]

We should not provide a half done solution to flaky tests as noted in #814

The correct solution to flaky tests is a plugin with supporting primitives not technical debt in xfail

[nicoddemus]

I think #1355 is a good solution: an strict_xfail ini option (defaulting to False for backward compatibility) to treat XPASS as failures. We could then change the default to True in a future release so pytest by default will be more strict (which I'm also in favor), providing we warn users before hand so they can adjust their pytest.ini before the default is switched over.

Seems simple to implement and will solve the problem in the long run, without the need for users to have to resort to a separate plugin.

[RonnyPfannschmidt]

i agree

[esiegerman]

@nicoddemus How would strict_xfail remove the need for a plugin?

(To be clear, I'm strongly in favour of reverting xfail to fail-on-XPASS, and also in favour of the strict_xfail ini option -- I think it's a good way to minimize the pain that the former will cause for some people. I just don't see how, in addition, strict_xfail helps people with flaky tests -- assuming that's the plugin you're referring to, that is...)

[nicoddemus]

Oh flaky tests are another issue... I was merely stating the solution for the fail-on-XPASS problem. xfail would probably need a separate plugin

[RonnyPfannschmidt]

@esiegermann xfail should not be used for flaky Tests to begin with

[esiegerman]

@RonnyPfannschmidt I know. I misinterpreted @nicoddemus's recent comment; when he mentioned no longer needing a plugin, I assumed (based on earlier discussion, e.g. in #814), that it was a flaky-supporting plugin he was referring to. Since he wasn't, my question is meaningless and can be ignored.

Sorry for the noise.