Appveyor tests are being flaky

[emmatyping]

This was first brought up in #3846. Moved here to avoid clutter in the PR.

You can see the results here: https://ci.appveyor.com/project/JukkaL/mypy/build/1.0.1042

The relevant exception is OSError: [Errno 9] Bad file descriptor which I believe is from pytest trying to read from stdout, however for some reason stdout is either locked, unavailable, or Appveyor is incorrectly reporting the handle of stdout.

@elazarg chimed in

IIRC it seemed to happen when multiple instances of pytest were active. It stopped happening after I've changed the PR to use only a single instance. I'm not sure if this information helps though...

If true, AIUI, this should explain the issue. However, Im not sure we currently spawn multiple pytest instances concurrently, so this may indicate our scheduling logic is broken, or it is an entirely different issue.

[gvanrossum]

What makes you think it's related to stdout? The tracebacks all seem to point to this line:

    def resume(self):
        self.syscapture.resume()
>       os.dup2(self.tmpfile_fd, self.targetfd)

This does appear to be part of a pytest internal class (FDCapture.resume()).

[emmatyping]

The previous stack is

    def resume_capturing(self):
        if self.out:
            self.out.resume()
        if self.err:
>           self.err.resume()

I suppose stderr is more accurate, my apologies, but the effect is the same. This is part of pytest capturing ouput to the console.

[emmatyping]

Apparently PyInstaller had similar issues with pytest-xdist, so I believe that we can put the source on issues with pytest-xdist's scheduling.

At pybay @pkch and I saw an interesting talk about pytest-concurrent. It may be too early on for a switch, but it might be nice to use in the future. Eg "allowing certain tests to be grouped so that they are executed sequentially".

[emmatyping]

Scrolling through twitter this morning I also ran into this https://twitter.com/pumpichank/status/903280328978178048

So maybe its Appveyor flakiness? Hard to tell.

[emmatyping]

Since it has been 3 weeks, I think (I hope) this can be safe to close.

[elazarg]

It happened again in #3973

[emmatyping]

Darn. Okay, I think that eliminates it being Appveyor being flaky. Seems that our entire test suite needs to be tested. (see also #3975).

[elazarg]

@ethanhs see my recent comment there. If my comment is related to the bug, then it seems to be unrelated to appveyor flakiness - or at least not precisely the same problem. Besides, the problems with appveyor only began after my PR (#3870), whereas the problems with Travis came up earlier. What do you think?

[emmatyping]

@elazarg The timeout issue seems quite plausible as the issue for Travis, that is a good find! However, the Appveyor flakes seem to be something different. Unless timeouts cause issues with streams, I don't see how the timed out test could cause issues with the duplication of the stderr stream (which I believe is the symptom the issue is causing).