Pytest's testdir fixture may end up creating flaky tests

[tarcisiofischer]

While trying to investigate a flaky test on my code, I discovered a problem in the testdir fixture.
I was able to reach the problem while running a test using the testdir fixture too many times and in parallel. The code is something like:

@pytest.mark.parametrize('n', range(1000))
def test_bla(testdir, ..., n):
  # Use testdir here

The problem seems to be here (Note that if the for loop ends with never setting the variable, the problem should happen):
https://github.com/pytest-dev/pytest/blob/master/_pytest/pytester.py#L424

My proposal is to change the whole code block with something like

self.tmpdir = tmpdir_factory.mktemp(name, numbered=True)

I'll make a PR soon.

[nicoddemus]

Sounds good, thanks @tarcisiofischer!

[nicoddemus]

Resolved by #2752