pytest-xdist regression since execnet 1.3.0

[pytestbot]

Originally reported by: Marc Schlaich (BitBucket: schlamar, GitHub: schlamar)

---

With the latest execnet release 1.3.0 a pytest-xdist test in pytest-cov fails with ImportError: no module named py.

This is the first error in this build: https://travis-ci.org/schlamar/pytest-cov/jobs/56056841

Pinning execnet to 1.2.0 solves this issue.

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/707

[pytestbot]

Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

---

this is a problem triggered by a bugfix in execnet

due to an oversight we would add the site-packages of a different python interpreter to sys.path of the slave interpreter
which is also why other modules would appear there

we need to ensue that on the slave a pylib/pytest of the same version is importable

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

So there was a bugfix in execnet which exhibited a problem in the pytest-xdist code which tries to ensure that py/pytest is importable remotely? If so, can you provide a PR fixing this?

[pytestbot]

Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

---

the problem is local import-ability if the python of the slave is different from the master

[pytestbot]

Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

---

to solve this propperly we need to ensure that pylib, pytest and potentially the pytest plugins in use + their dependencies are importable in the slave process

with pytest/pylib, we can probably just create module instance for _pytest, py as well as pytest and do something evil with execfile and __path__/__file__ as a first approximation,
later after we support the wheel distribution format we should look into passing pure python wheels straight to slaves via a execnet mechanism (needs some discussion)

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

Let's keep it simpler as a first iteration: just make sure that py and pytest are importable, no plugins and no considerations of wheels. This just requires rsyncing/copying the installed packages from site-packages and we know that py/pytest are interpreter independent.

[pytestbot]

Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

---

im not sure what to use as the rsync location, got any suggestions?

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

it's not really rsync but i guess you mean local destination for the copy of py and pytest. We could use a tempdir although it's not very elegant. What about teaching execnet to pre-import a list of modules and temporarily prepending the source's site-packages for that? It's a hack but to do all of this properly we need to create virtualenvs, and then we wouldn't do any copying.

[pytestbot]

Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

---

i'd like to avid adding such behavior to execnet, i suspect it will bring in fragile misbehavior, however it sounds like a perfect hack for pytest-xdist for handling the situation of different executable until we can create a proper solution

[Diaoul]

I just faced the same issue, any updates on this or workaround?

[RonnyPfannschmidt]

not yet

[RonnyPfannschmidt]

closing as the followup is pytest-dev/pytest-xdist#113