Improve virtualenv support & egg-link resolution

[immerrr]

This PR implements the offline part of the functionality implemented in #562 so that it's easier to discuss and commit regardless of the decision about fetching path from live interpreter. It also extends venv-related tests a bit.

It should fix #443 and #384, also might help with #560 and #535.

[davidhalter]

Very awesome. Thanks a lot!! There are some tests failing (haven't looked into it).

[immerrr]

@davidhalter it seems that the test failure is an issue in pytest, it cannot find a fixture and falls apart trying to report it (reported as pytest-dev/pytest#1148). I worked it around by adding testpaths = test to pytest.ini, but it might be yet another issue with doctest fixture discovery.

[immerrr]

huh, travis caught an actual failure, let me look into it

[immerrr]

There's also a hack in emacs jedi epc server that refers to #36 and mentions caching issues. I have added a test to ensure that sys_path= kwarg plays well with caching, just in case.

[immerrr]

All green after rebasing on top of #637.

[davidhalter]

I like this pull request a lot. It's really fine work!!! :)

I'm not merging yet, for one minor reason. I think I have talked about #562 before:
I struggle a bit about just including it, because it allows (as far as I remember) execution of arbitrary Python code, which is not allowed in Jedi, yet. Is there no other option of doing it maybe in a different way (with Jedi introspection)? Which (well-known) packages does it benefit? Examples?

[immerrr]

I have done a quick grep through pth files on my system and all of the custom code in pth files is either easy_install bootstrap code:

import sys; sys.__plen = len(sys.path)
./jeditestmodule-0.0.0-py2.7.egg
/home/immerrr/sources/jedi
/home/immerrr/sources/pytest
import sys; new = sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p = getattr(sys, '__egginsert', 0); sys.path[p:p] = new; sys.__egginsert = p + len(new)

or namespace package bootstrap code:

import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('sphinxcontrib',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('sphinxcontrib',types.ModuleType('sphinxcontrib')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)

I think those can be skipped without any harm to sys.path detection. What worries me is that eval SMTH in site.addpackage function cannot be disabled, so either mocking or copy-pasting it is required.

[davidhalter]

Since we're just adding to the sys.path, we can just add al the paths to it, ignore the import lines and that's it? Or would it be more complicated?

[immerrr]

standard library employs case-sensitivity hacks, so it seems better to just pull in some well-debugged code under a permissive license than to reimplement it from scratch.

[immerrr]

(which i have done)

[davidhalter]

Very, very nice! I like this PR a lot. Thanks so much!

It's a lot already, I'm really glad we didn't mix it with the VIRTUAL_ENV stuff anymore. Can you briefly describe in #562, what the leftovers are? What still would need to be done?

[immerrr]

There are several issues that are affected by this PR directly, #443 and #384 should be fixed now, #560 and #535 are a bit vague, but might be fixed too.

[davidhalter]

Wow that's a lot of fixes. :) Thanks a lot (for also searching all the issues).