Multiple packages : only one can be found

[asmodehn]

When having multiple package using catkin_pip in a workspace, only one of them can be imported.

This seems to be due to the easy-install.pth file, which contains in that case :

/home/alexv/ROS/gopher_bootstrap/src/pyros

Multiple egg links are present, but useless with this easy-install.pth file.

Note: I confirmed that this isn't a pip bug :

alexv@asmodehn:~/tmptest$ tree
.
├── p1
│   ├── pp1
│   │   └── __init__.py
│   └── setup.py
├── p2
│   ├── pp2
│   │   └── __init__.py
│   └── setup.py
└── pmain
    ├── ppmain
    │   └── __init__.py
    └── setup.py

6 directories, 6 files
alexv@asmodehn:~/tmptest$ mkvirtualenv tmptest
New python executable in /home/alexv/.virtualenvs/tmptest/bin/python
Installing setuptools, pip, wheel...done.
(tmptest) alexv@asmodehn:~/tmptest$ pip list --format=columns
Package    Version 
---------- --------
pip        9.0.1   
setuptools 33.1.1  
wheel      0.30.0a0
(tmptest) alexv@asmodehn:~/tmptest$ cd p1/
(tmptest) alexv@asmodehn:~/tmptest/p1$ pip install -e .
Obtaining file:///home/alexv/tmptest/p1
Installing collected packages: p1
  Running setup.py develop for p1
Successfully installed p1
(tmptest) alexv@asmodehn:~/tmptest/p1$ cd ../p2
(tmptest) alexv@asmodehn:~/tmptest/p2$ pip install -e .
Obtaining file:///home/alexv/tmptest/p2
Installing collected packages: p2
  Running setup.py develop for p2
Successfully installed p2
(tmptest) alexv@asmodehn:~/tmptest/p2$ cd ../pmain/
(tmptest) alexv@asmodehn:~/tmptest/pmain$ python -m ppmain
1
2
All GOOD

note easy-install.pth file in that case :

(tmptest) alexv@asmodehn:~/tmptest/pmain$ cat ~/.virtualenvs/tmptest/lib/python2.7/site-packages/easy-install.pth 
/home/alexv/tmptest/p1
/home/alexv/tmptest/p2

And also :

(tmptest) alexv@asmodehn:~/tmptest/pmain$ ls ~/.virtualenvs/tmptest/lib/python2.7/site-packages
easy-install.pth  easy_install.pyc  p2.egg-link  pip-9.0.1.dist-info  setuptools                   wheel
easy_install.py   p1.egg-link       pip          pkg_resources        setuptools-33.1.1.dist-info  wheel-0.30.0a0.dist-info

[asmodehn]

#76 now explicitely checks for this and doesnt exhibit the issue...

More investigation in our usecase where the problem was first spotted shows that the problem does NOT happen when using yujin_make -jobs=1.

Meaning this is likely a race condition. multiple catkin_make are calling multiple pip instances that modify the easy-install.pth at the same time...

We need to investigate more for ways to prevent this...

[asmodehn]

the yujin_make theory doesnt make sense... yujin_make only call make in parallel.

Also running a second time with multiple jobs (default yujin_make setting) doesnt exhibit the problem...

So we are now in a CANNOT REPRODUCE situation. at least with the version in #76

Lets keep this around for a while, but if it doesn't happen again, lets close this.

[asmodehn]

Still happening... reproducing steps are not really clear yet.

[asmodehn]

Reference : pip not handling parallel invocations : pypa/pip#2361

[asmodehn]

#61 address this by using a lock and dependencies chain to ensure there is only one instance of pip running at a time.