"Exception: Failed to create virtualenv" during packs.install

[arm4b]

Reproduce (Ubuntu14)

wget -qO - https://bintray.com/user/downloadSubjectPublicKey?username=bintray | apt-key add -
add-apt-repository 'deb https://dl.bintray.com/stackstorm/trusty_staging unstable main'

# Install Rabbit and Mongo
apt-get update
apt-get install rabbitmq-server mongodb-server

# Install st2
apt-get install st2api st2actions st2auth st2client st2common st2debug st2exporter st2reactor

# Set `enable = False` under `[auth]` section in file: `/etc/st2/st2/conf`
sudo sed -i '/^\[auth\]$/,/^\[/ s/^enable = True/enable = False/' /etc/st2/st2.conf

# Register content
st2-register-content --register-all --config-dir /etc/st2

# Start everything
service st2api start
service st2auth start
service st2actionrunner start
service st2notifier start
service st2resultstracker start
service st2rulesengine start
service st2sensorcontainer start
service st2exporter start

# Install dependencies for packs.install to work
apt-get install git python-virtualenv

# Install some st2 pack
st2 run packs.install packs=github

Error message

# st2 run packs.install packs=github

....
id: 5675a56dfad0d513e9647150
action.ref: packs.install
status: failed
error: st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "github"
st2.actions.python.SetupVirtualEnvironmentAction: INFO     Virtualenv path "/opt/stackstorm/virtualenvs/github" doesn't exist
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "github" in "/opt/stackstorm/virtualenvs/github"
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/github" using Python binary "/usr/share/python/st2actions/bin/python"
Traceback (most recent call last):
  File "/usr/share/python/st2actions/local/lib/python2.7/site-packages/st2actions/runners/python_action_wrapper.py", line 116, in <module>
    obj.run()
  File "/usr/share/python/st2actions/local/lib/python2.7/site-packages/st2actions/runners/python_action_wrapper.py", line 61, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/setup_virtualenv.py", line 60, in run
    self._setup_pack_virtualenv(pack_name=pack_name, update=update)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/setup_virtualenv.py", line 102, in _setup_pack_virtualenv
    self._create_virtualenv(virtualenv_path=virtualenv_path)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/setup_virtualenv.py", line 140, in _create_virtualenv
    (virtualenv_path, stderr))
Exception: Failed to create virtualenv in "/opt/stackstorm/virtualenvs/github": Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 2339, in <module>
    main()
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 825, in main
    symlink=options.symlink)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 985, in create_environment
    site_packages=site_packages, clear=clear, symlink=symlink))
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 1181, in install_python
    copy_required_modules(home_dir, symlink)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 1120, in copy_required_modules
    copyfile(filename, dst_filename, symlink)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 467, in copyfile
    os.makedirs(os.path.dirname(dest))
  File "/usr/share/python/st2actions/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/opt/stackstorm/virtualenvs/github/lib/python2.7/lib-dynload'


traceback: None
failed_on: setup_virtualenv
start_timestamp: 2015-12-19T18:43:57.690729Z
end_timestamp: 2015-12-19T18:44:04.361833Z
+--------------------------+-----------+-------------------+-------------------------+-------------------------------+
| id                       | status    | task              | action                  | start_timestamp               |
+--------------------------+-----------+-------------------+-------------------------+-------------------------------+
| 5675a56dfad0d513ff14be8d | succeeded | download          | packs.download          | Sat, 19 Dec 2015 18:43:57 UTC |
| 5675a572fad0d513ff14be90 | succeeded | virtualenv_prerun | packs.virtualenv_prerun | Sat, 19 Dec 2015 18:44:02 UTC |
| 5675a573fad0d513ff14be93 | failed    | setup_virtualenv  | packs.setup_virtualenv  | Sat, 19 Dec 2015 18:44:03 UTC |
+--------------------------+-----------+-------------------+-------------------------+-------------------------------+

Helpful Info:

Broken symlinks:

/opt/stackstorm/virtualenvs/github/lib/python2.7# ls -l
total 0
lrwxrwxrwx 1 root root 38 Dec 19 18:44 codecs.py -> ../../../../../lib/python2.7/codecs.py
lrwxrwxrwx 1 root root 38 Dec 19 18:44 encodings -> ../../../../../lib/python2.7/encodings
lrwxrwxrwx 1 root root 39 Dec 19 18:44 fnmatch.py -> ../../../../../lib/python2.7/fnmatch.py
lrwxrwxrwx 1 root root 43 Dec 19 18:44 genericpath.py -> ../../../../../lib/python2.7/genericpath.py
lrwxrwxrwx 1 root root 40 Dec 19 18:44 lib-dynload -> ../../../../../lib/python2.7/lib-dynload
lrwxrwxrwx 1 root root 38 Dec 19 18:44 locale.py -> ../../../../../lib/python2.7/locale.py
lrwxrwxrwx 1 root root 38 Dec 19 18:44 ntpath.py -> ../../../../../lib/python2.7/ntpath.py
lrwxrwxrwx 1 root root 34 Dec 19 18:44 os.py -> ../../../../../lib/python2.7/os.py
lrwxrwxrwx 1 root root 41 Dec 19 18:44 posixpath.py -> ../../../../../lib/python2.7/posixpath.py
lrwxrwxrwx 1 root root 36 Dec 19 18:44 stat.py -> ../../../../../lib/python2.7/stat.py
lrwxrwxrwx 1 root root 40 Dec 19 18:44 UserDict.py -> ../../../../../lib/python2.7/UserDict.py

https://github.com/StackStorm/st2/blob/master/contrib/packs/actions/pack_mgmt/setup_virtualenv.py might have answers.

[arm4b]

probably related: #50

[arm4b]

@dennybaa this one has higher priority, since it's blocker for puppet installer (where many st2 packages are installed).

[estee-tew]

https://stackstorm.slack.com/archives/stackstorm/p1450800171017578

[dennybaa]

Python bin directory location should NOT rely on PATH, it should use same mechanism which is used to locate python binary and fallback to PATH (optionally).

https://github.com/StackStorm/st2/blob/master/contrib/packs/actions/pack_mgmt/setup_virtualenv.py#L135

[Kami]

Just FYI there is actionrunner.python_binary option which allows you to specify which Python binary is used by the python runner action.

By default, that is the python binary which is used to run action runner process.

[arm4b]

Fix for this issue is discussed here: StackStorm/st2#2339

[DoriftoShoes]

I was able to work around this for actions by adding --always-copy to the virtualenv command here:
https://github.com/StackStorm/st2/blob/master/contrib/packs/actions/pack_mgmt/setup_virtualenv.py#L135

This allowed me to get the virtualenv created and to successfully run actions. The problem as it stands now is that sensors do not work. Since the virtualenv for the pack was created using the st2actions venv and not the st2reactor venv.

There were a couple other hacks to the packs pack necessary but that isn't relevant to this issue. I'll open a separate PR on those.

[dennybaa]

@DoriftoShoes nice, but we should not rely upon this work around... It's better setup_virtualenv.py is fixed to locate proper path to virtualenv binary. @Kami do you want me to try to implement this?

What comes to my mind, that we should get st2actionrunner (probably even st2 binary, i.e. python with which process is invoked) interpreter path, and evaluate virtualenv path accordingly. What do you think gentlemen?

[lakshmi-kannan]

@dennybaa @Kami: I was playing around with virtualenv paths and it doesn't make a difference. See gist here: https://gist.github.com/lakshmi-kannan/bddd20683f5fe4a49013

Patrick's --always-copy fixes the issue irrespective of which virtualenv binary is used. I will validate if this option works with existing packages.

[lakshmi-kannan]

Confirmed --always-copy works with existing packages. The difference in disk space between using --always-copy and not using it is below:

with --always-copy old packages
lakshmi@st2ops001:$ sudo du -sh /opt/stackstorm/virtualenvs/aws
52M /opt/stackstorm/virtualenvs/aws
lakshmi@st2ops001:$

without --always-copy old packages
lakshmi@st2ops001:$ sudo du -sh /opt/stackstorm/virtualenvs/aws
24M /opt/stackstorm/virtualenvs/aws
lakshmi@st2ops001:$

with --always-copy st2bundle packages
ubuntu@svetlana001:$ sudo du -sh /opt/stackstorm/virtualenvs/aws
50M /opt/stackstorm/virtualenvs/aws
ubuntu@svetlana001:$

[dennybaa]

@lakshmi-kannan , @here. Awesome, and thank you! I confirm that without copying (--always-copy) it won't work. However there's some positive researching from today. First let's cover fundamentals, based of course on my assumptions:))

First important: why virtualenv doesn't work without --always-copy or -p /usr/bin/python (system one).

• It's not clear totally how these above correlate, but invocation of virtualenv is not intended to run from another virtualenv. By design it's not relocatable...
• There's even an option --relocatable which makes existing virtualenv relocatable, but it doesn't work on 100%. So if you make our virtualenv relocatable using the switch, next you run ../virtualenv --system-site-packages /some/venv, not all of modules end up there, but a good point it that that virtualenv doesn't fail:)
• I've made some investigation, I found a way to completly copy via hardlinking prepared for relocation virtualenv here dennybaa/virtualenv-clone@ea8f9e0.

So just look, it's possible to run

# first we should preinstall virtualenv-clone
# second we need to make /usr/share/python/st2 relocatable using `--relocatable` switch
# then
/usr/share/python/st2/bin/virtualenv-clone /usr/share/python/st2 /tmp/vc

After this /tmp/vc looks fully functional.

Okay summing it up, it's possible to reduce virtualenv size to minimum (almost to 0) via hard linking.

But another important thing arises (upgrade story??), what is post upgrade st2 and pack story!? Using --always-copy freezes old st2 code inside a pack, same happens if we do hard linking. I.e. we upgrade st2 package and then we endup with outdated st2common, st2actions etc inside a pack... I have a few ideas, but we indeed need to discuss it, so we are not getting stuck at this point.

[dennybaa]

fixing last portion #76
So what we have:
All packages except el7 use:

virtualenv_opts = --always-copy

el7:

virtualenv_opts = 

[dennybaa]

i think closing. since already merged and tested using CircleCI...