Virtualenv support #55

swalkinshaw · 2019-08-17T20:04:19Z

trellis-cli exists to make Trellis easier to install, use, and manage. So far all the features have made it easier to create new projects and manage them, but nothing to help install Trellis' initial requirements. We've seen people struggle to use the proper python, install pip, install the correct versions of Ansible, etc.

This PR adds virtualenv support which is a tool to create isolated Python environments. The idea is that a Trellis user only needs Python installed and trellis-cli will take care of the rest. Thankfully most OS' have Python installed by default, so nothing else will be needed.

High-level workflow:

when creating a project with new, a virtualenv is created per project and activated
a new init command will "initialize" a project with virtualenv
any trellis-cli command in a project will automatically activate virtualenv and use its bin dir (as part of $PATH

trellis-cli will attempt the following to use virtualenv:

if python3 is installed: use built-in venv module
if virtualenv exists then just use it directly
finally install virtualenv locally and use it

Note: this relies on changes in roots/trellis#1100

swalkinshaw · 2019-08-18T16:56:07Z

This feature is basically complete though I'm sure there's bugs! Needs tests as well which won't be easy.

austinpray · 2019-08-18T22:53:47Z

If I'm reading this correctly it creates a virtualenv for every distinct project right? If so: 👍

swalkinshaw · 2019-08-18T23:47:39Z

yep exactly

…

On Sun, Aug 18, 2019 at 6:53 PM Austin Pray ***@***.***> wrote: If I'm reading this correctly it creates a virtualenv for every distinct project right? If so: 👍 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#55?email_source=notifications&email_token=AACIFNMGOL5IB32ZAXDWTH3QFHHHZA5CNFSM4IMQKBOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4RKNIY#issuecomment-522364579>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACIFNMTZYLSSBH6F5WLNELQFHHHZANCNFSM4IMQKBOA> .

trellis/virtualenv.go

tangrufus · 2019-08-31T22:41:30Z

github/main.go

+	return release.Version
+}
+
+func FetchLatestRelease(repo string) Release {


Can we provide a way to keep repo (e.g: roots/trellis) git history?

🤔 that's a tough one. I guess it's possible to use the same API to get the release tag_name, then do a Git clone (via https?).. and then? git clone --branch <tag_name> <repo_url> is supported in newer Gits.

swalkinshaw · 2019-09-05T21:26:48Z

I'm going to "unit" test this as much as possible but this feature makes that especially difficult and it won't really be testing that much functionality.

@tangrufus do you have any ideas for integration tests with it using docker? For the best coverage, I'd need to test multiple Python versions as well. And some things (such as installing Virtualenv) require the network as well 😞

tangrufus · 2019-09-05T21:53:01Z

This is my current setup:

➜ python -V
Python 2.7.16

➜ brew info python@2
python@2: stable 2.7.16 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@2/2.7.16_1 (3,763 files, 51.7MB) *
  Poured from bottle on 2019-09-03 at 09:03:44
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/python@2.rb

➜ python3 -V
Python 3.7.4

➜ brew info python
python: stable 3.7.4 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python/3.7.4 (3,943 files, 61MB) *
  Poured from bottle on 2019-07-10 at 13:24:49
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/python.rb

// Installed via pip
➜ ansible --version
ansible 2.7.10
  config file = /Users/me/Desktop/trellis/ansible.cfg
  configured module search path = [u'/Users/me/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.16 (default, Sep  2 2019, 11:59:44) [GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)]

Then I ran trellis init:

➜ hub clone roots/trellis
➜ cd trellis
➜ trellis init
Creating virtualenv in /Users/me/Desktop/trellis/.trellis/virtualenv
✓ Virtualenv created

✓ Dependencies installed

Question: How to I verify dependencies are installed via virtualenv ansible instead of my original ansible?

swalkinshaw · 2019-09-05T22:07:45Z

Question: How to I verify dependencies are installed via virtualenv ansible instead of my original ansible?

It's a little difficult right now because the virtualenv is only "activated" for the trellis-cli process. For now the main way to tell it's working properly is to pick a command (provision for example) manually add some code like:

versionCmd := execCommand("ansible-playbook", "--version")
logCmd(versionCmd, c.UI, true)
versionCmd.Run()

path, err := exec.LookPath("ansible-playbook")
c.UI.Info(path)

I would just uninstall Ansible locally to test this feature which is another way to ensure it works 🤷‍♂

Maybe some helpful debug information like this should be added to the info command (or a new one like environment).

swalkinshaw · 2019-09-06T01:28:22Z

I just pushed a new exec command which should go along with this feature:

trellis exec ansible-playbook --version

So it will exec any command within the activated virtual env. This mirrors Ruby's bundler exec

swalkinshaw · 2019-09-15T16:45:24Z

This should be ready to go now

tangrufus · 2019-09-19T16:50:45Z

$ trellis new testing.com creates 2 .trellis folders:

testing.com/.trellis
testing.com/trellis/.trellis

The first one seems unused (it is empty).

 	_, _ = os.OpenFile(venvPath, os.O_CREATE, 0555)

golang question: Do we need _, _ =?

Other than these 2, everything else looks good to me.

cmd/init.go

trellis/virtualenv_test.go

`TRELLIS_VENV=false` won't activate virtualenv

swalkinshaw · 2019-09-21T04:04:20Z

Thanks @tangrufus. Fixed the issue with creating two .trellis dirs.

Also caused me to do more testing and identified another issue using python3's venv so that's fixed now too 🎉

tangrufus · 2019-09-22T14:06:37Z

Using $ trellis new on f44d238, I see virtualenv is created and there are ansible binaries in trellis/.trellis/virtualenv/bin. However, $ trellis exec ansible --version and trellis galaxy install throw errors.

Am I missing something?

➜ trellis new venv.com
Creating new Trellis project in /Users/me/Desktop/venv.com

Site domain [venv.com]:

✔ venv.com

Fetching latest versions of Trellis and Bedrock...
Creating virtualenv in /Users/me/Desktop/venv.com/trellis/.trellis/virtualenv
✓ Virtualenv created

✓ Dependencies installed

venv.com project created with versions:
  Trellis v1.1.0
  Bedrock v1.12.8

14:57:23 ~/Desktop 22s
➜ cd venv.com/trellis

14:57:31 Desktop/venv.com/trellis
➜ trellis exec ansible --version
Traceback (most recent call last):
  File "/Users/me/Desktop/venv.com/trellis/.trellis/virtualenv/bin/ansible", line 40, in <module>
    from ansible.errors import AnsibleError, AnsibleOptionsError, AnsibleParserError
ModuleNotFoundError: No module named 'ansible'

15:02:02 Desktop/venv.com/trellis
➜ tree -L 3 .trellis
.trellis
└── virtualenv
    ├── bin
    │   ├── activate
    │   ├── activate.csh
    │   ├── activate.fish
    │   ├── ansible
    │   ├── ansible-config
    │   ├── ansible-connection
    │   ├── ansible-console
    │   ├── ansible-doc
    │   ├── ansible-galaxy
    │   ├── ansible-inventory
    │   ├── ansible-playbook
    │   ├── ansible-pull
    │   ├── ansible-vault
    │   ├── easy_install
    │   ├── easy_install-3.7
    │   ├── pip
    │   ├── pip3
    │   ├── pip3.7
    │   ├── python -> python3
    │   └── python3 -> /usr/local/bin/python3
    ├── include
    ├── lib
    │   └── python3.7
    └── pyvenv.cfg

5 directories, 21 files

15:05:38 Desktop/venv.com/trellis
➜ trellis galaxy install
Running command => ansible-galaxy install -r galaxy.yml
Traceback (most recent call last):
  File "/Users/me/Desktop/venv.com/trellis/.trellis/virtualenv/bin/ansible-galaxy", line 40, in <module>
    from ansible.errors import AnsibleError, AnsibleOptionsError, AnsibleParserError
ModuleNotFoundError: No module named 'ansible'
Error running ansible-galaxy: exit status 1

tangrufus · 2019-09-22T14:12:52Z

Besides, TRELLIS_VENV=false seems has no effects on trellis new.

15:09:52 ~/Desktop
➜ TRELLIS_VENV=false trellis new falsey.com
Creating new Trellis project in /Users/me/Desktop/falsey.com

Site domain [falsey.com]:

✔ falsey.com

Fetching latest versions of Trellis and Bedrock...
Creating virtualenv in /Users/me/Desktop/falsey.com/trellis/.trellis/virtualenv
✓ Virtualenv created

✓ Dependencies installed

falsey.com project created with versions:
  Trellis v1.1.0
  Bedrock v1.12.8

15:11:08 ~/Desktop
➜ cd falsey.com/trellis

15:11:09 Desktop/falsey.com/trellis
➜ tree -L 3 .trellis
.trellis
└── virtualenv
    ├── bin
    │   ├── activate
    │   ├── activate.csh
    │   ├── activate.fish
    │   ├── ansible
    │   ├── ansible-config
    │   ├── ansible-connection
    │   ├── ansible-console
    │   ├── ansible-doc
    │   ├── ansible-galaxy
    │   ├── ansible-inventory
    │   ├── ansible-playbook
    │   ├── ansible-pull
    │   ├── ansible-vault
    │   ├── easy_install
    │   ├── easy_install-3.7
    │   ├── pip
    │   ├── pip3
    │   ├── pip3.7
    │   ├── python -> python3
    │   └── python3 -> /usr/local/bin/python3
    ├── include
    ├── lib
    │   └── python3.7
    └── pyvenv.cfg

5 directories, 21 files

swalkinshaw · 2019-09-22T16:22:51Z

TRELLIS_VENV=false stops activation of the venv for any command basically. new has a --skip-virtualenv flag.

Looking to see if I can reproduce your other issue. Do you have virtualenv installed in your global path? If not I'm guessing it's defaulting to using python3 -m venv.

tangrufus · 2019-09-22T16:34:12Z

$ trellis new --skip-virtualenv work as expected!

I don't have virtualenv installed.

➜ which virtualenv
virtualenv not found

If it helps, I installed both python and python@2 via homebrew.

➜ python3 -m venv
usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear]
            [--upgrade] [--without-pip] [--prompt PROMPT]
            ENV_DIR [ENV_DIR ...]
venv: error: the following arguments are required: ENV_DIR

17:36:25 ~
➜ python2 -m venv
/usr/local/opt/python@2/bin/python2.7: No module named venv

17:36:31 ~
➜ python -m venv
/usr/local/opt/python@2/bin/python2.7: No module named venv

Do you need any other info for debugging?

swalkinshaw · 2019-09-22T17:17:55Z

I'm guessing it was the addition of --system-site-packages when using python3's venv module. My issue was that without that flag it wouldn't create the pip binary. I can't reproduce it since it's working fine for me but that's my guess so far.

Can you try removing that flag and see if it helps?

swalkinshaw · 2019-09-22T17:24:41Z

And now I can't reproduce the original problem without --system-site-packages 😰 I think it makes more sense without it anyway. Basically it just means it should fallback to the system package so long as pip exists somewhere, it's fine?

tangrufus · 2019-09-22T17:36:00Z

Removing --system-site-packages works! See:#66

swalkinshaw · 2019-09-22T17:44:42Z

Fixed in here too

tangrufus · 2019-09-22T17:48:12Z

1003b69 works! 🎉 🎉 🎉

swalkinshaw · 2019-09-22T19:20:13Z

Thanks for all the testing ❤️

swalkinshaw changed the title ~~Virtualenv support~~ [WIP] Virtualenv support Aug 17, 2019

swalkinshaw force-pushed the virtualenv-support branch from 45017e6 to f6fcd68 Compare August 17, 2019 22:10

tangrufus reviewed Aug 31, 2019

View reviewed changes

trellis/virtualenv.go Show resolved Hide resolved

tangrufus reviewed Aug 31, 2019

View reviewed changes

swalkinshaw force-pushed the virtualenv-support branch from 364cd99 to c199814 Compare September 6, 2019 00:49

swalkinshaw force-pushed the virtualenv-support branch 2 times, most recently from 3d1159e to f3f58fa Compare September 9, 2019 00:08

swalkinshaw added 3 commits September 15, 2019 12:40

Add github package for downloading releases

411d5e7

Add Virtualenv struct

ece2cc2

Integrate Virtualenv

5b75eea

swalkinshaw force-pushed the virtualenv-support branch from f3f58fa to 24f07da Compare September 15, 2019 16:41

swalkinshaw added 2 commits September 15, 2019 12:43

Add init command

4d90b00

Create config dir if necessary

86ccd8f

swalkinshaw force-pushed the virtualenv-support branch from 24f07da to 2a47431 Compare September 15, 2019 16:44

swalkinshaw changed the title ~~[WIP] Virtualenv support~~ Virtualenv support Sep 15, 2019

swalkinshaw requested a review from tangrufus September 15, 2019 16:45

tangrufus requested changes Sep 19, 2019

View reviewed changes

cmd/init.go Outdated Show resolved Hide resolved

trellis/virtualenv_test.go Outdated Show resolved Hide resolved

trellis/virtualenv_test.go Outdated Show resolved Hide resolved

trellis/virtualenv_test.go Outdated Show resolved Hide resolved

swalkinshaw added 4 commits September 19, 2019 23:52

Install virtualenv as a fallback

e5c6c59

Fix project structure test

29a54c2

Add skipVirtualenv flag to new command

a015154

Support TRELLIS_VENV to disable virtualenv

297977e

`TRELLIS_VENV=false` won't activate virtualenv

swalkinshaw added 2 commits September 19, 2019 23:52

Add exec command

7139f46

Virtualenv tests

9386451

swalkinshaw force-pushed the virtualenv-support branch from 2a47431 to 9386451 Compare September 20, 2019 03:52

Fix CreateConfigDir

63910bd

swalkinshaw force-pushed the virtualenv-support branch from e031cc5 to f44d238 Compare September 21, 2019 04:06

tangrufus mentioned this pull request Sep 22, 2019

Remove --system-site-packages #66

Closed

Set proper env name

1003b69

swalkinshaw force-pushed the virtualenv-support branch from f44d238 to 1003b69 Compare September 22, 2019 17:42

tangrufus approved these changes Sep 22, 2019

View reviewed changes

swalkinshaw merged commit 75aed6d into master Sep 22, 2019

swalkinshaw deleted the virtualenv-support branch September 22, 2019 20:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Virtualenv support #55

Virtualenv support #55

swalkinshaw commented Aug 17, 2019 •

edited

Loading

swalkinshaw commented Aug 18, 2019

austinpray commented Aug 18, 2019

swalkinshaw commented Aug 18, 2019 via email

tangrufus Aug 31, 2019

swalkinshaw Sep 2, 2019

swalkinshaw commented Sep 5, 2019

tangrufus commented Sep 5, 2019 •

edited

Loading

swalkinshaw commented Sep 5, 2019 •

edited

Loading

swalkinshaw commented Sep 6, 2019

swalkinshaw commented Sep 15, 2019

tangrufus commented Sep 19, 2019 •

edited

Loading

swalkinshaw commented Sep 21, 2019

tangrufus commented Sep 22, 2019 •

edited

Loading

tangrufus commented Sep 22, 2019

swalkinshaw commented Sep 22, 2019

tangrufus commented Sep 22, 2019 •

edited

Loading

swalkinshaw commented Sep 22, 2019

swalkinshaw commented Sep 22, 2019

tangrufus commented Sep 22, 2019

swalkinshaw commented Sep 22, 2019

tangrufus commented Sep 22, 2019

swalkinshaw commented Sep 22, 2019

Virtualenv support #55

Virtualenv support #55

Conversation

swalkinshaw commented Aug 17, 2019 • edited Loading

swalkinshaw commented Aug 18, 2019

austinpray commented Aug 18, 2019

swalkinshaw commented Aug 18, 2019 via email

tangrufus Aug 31, 2019

Choose a reason for hiding this comment

swalkinshaw Sep 2, 2019

Choose a reason for hiding this comment

swalkinshaw commented Sep 5, 2019

tangrufus commented Sep 5, 2019 • edited Loading

swalkinshaw commented Sep 5, 2019 • edited Loading

swalkinshaw commented Sep 6, 2019

swalkinshaw commented Sep 15, 2019

tangrufus commented Sep 19, 2019 • edited Loading

swalkinshaw commented Sep 21, 2019

tangrufus commented Sep 22, 2019 • edited Loading

tangrufus commented Sep 22, 2019

swalkinshaw commented Sep 22, 2019

tangrufus commented Sep 22, 2019 • edited Loading

swalkinshaw commented Sep 22, 2019

swalkinshaw commented Sep 22, 2019

tangrufus commented Sep 22, 2019

swalkinshaw commented Sep 22, 2019

tangrufus commented Sep 22, 2019

swalkinshaw commented Sep 22, 2019

swalkinshaw commented Aug 17, 2019 •

edited

Loading

tangrufus commented Sep 5, 2019 •

edited

Loading

swalkinshaw commented Sep 5, 2019 •

edited

Loading

tangrufus commented Sep 19, 2019 •

edited

Loading

tangrufus commented Sep 22, 2019 •

edited

Loading

tangrufus commented Sep 22, 2019 •

edited

Loading