Fix losing the venv context on Linux

[karlshea]

Possibly fixes #42

It looks like calling the executable as a quoted string is doing something when executed to lose the venv context, which is causing all of the module not found errors.

[swar]

Can you share the error that you were getting? I rejected this exact commit from someone else because it was working for everyone besides certain Linux users. If I can get a better exception that happens to you, I can see if there is a way I can make it more cross-platform.

[karlshea]

It's all my comments in #42, everything works until the viewer reloads and then it can't find the modules anymore:

Restarting view due to psutil going stale...
Traceback (most recent call last):
  File "manager.py", line 4, in <module>
    from plotmanager.library.utilities.commands import start_manager, stop_manager, view, analyze_logs
  File "/root/Swar-Chia-Plot-Manager/plotmanager/library/utilities/commands.py", line 3, in <module>
    import psutil
ModuleNotFoundError: No module named 'psutil'

After installing python3-psutil so it was available globally, it errored on the next import which was 'dateutil'.

I think passing the quoted string is causing execv to create the process outside of the current venv? Or maybe the python interpreter inside the venv uses its path to find the modules directory, and isn't accepting the quotes?

[karlshea]

This is the installed modules inside the venv:

(venv) root@citywall:/opt/plot-manager# pip list
Package         Version
--------------- ---------
certifi         2020.12.5
chardet         4.0.0
dateparser      1.0.0
discord-notify  1.0.0
idna            2.10
pip             20.0.2
pkg-resources   0.0.0
playsound       1.2.2
psutil          5.8.0
python-dateutil 2.8.1
python-pushover 0.4
pytz            2021.1
PyYAML          5.4.1
regex           2021.4.4
requests        2.25.1
setuptools      44.0.0
six             1.16.0
tzlocal         2.1
urllib3         1.26.4
wheel           0.36.2

[karlshea]

Minimum reproduction

(test.py)

import dateutil

print('test!')

Not working

import sys
import os

system_args = [f'"{sys.executable}"'] + ['test.py']
os.execv(sys.executable, system_args)

(venv) $ python launch.py
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    import dateutil
ModuleNotFoundError: No module named 'dateutil'

Working

import sys
import os

system_args = [f'{sys.executable}'] + ['test.py']
os.execv(sys.executable, system_args)

(venv) $ python launch.py
test!

[karlshea]

It is an issue with the venv executable's path. Both the executable path is wrong and the venv packages directory isn't added to sys.path.

test.py

import sys

print(sys.path)
print(sys.executable)
print(sys.argv)

With quotes

['/opt/plot-manager', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']
/opt/plot-manager/"/opt/plot-manager/venv/bin/python"
['test.py']

Without quotes

['/opt/plot-manager', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/opt/plot-manager/venv/lib/python3.8/site-packages']
/opt/plot-manager/venv/bin/python
['test.py']

[Feromrk]

I can confim, that this PR fixed my issue when running python manager.py view

Restarting view due to psutil going stale...
Traceback (most recent call last):
  File "manager.py", line 36, in <module>
    view()
  File "/home/johannes/repos/Swar-Chia-Plot-Manager/plotmanager/library/utilities/commands.py", line 123, in view
    os.execv(sys.executable, system_args)
FileNotFoundError: [Errno 2] No such file or directory

For the installation of swar I followed the instructions in detail (including venv).

cat /etc/issue

Ubuntu 20.04.2 LTS \n \l

Inside venv:

pip list

Package         Version
--------------- ---------
certifi         2020.12.5
chardet         4.0.0
dateparser      1.0.0
discord-notify  1.0.0
idna            2.10
pip             20.0.2
pkg-resources   0.0.0
playsound       1.2.2
psutil          5.8.0
python-dateutil 2.8.1
python-pushover 0.4
pytz            2021.1
PyYAML          5.4.1
regex           2021.4.4
requests        2.25.1
setuptools      44.0.0
six             1.16.0
tzlocal         2.1
urllib3         1.26.4

python --version

Python 3.8.5

[swar]

@karlshea

Sorry for the delay in my response. I have been trying to knock out a bunch of these PRs while also answering the pile of questions I've been getting messaged from every different avenue.

I ran this change without the quotes and here is the issue that I have:

Restarting view due to psutil going stale...

> A:\Program Files\Python\Python38\python.exe: can't open file 'Storage\Github\plotter\venv\Scripts\python.exe': [Errno 2] No such file or directory

Now I wonder if this issue is how Linux works in general or if it's specifically with certain Linux users. Because if it's just with certain Linux users, this is a super easy fix for me to implement with just detecting if it's Windows or Linux.

[swar]

Oh it doesn't work if you have spaces like I do right now, that's why I added quotes. Need to figure out a quick fix to this issue

[swar]

@karlshea

when you get a moment, can you see if the following line of code works for you.

system_args = ['python'] + sys.argv

I think that should work, I just need a quick sanity check. I don't know why I didn't do that before.

[karlshea]

Sorry for the delay in my response. I have been trying to knock out a bunch of these PRs while also answering the pile of questions I've been getting messaged from every different avenue.

No worries! I see that, it looks overwhelming tbh. Especially when the answers to a lot of this are "your Chia install is broken" or "you need to forward port 8444" or "you should learn about file permissions" 😆

see if the following line of code works for you.

Lol that totally works... it seems so obvious but I didn't think of it either. I'm sort of flabbergasted that there's not a way to do this more easily, I did a lot of googling about it before this PR.

There is a difference though if I look at my minimal case: the search path doesn't include the venv packages, but when I run venv/bin/python manager.py view it's refreshing just fine so it must be able to find them.

test.py

import sys

print(f'path: {sys.path}')
print(f'executable: {sys.executable}')
print(f'argv: {sys.argv}')

Before

launch.py

import sys
import os

system_args = [f'{sys.executable}'] + ['test.py']
os.execv(sys.executable, system_args)

$ venv/bin/python launch.py
path: ['/opt/plot-manager', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/opt/plot-manager/venv/lib/python3.8/site-packages']
executable: /opt/plot-manager/venv/bin/python
argv: ['test.py']

After

launch.py

import sys
import os

system_args = ['python'] + ['test.py']
os.execv(sys.executable, system_args)

$ venv/bin/python launch.py
path: ['/opt/plot-manager', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/karl/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']
executable:
argv: ['test.py']

[swar]

Thank you! Very quick response. I'll get this merged out to the development branch.

[karlshea]

Thank you!