Problems with poetry and incorrect python version

[ColdTeapot273K]

Environment:
clearml==1.1.6
clearml-agent==1.1.1
poetry==1.1.12
pyenv==2.2.3
OS==4.14.256-197.484.amzn2.x86_64

---

The problem:
If I override system python with arbitrary python version using pyenv with pyenv global ..., and configure poetry as package manager in clearml.conf, a clearml-agent worker can crash on startup because it tries to link to system python, rather than pyenv python, which is easily incompatibe with dependencies.

E.g. Amazon Linux 2 standard python is 3.7 while fresh numpy requires python 3.9:

This is not solved by pointing python_binary: in clearml.conf to the pyenv python binary explicitly, see #523 (comment)

---

Solution:

I've tracked the issue to poetry repo, e.g. python-poetry/poetry#4199, python-poetry/poetry#4852 among others. The problem is, poetry with the new installer (install-poetry.py) doesn't respect pyenv global setting untill explicitly told to use that python in the current session via poetry use ....
Now, we cannot explicitly add any shell commands on clearml-agent startup (poetry use ...) unless we use docker (at least i didn't found any other way).
And poetry + docker don't play together well (besides docker would be overkill to use for any task).

What can be done is:

1. either using unstable poetry 1.2 patched with Add (experimental) option to use current active python to create venv ("pyenv way") python-poetry/poetry#4852
2. or, more stable way - use poetry installed with deprecateed installer (the get-poetry.py, not the install-poetry.py)

Maybe add some note to clearml docs?

[jkhenning]

Hi @ColdTeapot273K,

Thanks for sharing such a detailed description and possible solutions 👍

Maybe add some note to clearml docs?

I think the FAQ is a good place to start, WDTY?

[ainoam]

@ColdTeapot273K I've taken the liberty to add your comments here, Makes sense?