Env.run_python_script should use the executable returned by sys.executable, avoiding "Unknown option: -I"

[hartzell]

• Poetry version: 1.3.0 or newer
• Python version: 3.11.3, but not really because /usr/bin/python is 2.7.16
• OS version and name: macOS 11.7.6
• pyproject.toml: N/A
• I am on the latest stable Poetry version, installed using a recommended method.
  • This issue is reporting a failure when installing via the curl method.
• I have searched the issues of this repo and believe that this is not a duplicate.
  • it is related to getting error while running poetry show #7854
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

poetry -vvv show output

poetry -vvv show
Creating virtualenv foo-cUWjX3Ei-py3.11 in /Users/georgehartzell/Library/Caches/pypoetry/virtualenvs

  Stack trace:

  1  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/utils/env.py:1501 in _run
      1499│         try:
      1500│             if input_:
    → 1501│                 output = subprocess.run(
      1502│                     cmd,
      1503│                     stdout=subprocess.PIPE,

  CalledProcessError

  Command '['python', '-I', '-W', 'ignore', '-']' returned non-zero exit status 2.

  at /usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py:571 in run
       567│             # We don't call process.wait() as .__exit__ does that for us.
       568│             raise
       569│         retcode = process.poll()
       570│         if check and retcode:
    →  571│             raise CalledProcessError(retcode, process.args,
       572│                                      output=stdout, stderr=stderr)
       573│     return CompletedProcess(process.args, retcode, stdout, stderr)
       574│
       575│

The following error occurred when trying to handle this error:


  Stack trace:

  13  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/cleo/application.py:327 in run
       325│
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  12  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│

  11  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│

  10  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/cleo/application.py:473 in _run_command
       471│
       472│         if error is not None:
     → 473│             raise error
       474│
       475│         return terminate_event.exit_code

   9  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/cleo/application.py:454 in _run_command
       452│
       453│         try:
     → 454│             self._event_dispatcher.dispatch(command_event, COMMAND)
       455│
       456│             if command_event.command_should_run():

   8  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/cleo/events/event_dispatcher.py:26 in dispatch
        24│
        25│         if listeners:
     →  26│             self._do_dispatch(listeners, event_name, event)
        27│
        28│         return event

   7  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/cleo/events/event_dispatcher.py:89 in _do_dispatch
        87│                 break
        88│
     →  89│             listener(event, event_name, self)
        90│
        91│     def _sort_listeners(self, event_name: str) -> None:

   6  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/console/application.py:299 in configure_env
       297│
       298│         env_manager = EnvManager(poetry, io=io)
     → 299│         env = env_manager.create_venv()
       300│
       301│         if env.is_venv() and io.is_verbose():

   5  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/utils/env.py:1045 in create_venv
       1043│             return self.get_system_env()
       1044│
     → 1045│         return VirtualEnv(venv)
       1046│
       1047│     @classmethod

   4  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/utils/env.py:1669 in __init__
       1667│         # from inside the virtualenv.
       1668│         if base is None:
     → 1669│             output = self.run_python_script(GET_BASE_PREFIX)
       1670│             self._base = Path(output.strip())
       1671│

   3  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/utils/env.py:1479 in run_python_script
       1477│
       1478│     def run_python_script(self, content: str, **kwargs: Any) -> str:
     → 1479│         return self.run(
       1480│             self._executable,
       1481│             "-I",

   2  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/utils/env.py:1471 in run
       1469│     def run(self, bin: str, *args: str, **kwargs: Any) -> str:
       1470│         cmd = self.get_command_from_bin(bin) + list(args)
     → 1471│         return self._run(cmd, **kwargs)
       1472│
       1473│     def run_pip(self, *args: str, **kwargs: Any) -> str:

   1  ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/utils/env.py:1723 in _run
       1721│     def _run(self, cmd: list[str], **kwargs: Any) -> str:
       1722│         kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
     → 1723│         return super()._run(cmd, **kwargs)
       1724│
       1725│     def get_temp_environ(

  EnvCommandError

  Command ['python', '-I', '-W', 'ignore', '-'] errored with the following return code 2

  Error output:
  Unknown option: -I
  usage: /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python [option] ... [-c cmd | -m mod | file | -] [arg] ...
  Try `python -h' for more information.


  Input:
  import sys

  if hasattr(sys, "real_prefix"):
      print(sys.real_prefix)
  elif hasattr(sys, "base_prefix"):
      print(sys.base_prefix)
  else:
      print(sys.prefix)


  at ~/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/utils/env.py:1517 in _run
      1513│                 output = ""
      1514│             else:
      1515│                 output = subprocess.check_output(cmd, stderr=stderr, env=env, **kwargs)
      1516│         except CalledProcessError as e:
    → 1517│             raise EnvCommandError(e, input=input_)
      1518│
      1519│         return decode(output)
      1520│
      1521│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

Issue

Poetry installed via curl only works if executable returned which python is python version 3. Otherwise it complains about Unknown option: -I.

A rough fix is for Env.run_python_script to use the same python interpreter that's running poetry (e.g. as returned by sys.executable), rather than invoking python (see POC change below).

---

I'm on a mac where /usr/bin/python is 2.7.16 and/but I have python 3.11.3 installed via homebrew in /usr/local/bin . The homebrew recipe does not provide an executable named python that is version 3. In other words, which python returns /usr/bin/python and running python gets v2.7.16.

I've installed python using the recommended method:

curl -sSL https://install.python-poetry.org | python3 -

This creates a virtual environment for poetry in /Users/georgehartzell/Library/Application Support/pypoetry/venv/bin/python and ~/.local/bin/poetry has a shebang line that ensure that the python executable used (named python) is from that venv.

This is enough to let simple commands run:

➜  foo which poetry
/Users/georgehartzell/.local/bin/poetry
➜  foo poetry --version3.4
Poetry (version 1.5.0)

BUT, Env.run_python_script, here invokes an executable named python with the -I flag, which is new in python 3 (3.4, I think?).

Because the function does not activate poetry's virtual environment, it ends up running /usr/bin/python, which in my case is 2.7.11 and does not support -I.

This results in the error message in the "requested info" section above.

Poetry 1.5.0 works if I activate poetry's virtual environment manually first:

. "/Users/georgehartzell/Library/Application Support/pypoetry/venv/bin/activate"

Things also works if I manually create a virtual environment, activate it, use pip to install Poetry in the venv, and work with it activated.

A proof of concept fix is (apologies for just editing the installed python, but I just wanted to Prove the Concept and didn't want to figure out how to replicate the curl install for a branched fix):

$ diff -u "/Users/georgehartzell/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/utils/env.py"  "/Users/georgehartzell/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/utils/env.py.orig"
--- /Users/georgehartzell/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/utils/env.py	2023-05-25 08:04:05.000000000 -0600
+++ /Users/georgehartzell/Library/Application Support/pypoetry/venv/lib/python3.11/site-packages/poetry/utils/env.py.orig	2023-05-25 08:03:40.000000000 -0600
@@ -1205,7 +1205,7 @@
         self._path = path
         self._bin_dir = self._path / bin_dir

-        self._executable = sys.executable or "python"
+        self._executable = "python"
         self._pip_executable = "pip"

         self.find_executables()

[dimbleby]

this doesn't look right, but per #7854 (comment) I expect this already is fixed anyway

[hartzell]

I'm not sure what doesn't look right about it, but it does line up with various other issues (including #7854, cited above). I'm not attached to the little POC hack as the right way to fix it, but it does seem that Poetry needs to do something to ensure that it's using the venv that it built when it was installed via curl.

I'm not sure which fix you're referring to in #7977 which seems to concern in-project-venv and does not seem relevant to what I'm seeing. I'm initializing a fresh project via poetry init and do not have any personal config files set.

Also #7597, only saw the failure with v1.5.0, v1.4.2 worked for them.

On the other hand, I see the failure for releases >= 1.3.0, where the -I argument was added.

Here's an example with v1.4.2

$ curl -sSL https://install.python-poetry.org | python3 - --uninstall Removing Poetry (1.5.0) $ curl -sSL https://install.python-poetry.org | python3 - --version 1.4.2 Retrieving Poetry metadata

Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the poetry command to Poetry's bin directory, located at:

/Users/georgehartzell/.local/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (1.4.2): Done

Poetry (1.4.2) is installed now. Great!

You can test that everything is set up by executing:

poetry --version

$ mkdir foo
$ cd foo
$ which poetry
/Users/georgehartzell/.local/bin/poetry
$ which python
/usr/bin/python
$ which python3
/usr/local/bin/python3
$ poetry init

This command will guide you through creating your pyproject.toml config.

Package name [foo]:
Version [0.1.0]:
Description []: foo
Author [george.hartzell george.hartzell@sana.com, n to skip]:
License []:
Compatible Python versions [^3.11]:

Would you like to define your main dependencies interactively? (yes/no) [yes]
You can specify a package in the following forms:

• A single name (requests): this will search for matches on PyPI
• A name and a constraint (requests@^2.23.0)
• A git url (git+https://github.com/python-poetry/poetry.git)
• A git url with a revision (git+https://github.com/python-poetry/poetry.git#develop)
• A file path (../my-package/my-package.whl)
• A directory (../my-package/)
• A url (https://example.com/packages/my-package-0.1.0.tar.gz)

Package to add or search for (leave blank to skip): requests
Found 20 packages matching requests
Showing the first 10 matches

Enter package # to add, or the complete package name if it is not listed []:
[ 0] requests
[ 1] requests5
[ 2] requests2
[ 3] requests3
[ 4] play-requests
[ 5] pycopy-requests
[ 6] batch-requests
[ 7] requests-threads
[ 8] requests_spider
[ 9] requests-middleware
[ 10]

0
Enter the version constraint to require (or leave blank to use the latest version):
Using version ^2.31.0 for requests

Add a package (leave blank to skip):

Would you like to define your development dependencies interactively? (yes/no) [yes] no
Generated file

[tool.poetry]
name = "foo"
version = "0.1.0"
description = "foo"
authors = ["george.hartzell george.hartzell@sana.com"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"
requests = "^2.31.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Do you confirm generation? (yes/no) [yes] yes
$ poetry install
Creating virtualenv foo-XlONOBww-py3.11 in /Users/georgehartzell/Library/Caches/pypoetry/virtualenvs

Command ['python', '-I', '-W', 'ignore', '-'] errored with the following return code 2

Error output:
Unknown option: -I
usage: /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.

Input:
import sys

if hasattr(sys, "real_prefix"):
print(sys.real_prefix)
elif hasattr(sys, "base_prefix"):
print(sys.base_prefix)
else:
print(sys.prefix)

$

[hartzell]

Just checked, and my version of the problem is not fixed in the master branch (just installed specifying master, should be commit f3f71ea):

Example of failure using the master branch.

$ curl -sSL https://install.python-poetry.org | python3 - --uninstall
Removing Poetry (1.5.0)
$ curl -sSL https://install.python-poetry.org | python3 - --git https://github.com/python-poetry/poetry.git@master

Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the poetry command to Poetry's bin directory, located at:

/Users/georgehartzell/.local/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (https://github.com/python-poetry/poetry.git@master): Done

Poetry (https://github.com/python-poetry/poetry.git@master) is installed now. Great!

You can test that everything is set up by executing:

poetry --version

$ which poetry
/Users/georgehartzell/.local/bin/poetry
$ poetry --version
Poetry (version 1.6.0.dev0)
$ ls
pyproject.toml
$ rm *
zsh: sure you want to delete the only file in /Users/georgehartzell/.emacs.d/foo [yn]? y
$
$ poetry init

This command will guide you through creating your pyproject.toml config.

Package name [foo]:
Version [0.1.0]:
Description []: foo
Author [george.hartzell george.hartzell@sana.com, n to skip]:
License []:
Compatible Python versions [^3.11]:

Would you like to define your main dependencies interactively? (yes/no) [yes] yes
You can specify a package in the following forms:

• A single name (requests): this will search for matches on PyPI
• A name and a constraint (requests@^2.23.0)
• A git url (git+https://github.com/python-poetry/poetry.git)
• A git url with a revision (git+https://github.com/python-poetry/poetry.git#develop)
• A file path (../my-package/my-package.whl)
• A directory (../my-package/)
• A url (https://example.com/packages/my-package-0.1.0.tar.gz)

Package to add or search for (leave blank to skip): requests
Found 20 packages matching requests
Showing the first 10 matches

Enter package # to add, or the complete package name if it is not listed []:
[ 0] requests
[ 1] requests5
[ 2] requests2
[ 3] requests3
[ 4] play-requests
[ 5] pycopy-requests
[ 6] batch-requests
[ 7] requests-threads
[ 8] requests_spider
[ 9] requests-middleware
[ 10]

0
Enter the version constraint to require (or leave blank to use the latest version):
Using version ^2.31.0 for requests

Add a package (leave blank to skip):

Would you like to define your development dependencies interactively? (yes/no) [yes] no
Generated file

[tool.poetry]
name = "foo"
version = "0.1.0"
description = "foo"
authors = ["george.hartzell george.hartzell@sana.com"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"
requests = "^2.31.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Do you confirm generation? (yes/no) [yes] yes
$ poetry install

Command ['python', '-I', '-W', 'ignore', '-'] errored with the following return code 2

Error output:
Unknown option: -I
usage: /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.

Input:
import sys

if hasattr(sys, "real_prefix"):
print(sys.real_prefix)
elif hasattr(sys, "base_prefix"):
print(sys.base_prefix)
else:
print(sys.prefix)

$

[dimbleby]

the latest comment in #7854 picks out virtualenvs.in-project as necessary to reproduce, that version probably is a duplicate as suggested. But there can be more than one thing happening with similar symptoms.

When I say "this doesn't look right" I mean that falling back to poetry's own python interpreter does not look like a desirable fix. It might or might not be safe for poetry show but it's likely to break poetry if you accidentally poetry install using that interpreter.

it would probably be better to have no fallback at all, and fail more explicitly if failing to find a good python.

[hartzell]

I mean that falling back to poetry's own python interpreter does not look like a desirable fix. It might or might not be safe for poetry show but it's likely to break poetry if you accidentally poetry install using that interpreter.

Thanks for the clarification.

If I can restate your concern to see if I understand it, are you're worried that (for example) there are times when one would want to run a python script in the context of the project being worked on? Running e.g. /usr/local/bin/python or /usr/bin/python, or whatever python happened to resolve to wouldn't support that either.

Because python is what it is (XKCD #1987), it seems that insisting that the thing called python is python 3 isn't really safe.

For kicks, I changed the function to call python3 instead of python (in v1.4.2) and this "fix" (hahaha) works (init, install, add) too. I like this less than just using the venv that poetry set up for itself, but perhaps there should be a config variable to speficy the python name?

Call python3 instead of python

   self._executable = "python3"

•    self._executable = "python"
     self._pip_executable = "pip"
  
     self.find_executables()
  

</details>

Can you describe which `python` _should_ be being used?

[dimbleby]

poetry nearly always wants to run in the context of the virtual environment associated with the project being managed. virtual environments are required to have a python that is the right python. So it's not a crazy default.

guessing at python3 feels like a more reasonable try, but also bound to expose examples where the opposite is desired.

I don't know why we're having this discussion in a new issue: if you want to submit an MR then please submit an MR, if you want to discuss #7854 then let's do it there.

[odoublewen]

@dimbleby +1 for this comment:

it would probably be better to have no fallback at all, and fail more explicitly if failing to find a good python.

[dimbleby]

@hartzell please close this as a duplicate

[hartzell]

it would probably be better to have no fallback at all, and fail more explicitly if failing to find a good python.

The only problem with this is that it means that, at least on my may, poetry as installed by the recommended route doesn't work.

[hartzell]

@hartzell please close this as a duplicate

I'm not convinced that it's an exact duplicate of #7854, but I'll close it as a duplicate and continue the conversation over there.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.