Non UTF-8 encoding in terminal breaks poetry commands

[Makman2]

• Poetry version: 1.6.1 and 1.5.0
• Python version: 3.11.2
• OS version and name: Raspberry PI OS (Linux raspberrypi 6.1.0-rpi4-rpi-v8 #1 SMP PREEMPT Debian 1:6.1.54-1+rpt2 (2023-10-05) aarch64 GNU/Linux)
• pyproject.toml: Might be any valid and simple project.

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• [ x I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

(Maybe similar issue: #6784)

Hi 😄

I was developing a project for a Raspberry Pi for with a recent Raspberry Pi OS (installed last week - see above). On my development machine (Ubuntu 20.04 LTS) everything worked fine, I could use poetry to add and remove dependencies etc. poetry install did work too.

However, once I migrated it to the Raspberry Pi, I couldn't do poetry install and it silently exited without installing anything. Because the default poetry installation coming from the apt sources is too old (v1.3.2), I used a new virtualenv to get the newest Poetry (v1.6.1).

python3 -m venv venv
source venv/bin/activate
python3 -m pip install poetry
poetry --version  # yields: Poetry (version 1.6.1)

That seems to be working.

However, when I now cd into my project and do poetry install -vvv, the output is just:

Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 11 installs, 2 updates, 0 removals, 4 skipped

Nothing happens, but poetry errors out with exit-code 1.

When I manually attempt to add a dependency again, e.g. poetry add click -vvv, I finally get a traceback:

Using virtualenv: /home/pi/dev/critical-vibration-experiment2/venv
The following packages are already present in the pyproject.toml and will be skipped:


  Stack trace:

  13  venv/lib/python3.11/site-packages/cleo/application.py:327 in run
       325|
       326|             try:
     > 327|                 exit_code = self._run(io)
       328|             except BrokenPipeError:
       329|                 # If we are piped to another process, it may close early and send a

  12  venv/lib/python3.11/site-packages/poetry/console/application.py:190 in _run
       188|         self._load_plugins(io)
       189|
     > 190|         exit_code: int = super()._run(io)
       191|         return exit_code
       192|

  11  venv/lib/python3.11/site-packages/cleo/application.py:431 in _run
       429|             io.input.interactive(interactive)
       430|
     > 431|         exit_code = self._run_command(command, io)
       432|         self._running_command = None
       433|

  10  venv/lib/python3.11/site-packages/cleo/application.py:473 in _run_command
       471|
       472|         if error is not None:
     > 473|             raise error
       474|
       475|         return terminate_event.exit_code

   9  venv/lib/python3.11/site-packages/cleo/application.py:457 in _run_command
       455|
       456|             if command_event.command_should_run():
     > 457|                 exit_code = command.run(io)
       458|             else:
       459|                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

   8  venv/lib/python3.11/site-packages/cleo/commands/base_command.py:117 in run
       115|         io.input.validate()
       116|
     > 117|         return self.execute(io) or 0
       118|
       119|     def merge_application_definition(self, merge_args: bool = True) -> None:

   7  venv/lib/python3.11/site-packages/cleo/commands/command.py:61 in execute
        59|
        60|         try:
     >  61|             return self.handle()
        62|         except KeyboardInterrupt:
        63|             return 1

   6  venv/lib/python3.11/site-packages/poetry/console/commands/add.py:152 in handle
       150|
       151|         if existing_packages:
     > 152|             self.notify_about_existing_packages(existing_packages)
       153|
       154|         packages = [name for name in packages if name not in existing_packages]

   5  venv/lib/python3.11/site-packages/poetry/console/commands/add.py:301 in notify_about_existing_packages
       299|         )
       300|         for name in existing_packages:
     > 301|             self.line(f"  \u2022 {name}")
       302|         self.line(self._hint_update_packages)
       303|

   4  venv/lib/python3.11/site-packages/cleo/commands/command.py:237 in line
       235|         styled = f"<{style}>{text}" if style else text
       236|
     > 237|         self._io.write_line(styled, verbosity=verbosity)
       238|
       239|     def line_error(

   3  venv/lib/python3.11/site-packages/cleo/io/io.py:52 in write_line
        50|         type: OutputType = OutputType.NORMAL,
        51|     ) -> None:
     >  52|         self._output.write_line(messages, verbosity=verbosity, type=type)
        53|
        54|     def write(

   2  venv/lib/python3.11/site-packages/cleo/io/outputs/output.py:86 in write_line
        84|         type: Type = Type.NORMAL,
        85|     ) -> None:
     >  86|         self.write(messages, new_line=True, verbosity=verbosity, type=type)
        87|
        88|     def write(

   1  venv/lib/python3.11/site-packages/cleo/io/outputs/output.py:107 in write
       105|                 message = strip_tags(self._formatter.format(message))
       106|
     > 107|             self._write(message, new_line=new_line)
       108|
       109|     def flush(self) -> None:

  UnicodeEncodeError

  'latin-1' codec can't encode character '\u2022' in position 2: ordinal not in range(256)

  at venv/lib/python3.11/site-packages/cleo/io/outputs/stream_output.py:78 in _write
       74|     def _write(self, message: str, new_line: bool = False) -> None:
       75|         if new_line:
       76|             message += "\n"
       77|
    >  78|         self._stream.write(message)
       79|         self._stream.flush()
       80|
       81|     def _has_color_support(self) -> bool:
       82|         # Follow https://no-color.org/

So something encoding related. And as suspected, in contrast to a standard Ubuntu where the terminal encoding is "utf-8", the encoding on a Raspberry Pi terminal (or SSH session) is:

python3 -c "import sys; print(sys.stdout.encoding)"
# output: iso8859-1

I tried to work around this issue successfully by using

export PYTHONIOENCODING=utf-8

The output is a bit broken (the fancy progress char animations don't work exactly like they should), but at least it installs stuff again.

Wild guessing from the traceback I would assume that the cleo package has no encoding fallback, or poetry does not use cleo with an explicit encoding fallback specified (or some non-strict encoding parameter).

I also tried this with a clean new poetry project after poetry init and the same effect appears on my Raspberry Pi. So I highly doubt this is pyproject.toml related (but if you still need a pyproject.toml example please feel free to ask for it 😁).

So to clarify, there are 2 bugs here:

• For poetry install, the encoding bug is silently swallowed and no traceback is visible. At least there is a non-zero return code.
• The actual bug that poetry doesn't work at all (or at least poetry add and poetry install) when the output terminal has a non-UTF8 encoding (there might be other encodings that work different from UTF-8, but just for the sake of simplicity).

[dimbleby]

looks the same as #1427, #3078, #3672 - which mostly seem to claim that you should live in an environment that does support utf-8.

not sure I buy that. IMO it would be fine to remove the bullets from poetry install and just use regular hyphens. But the suggestion of those earlier issues is that this might meet resistance.

You could submit a merge request trying it, if you want to find out.

[Makman2]

Thanks for finding those. I just facepalm myself a bit because I forgot searching for already closed issues... sorry for that.

Nevertheless, if the requirement is to have a working UTF-8 environment, then I would at least suspect a message to tell me to do so. Or something similar. In any case, I think this is not specified well enough. And at least that should be in the FAQs. Otherwise nobody will easily find that.
So I also definitely disagree with the linked opinion. For me this comment summarizes it well.

You could submit a merge request trying it, if you want to find out.

Don't get me wrong, but I don't want to take the time to maybe find out that my PR will be simply closed because of "Won't do" 😉

[dimbleby]

I don't want to take the time to maybe find out that my PR will be simply closed because of "Won't do"

I understand; but I don't have the power to tell you that it will or won't

I would guess that it's not a very hard search-replace in the codebase - ie it shouldn't take very much time to try it. But that's easy to say from the perspective of someone who has no intention of doing it themselves.

[Secrus]

Fixed in #8608

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.