UnicodeDecodeError on Windows when not using `-vv`.

[Holt59]

When running tox or tox -v on my project, I get the UnicodeDecodeError below, but not if I run tox -vv.

How to reproduce?

I can reproduce with this toy example: https://github.com/ferencberes/tox-example

System

• Windows 10 Build 18362
• tox --version: 3.14.6
• python --version: Python 3.7.2

---

The error I get:

GLOB sdist-make: I:\Projects\Python\tox-example\setup.py
tox_example_env create: I:\Projects\Python\tox-example\.tox\tox_example_env
tox_example_env installdeps: pytest, numpy, networkx
tox_example_env inst: I:\Projects\Python\tox-example\.tox\.tmp\package\1\tox-example-0.0.1.zip
___________________________________________________________________________________________ summary ___________________________________________________________________________________________
  tox_example_env: commands succeeded
ERROR:   tox_2-7-10: undefined
Traceback (most recent call last):
  File "i:\languages\python\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "i:\languages\python\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "I:\Languages\Python\Python37\Scripts\tox.exe\__main__.py", line 9, in <module>
  File "i:\languages\python\python37\lib\site-packages\tox\session\__init__.py", line 44, in cmdline
    main(args)
  File "i:\languages\python\python37\lib\site-packages\tox\session\__init__.py", line 68, in main
    exit_code = session.runcommand()
  File "i:\languages\python\python37\lib\site-packages\tox\session\__init__.py", line 194, in runcommand
    return self.subcommand_test()
  File "i:\languages\python\python37\lib\site-packages\tox\session\__init__.py", line 222, in subcommand_test
    run_sequential(self.config, self.venv_dict)
  File "i:\languages\python\python37\lib\site-packages\tox\session\commands\run\sequential.py", line 18, in run_sequential
    installpkg(venv, venv.package)
  File "i:\languages\python\python37\lib\site-packages\tox\session\commands\run\sequential.py", line 46, in installpkg
    venv.installpkg(path, action)
  File "i:\languages\python\python37\lib\site-packages\tox\venv.py", line 374, in installpkg
    self.install_pkg(sdistpath, action, "inst")
  File "i:\languages\python\python37\lib\site-packages\tox\venv.py", line 368, in install_pkg
    self._install(target, extraopts=pip_flags, action=action)
  File "i:\languages\python\python37\lib\site-packages\tox\venv.py", line 459, in _install
    self.run_install_command(packages=packages, options=options, action=action)
  File "i:\languages\python\python37\lib\site-packages\tox\venv.py", line 409, in run_install_command
    env=env,
  File "i:\languages\python\python37\lib\site-packages\tox\venv.py", line 576, in _pcall
    report_fail=not is_test_command,
  File "i:\languages\python\python37\lib\site-packages\tox\action.py", line 118, in popen
    lines = out_path.read_text("UTF-8").split("\n")
  File "i:\languages\python\python37\lib\site-packages\py\_path\common.py", line 171, in read_text
    return f.read()
  File "i:\languages\python\python37\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xeb in position 537: invalid continuation byte

[gaborbernat]

Feel free to open a PR against master and fix it. My available efforts at the moment are aimed at fixing this as part of #1394, but that probably will take a while (ETA September).

[jtpavlock]

I'm unable to reproduce this (the -vv part) with python 3.8.5

However, I get a similar unidecode error on Windows regardless of -vv or not with versions of tox >= 3.8.4

I suspect #1237, which was merged in 3.8.4 is related

[Holt59]

Any news on this? I still get the issue with tox 3.20.1 and Python 3.8.2.

Edit: Just tested with Python 3.8.6 and the issue is still there.

[sbraz]

I can confirm what @jtpavlock said: 566b6b6 broke it (I tested it and its parent commit to make sure). Changing verbosity level did nothing for me and I ended up pinning tox to v3.8.3.

@gaborbernat can you please look into reverting / fixing said commit? I think that at the moment, it is impossible to run the latest tox on Windows if tests contain a non-ASCII character (at least with Python 3).

[hf-krechan]

Thanks for this hint with the tox v3.8.3. I could solve my problem with: pip install tox==3.8.3
Before I used the latest version (v3.20.1) and got the following error message:

PS D:\Hochfrequenz\Workspaces\BO4E-python> tox -e dev
dev create: D:\Hochfrequenz\Workspaces\BO4E-python\.tox\dev
_____________________________________________________________________________ summary ___________
  dev: commands succeeded
  congratulations :)
Traceback (most recent call last):
  File "C:\Program Files\Python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "D:\Hochfrequenz\Workspaces\.toxbase\Scripts\tox.exe\__main__.py", line 7, in <module>
  File "d:\hochfrequenz\workspaces\.toxbase\lib\site-packages\tox\session\__init__.py", line 44,
    main(args)
  File "d:\hochfrequenz\workspaces\.toxbase\lib\site-packages\tox\session\__init__.py", line 69,
    exit_code = session.runcommand()
  File "d:\hochfrequenz\workspaces\.toxbase\lib\site-packages\tox\session\__init__.py", line 197,
    return self.subcommand_test()
  File "d:\hochfrequenz\workspaces\.toxbase\lib\site-packages\pluggy\hooks.py", line 286, in _25,_call__
  File "d:\hochfrequenz\workspaces\.toxbase\lib\site-packages\pluggy\hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "d:\hochfrequenz\workspaces\.toxbase\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "d:\hochfrequenz\workspaces\.toxbase\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "d:\hochfrequenz\workspaces\.toxbase\lib\site-packages\pluggy\callers.py", line 208, in _multicall
    return outcome.get_result()
  File "d:\hochfrequenz\workspaces\.toxbase\lib\site-packages\pluggy\callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "d:\hochfrequenz\workspaces\.toxbase\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "d:\hochfrequenz\workspaces\.toxbase\lib\site-packages\tox\venv.py", line 708, in tox_testenv_create
    venv._pcall(
  File "d:\hochfrequenz\workspaces\.toxbase\lib\site-packages\tox\venv.py", line 593, in _pcall
    return action.popen(
  File "d:\hochfrequenz\workspaces\.toxbase\lib\site-packages\tox\action.py", line 124, in popen
    lines = out_path.read_text("UTF-8").split("\n")
  File "d:\hochfrequenz\workspaces\.toxbase\lib\site-packages\py\_path\common.py", line 171, in read_text
    return f.read()
  File "C:\Program Files\Python38\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 533: invalid start byte

In my last name of my user account there is a German umlaut "ö" which belongs probably to the unicode byte 0xf6 in the error message.

[jonathancychow]

Confirm with what @jtpavlock and @sbraz said. Just tested with python 3.7.5 and tox 3.21.3 on Window, issue is still there. If I use tox 3.8.3 then it's fine.

[methane]

Environment variable PYTHONIOENCODING=utf-8 or PYTHONUTF8=1 will be a workaround.