Conan 0.21.2 install on Windows return error

[KindDragon]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc0 in position 10: ordinal not in range(128) conan returned -1

conan install -f=D:/Work/project/conanfile.txt -g cmake_multi -s build_type=Release -s os=Windows -s compiler=Visual Studio -s compiler.version=14 -s arch=x86_64 -s
 compiler.runtime=MD --build=missing
Requirements
    gtest/1.7.0@lasote/stable from conan.io
    ...
Packages
    gtest/1.7.0@lasote/stable:c85f9b402dd4d46acdf074e1c63b768a41181d7a
    ...

gtest/1.7.0@lasote/stable: Already installed!
...
PROJECT: Generated cmake created conanbuildinfo.cmake
PROJECT: Generated cmake_multi created conanbuildinfo_multi.cmake
PROJECT: Generated cmake_multi created conanbuildinfo_release.cmake
PROJECT: Generated cmake_multi created conanbuildinfo_release.cmake
PROJECT: Generated conaninfo.txt
Traceback (most recent call last):
  File "<string>", line 10, in <module>
  File "<string>", line 6, in run
  File "conan\conans\client\command.py", line 990, in main
  File "conan\conans\client\command.py", line 805, in run
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc0 in position 10: ordinal not in range(128)
conan returned -1

Conan install failed

Conan installed using Windows installer

[KindDragon]

Yes, bug appeared between 0.19.3 and 0.20.0

[lasote]

This issue is usually related with an error message thrown by the operating system containing some character with an unexpected encoding. What's your windows local language? It's failing only with that conanfile or always? We have to investigate it.

[KindDragon]

My windows language is Russian. Issue somehow appears only when our grpc package in conanfile.txt, maybe because we changing env_info.path for this package

    def package_info(self):
        self.env_info.path.append(os.path.join(self.package_folder, "bin"))

[memsharded]

I think it could be failing somewhere in the imports of conanfile.txt. Does it fail if you remove the [imports] section? the run_imports method is not capturing python Exceptions and raising a ConanException.

Not sure how to reproduce it, I have tried to generate non-ascii chars, but no success.
Probably, if you could run from sources, or just go to your Python installation -> Lib/site-packages/conans/client/command.py and uncomment lines 798, 799:

except Exception as exc:
            import traceback
            print(traceback.format_exc())
            msg = exception_message_safe(exc)

and try again, that could help. Thanks!

[KindDragon]

Does it fail if you remove the [imports] section?

Yes

I will try later other things

[KindDragon]

Everything works fine with conan.py from github, even when I checkout tag `0.21.2'

[KindDragon]

I can't try conan from pip because of #298

[KindDragon]

I reinstall my Python 3.6 (64bit) to root directory and install conan from it. Conan works fine from pip

[KindDragon]

I tried to install Conan installer to root directory and this doesn't help. Same error:

PROJECT: Generated conaninfo.txt
Traceback (most recent call last):
  File "conan\conans\conan.py", line 10, in <module>
  File "conan\conans\conan.py", line 6, in run
  File "conan\conans\client\command.py", line 1047, in main
  File "conan\conans\client\command.py", line 862, in run
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 958: ordinal not in range(128)
Failed to execute script conan

[KindDragon]

This error only with Python 2(2.7)!

PROJECT: Generated conaninfo.txt
Traceback (most recent call last):
  File "c:\Python27\lib\site-packages\conans\client\command.py", line 840, in run
    method(args[0][1:])
  File "c:\Python27\lib\site-packages\conans\client\command.py", line 310, in install
    no_imports=args.no_imports)
  File "c:\Python27\lib\site-packages\conans\client\manager.py", line 303, in install
    run_imports(conanfile, current_path, output)
  File "c:\Python27\lib\site-packages\conans\client\importer.py", line 54, in run_imports
    with environment_append(conanfile.env):
  File "c:\Python27\lib\contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "c:\Python27\lib\site-packages\conans\tools.py", line 116, in environment_append
    env_vars[name] += os.pathsep + old_env[name]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 979: ordinal not in range(128)

Traceback (most recent call last):
  File "c:\Python27\Scripts\conan-script.py", line 9, in <module>
    load_entry_point('conan==0.22.1', 'console_scripts', 'conan')()
  File "c:\Python27\lib\site-packages\conans\conan.py", line 6, in run
    main(sys.argv[1:])
  File "c:\Python27\lib\site-packages\conans\client\command.py", line 1047, in main
    error = command.run(args)
  File "c:\Python27\lib\site-packages\conans\client\command.py", line 862, in run
    raise exc
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 979: ordinal not in range(128)

[memsharded]

I can't try conan from pip because of #298

Pip is the most recommended way to install conan. If you cannot do it because your main Python installation is in a path with spaces, I would recommend using virtualenvs, they are easy, convenient and clean solution, please check http://docs.conan.io/en/latest/faq/troubleshooting.html#error-failed-to-create-process

[memsharded]

Thanks very much for your detailed reports. If it doesn't happen with the pip package, it seems some issue with the package generated by pyinstaller, and there is little we can do.
If your last report, the one failing with 2.7 in the imports run_imports(conanfile, current_path, output) call, still fails with the pip package, please confirm, so we could try to fix the origin which is the string concatenation between os.pathsep and old_env[name] which most likely the variable is a path containing characters that cannot be mapped to os.pathsep ascii encoding.

[KindDragon]

Yes, it still fails with pip in Python 2.7

[memsharded]

I have been having a look at this for a while, still no idea how to fix, because it is very convoluted with the environment variables encoding.

Quick question: is it possible that you are adding some environment variables to self.env_info in some package in the dependencies, which value is force to unicode? Something like:

self.env_info.PATH = [u"some/unicode/path"]

or maybe a package recipe declaring encoding in the first line like # -*- coding: utf-8 -*- or using from future import unicode_literals?

[KindDragon]

I find out that in Python2 env_vars (from conanfile.env) is a list of Unicode strings, but os.environ is a list of ordinary strings.

This fix issue for me:

def environment_append(env_vars): # tools.py
    ...
        if isinstance(value, list): # line 112
            env_vars[name] = str(os.pathsep.join(value))

I think bug appear after commit 5d46157 /cc @lasote

[memsharded]

I am also working on this, but so far, in my tests, if I provide env_vars as Unicode strings, str(os.pathsep.join(value)) also raises (Python 2)

UnicodeEncodeError: 'ascii' codec can't encode character u'\xc4' in position 1:

Also, if I try to convert str to unicode, then the line os.environ.update(env_vars) which is the one actually updating the environment, will fail with:

File "C:\Users\memsharded\conanws\conan\conans\tools.py", line 122, in environment_append
    os.environ.update(env_vars)
  File "C:\Users\memsharded\Envs\conan\lib\os.py", line 461, in update
    self[k] = dict[k]
  File "C:\Users\memsharded\Envs\conan\lib\os.py", line 422, in __setitem__
    putenv(key, item)
UnicodeEncodeError: 'ascii' codec can't encode character ...

Because, as you said, os.environ is ordinary strings, so it tries to convert them from unicode to ascii. If we do it explicitely, non-ascii characters will be lost, and if the environment variables are Paths that contain Russian characters, it will not work. Maybe python2 support is insufficient for handling environament variables referencing filesystems with non-ascii characters, and python3 should be the default.

[KindDragon]

I think probably because my default encoding CP1251, Python2 can convert Unicode string like "привет" to ordinary string for me.

[KindDragon]

Can you just use my quick fix?

it seems some issue with the package generated by pyinstaller, and there is little we can do.

Is it possible to use pyinstaller with Python3?

[lasote]

@KindDragon We could put the 'str' but first we need to understand why your values in conanfile.env are Unicode strings. In python 2 by default the strings are not Unicode so somewhere a Unicode conversion is being done. You are converting to unicode in your conanfile when you add to self.env_info?

[KindDragon]

but first we need to understand why your values in conanfile.env are Unicode strings

It's from our grpc package:
{'path': [u'D:/.conan\hhkgec\1\bin']}

Our grpc package contains this line:

    def package_info(self):
        self.env_info.path.append(os.path.join(self.package_folder, "bin"))

[lasote]

Thanks for the info,
It looks like an issue with the "short paths" feature, we will take a look. I'm assigning it to the next release.

[memsharded]

Can you just use my quick fix?

Sorry about the delay. The problem with your quick fix is that it might be forcing a conversion to str in an earlier point than previously, and that could raise encoding errors for other users with other default encoding, that wouldn't happen for them otherwise. This need some caution. We'll try to announce a release candidate in due time, so users can verify everything is fine.

[memsharded]

Now there are pyinstaller created installer for python3 too, which might solve this issue. Recent work from @mpdelbuono shows that it is nearly impossible to have reliable unicode pathnames with python 2.7, so indeed the recommended way is to use the python3 based conan.

I am closing this issue, but if the problem still persists in python3, please re-open or comment. Thanks!