TypeError in svn_utils.py:decode_as_string to crash

[ghost]

Originally reported by: guyroz (Bitbucket: guyroz, GitHub: Unknown)

---

Traceback (most recent call last):
  File "setup.py", line 204, in <module>
    dist = setuptools.setup(**setup_params)
  File "/Users/jenkins/python/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/Users/jenkins/python/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/Users/jenkins/python/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/private/var/folders/39/kgfk3gv137n9slxvv4_mdvcr0000gs/T/tmpLLUGaS/setuptools-1.3.2/setuptools/command/bdist_egg.py", line 177, in run
    self.run_command("egg_info")
  File "/Users/jenkins/python/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/Users/jenkins/python/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/private/var/folders/39/kgfk3gv137n9slxvv4_mdvcr0000gs/T/tmpLLUGaS/setuptools-1.3.2/setuptools/command/egg_info.py", line 187, in run
    self.find_sources()
  File "/private/var/folders/39/kgfk3gv137n9slxvv4_mdvcr0000gs/T/tmpLLUGaS/setuptools-1.3.2/setuptools/command/egg_info.py", line 230, in find_sources
    mm.run()
  File "/private/var/folders/39/kgfk3gv137n9slxvv4_mdvcr0000gs/T/tmpLLUGaS/setuptools-1.3.2/setuptools/command/egg_info.py", line 296, in run
    self.add_defaults()
  File "/private/var/folders/39/kgfk3gv137n9slxvv4_mdvcr0000gs/T/tmpLLUGaS/setuptools-1.3.2/setuptools/command/egg_info.py", line 335, in add_defaults
    rcfiles = list(walk_revctrl())
  File "/private/var/folders/39/kgfk3gv137n9slxvv4_mdvcr0000gs/T/tmpLLUGaS/setuptools-1.3.2/setuptools/command/sdist.py", line 18, in walk_revctrl
    for item in ep.load()(dirname):
  File "/private/var/folders/39/kgfk3gv137n9slxvv4_mdvcr0000gs/T/tmpLLUGaS/setuptools-1.3.2/setuptools/command/sdist.py", line 58, in _default_revctrl
    for item in finder(dirname):
  File "/private/var/folders/39/kgfk3gv137n9slxvv4_mdvcr0000gs/T/tmpLLUGaS/setuptools-1.3.2/setuptools/svn_utils.py", line 414, in svn_finder
    info = SvnInfo.load(dirname)
  File "/private/var/folders/39/kgfk3gv137n9slxvv4_mdvcr0000gs/T/tmpLLUGaS/setuptools-1.3.2/setuptools/svn_utils.py", line 221, in load
    code, data = _run_command(['svn', 'info', normdir])
  File "/private/var/folders/39/kgfk3gv137n9slxvv4_mdvcr0000gs/T/tmpLLUGaS/setuptools-1.3.2/setuptools/svn_utils.py", line 41, in _run_command
    data = decode_as_string(data, encoding)
  File "/private/var/folders/39/kgfk3gv137n9slxvv4_mdvcr0000gs/T/tmpLLUGaS/setuptools-1.3.2/setuptools/svn_utils.py", line 93, in decode_as_string
    text = text.decode(encoding)
TypeError: decode() argument 1 must be string, not None

locale.getpreferredencoding may return None (if LC_ALL= on Mac), causes svn_utils.py:decode_as_string to crash

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/105

[ghost]

Original comment by philip_thiem (Bitbucket: philip_thiem, GitHub: Unknown):

---

Thx Guy. I knew that there was a reason I didn't want to use getprefferedencoding. Will have to have a think on this one. Are you on a mac? I'm probably going to have to get my hands on an environment to do some experimenting on.

[ghost]

Original comment by guyroz (Bitbucket: guyroz, GitHub: Unknown):

---

Yes Philip I am using a Mac
Let me know if I can help

[ghost]

Original comment by philip_thiem (Bitbucket: philip_thiem, GitHub: Unknown):

---

Thx, I think I will definitely take you up on that. It looks like between the various versions of pythons supported there are a number of "fun" nuisances with mac. Sufficiently old versions of python appear to bus error on certain calls even. I have a few ideas, but I figure that instead of guessing I might as well see how these things will behave. So I have an encoding test which I tossed together. Should be able to run it in a console with py26+ So if you would be so kind, ;), I would appreciate the output.

[ghost]

Original comment by guyroz (Bitbucket: guyroz, GitHub: Unknown):

---

here are the results:

ci-macpro:~ jenkins$ python /tmp/encodingtestv2.py
darwin
STDOUT Encoding: US-ASCII
Has locale.CODESET? False
PreferredEncoding: None
DefaultLocale: None, None
Initializing Local?
PreferredEncoding: None
DefaultLocale: None, None

Other interesting results:

ci-macpro:~ jenkins$ python -c "import locale; print locale.getpreferredencoding()"
None

ci-macpro:~ jenkins$ LC_ALL=en_US python -c "import locale; print locale.getpreferredencoding()"
ISO8859-1

ci-macpro:~ jenkins$ LC_ALL=en_US python /tmp/encodingtestv2.py
darwin
Traceback (most recent call last):
  File "/tmp/encodingtestv2.py", line 6, in <module>
    print("STDOUT Encoding: " + sys.stdout.encoding)
TypeError: cannot concatenate 'str' and 'NoneType' objects
ci-macpro:~ jenkins$ 

[ghost]

Original comment by philip_thiem (Bitbucket: philip_thiem, GitHub: Unknown):

---

Thank you! I find it interesting that the stdout.encoding is None when LC_ALL is given. That and I used the wrong category in my script. Here is a newer version which you can try, please, with and without LC_ALL=en_US. I think I've figured out how to possibly get a mac VM with travis-ci.org it feels like a bit of a hack to tell it that I need an objective-c environment, but should have access to at least py27 and py3 interpreter.

[ghost]

Original comment by philip_thiem (Bitbucket: philip_thiem, GitHub: Unknown):

---

Ok, got the vm's working. With 2.7 and 3.3 I get

darwin
STDOUT Encoding: US-ASCII
Has locale.CODESET? True
nl_langinfo returned US-ASCII
PreferredEncoding: US-ASCII
DefaultLocale: None, None
Initializing Local LC_ALL?
nl_langinfo returned US-ASCII
PreferredEncoding: US-ASCII
DefaultLocale: None, None
/usr/local/bin/python

darwin
STDOUT Encoding: US-ASCII
Has locale.CODESET? True
nl_langinfo returned US-ASCII
PreferredEncoding: US-ASCII
DefaultLocale: None, None
Initializing Local LC_ALL?
nl_langinfo returned US-ASCII
PreferredEncoding: US-ASCII
DefaultLocale: None, None

Which I can't envision is right, however, US-ASCII should decode as UTF-8 and I'm pretty sure that osX pretty much enforces UTF-8 for the filesystem and tends to prefer that, so it might be a more reasonable default... Also seems like trying the nudge initialization of the locale with setlocale doesn't work thus far on any mac setup I've seen. ISO8859-1 is latin-1. LC_ALL=en_US.utf8 will probably get default encoding to return (en_us, utf-8)..

[ghost]

Original comment by philip_thiem (Bitbucket: philip_thiem, GitHub: Unknown):

---

https://bitbucket.org/pypa/setuptools/pull-request/26/svn-encoding-issues/diff
Basically for determining the terminal encoding:

1. Try getpreferredencoding
2. If that is US_ASCII or None try the encoding from getdefaultlocale() if that was a "fallback" because python couldn't figure it out from the environment or OS, this wll result in None.
3. If not none, then make sure python actually knows about the encoding.
4. On the event of an error or unknown codec, revert to fallbacks
5. That is, on darwin try utf-8 everything else ASCII. Also on the darwin and mac-roman, switch to utf-8 this was a bug on older pythons.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Update changelog. Fixes #105 and Fixes #113.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

1.4.1 includes these fixes. Please test and confirm.