Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix unicode encoding errors inside pip #567

Closed
wants to merge 1 commit into from

Conversation

suutari-ai
Copy link
Contributor

On Python 2, if a distribution package contains non-ASCII file names,
the Resolver failed with the following UnicodeDecodeError, because
RequirementSet was initialized with unicode paths rather than str.

Fix this by making sure that the path components used in PyPIRepository
are all filesystem encoded str.

Drop the unicode_literals import from _compat/tempfile.py, because the
strings used in there should also be str, not unicode.

Also make sure that the InstallRequirement is always constructed from
str, not unicode.

Traceback of the error looked like this:

Traceback (most recent call last):
  File ".../piptools/scripts/compile.py", line 184, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File ".../piptools/resolver.py", line 107, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File ".../piptools/resolver.py", line 195, in _resolve_one_round
    for dep in self._iter_dependencies(best_match):
  File ".../piptools/resolver.py", line 274, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File ".../piptools/repositories/pypi.py", line 145, in get_dependencies
    self._dependencies_cache[ireq] = reqset._prepare_file(self.finder, ireq)
  File ".../pip/req/req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File ".../pip/download.py", line 821, in unpack_url
    hashes=hashes
  File ".../pip/download.py", line 663, in unpack_http_url
    unpack_file(from_path, location, content_type, link)
  File ".../pip/utils/__init__.py", line 605, in unpack_file
    untar_file(filename, location)
  File ".../pip/utils/__init__.py", line 551, in untar_file
    path = os.path.join(location, fn)
  File "/usr/local/lib/python2.7/posixpath.py", line 73, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 66:
  ordinal not in range(128)

Fixes #564

On Python 2, if a distribution package contains non-ASCII file names,
the Resolver failed with the following UnicodeDecodeError, because
RequirementSet was initialized with unicode paths rather than str.

Fix this by making sure that the path components used in PyPIRepository
are all filesystem encoded str.

Drop the unicode_literals import from `_compat/tempfile.py`, because the
strings used in there should also be str, not unicode.

Also make sure that the InstallRequirement is always constructed from
str, not unicode.

Traceback of the error looked like this:

    Traceback (most recent call last):
      File ".../piptools/scripts/compile.py", line 184, in cli
        results = resolver.resolve(max_rounds=max_rounds)
      File ".../piptools/resolver.py", line 107, in resolve
        has_changed, best_matches = self._resolve_one_round()
      File ".../piptools/resolver.py", line 195, in _resolve_one_round
        for dep in self._iter_dependencies(best_match):
      File ".../piptools/resolver.py", line 274, in _iter_dependencies
        dependencies = self.repository.get_dependencies(ireq)
      File ".../piptools/repositories/pypi.py", line 145, in get_dependencies
        self._dependencies_cache[ireq] = reqset._prepare_file(self.finder, ireq)
      File ".../pip/req/req_set.py", line 620, in _prepare_file
        session=self.session, hashes=hashes)
      File ".../pip/download.py", line 821, in unpack_url
        hashes=hashes
      File ".../pip/download.py", line 663, in unpack_http_url
        unpack_file(from_path, location, content_type, link)
      File ".../pip/utils/__init__.py", line 605, in unpack_file
        untar_file(filename, location)
      File ".../pip/utils/__init__.py", line 551, in untar_file
        path = os.path.join(location, fn)
      File "/usr/local/lib/python2.7/posixpath.py", line 73, in join
        path += '/' + b
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 66:
      ordinal not in range(128)

Fixes jazzband#564
@vphilippon vphilippon self-requested a review September 26, 2017 15:42
Copy link
Member

@vphilippon vphilippon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add the bugfix in the changelog? That fix deserves a mention.

For the change, LGTM

@vphilippon
Copy link
Member

Checked out your branch, tested, it works.
This also fixes #433 (I can reproduce everytime using the --rebuild option).

@vphilippon
Copy link
Member

vphilippon commented Sep 26, 2017

I noticed your branch isn't based on master. I'll rebase your branch, add the changelog entry and submit a PR, that'll save you the trouble.

Thanks for the nice work!

vphilippon added a commit that referenced this pull request Sep 26, 2017
Fix unicode encoding errors inside pip (Copy of #567)
@vphilippon
Copy link
Member

"Merged" by #568
Thanks again @suutari-ai

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

UnicodeDecodeError 'ascii' codec can't decode byte 0xe3 in position 66: ordinal not in range(128)
3 participants