New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid UnicodeDecodeError from command output #2970

Merged

gaborbernat merged 2 commits into tox-dev:main from masenf:unicode-decode-error

Apr 5, 2023

Collaborator

masenf commented Apr 3, 2023

Instead of raising UnicodeDecodeError when command output includes non-utf-8 bytes,
tox will now use surrogateescape error handling to convert the unrecognized bytes
to escape sequences according to PEP-383.

Tox has no way of knowing that the bytestream emitted by a command will be valid utf-8, even if utf-8 is ostensibly the "correct" encoding for the stream or system. It's always possible for an arbitrary command to return non-utf-8 bytes, and this situation should not break tox.

Thanks for contribution

Please, make sure you address all the checklists (for details on how see
development documentation)!

ran the linter to address style issues (tox -e fix)
wrote descriptive pull request text
ensured there are test(s) validating the fix
added news fragment in docs/changelog folder
updated/extended the documentation

masenf added 2 commits

April 3, 2023 20:13


          test_sync_write_decode_surrogate: utf-8 decode

When SyncWrite decodes bytes as utf-8, it should replace unknown
sequences with the unicode surrogate codepoint instead of crashing the
program.

Test case for tox-dev#2969


          SyncWrite: decode with errors='surrogateescape'

c4fdf2c

Avoid bubbling UnicodeDecodeError up from stream handling internals.

Tox has no way of knowing that the bytestream emitted by a command will
be valid utf-8, even if utf-8 is ostensibly the "correct" encoding for
the stream. It's always possible for an arbitrary command to return
non-utf-8 bytes, and this situation should not break tox.

Fix tox-dev#2969

masenf requested a review from gaborbernat as a code owner

April 3, 2023 20:53

psf-chronographer bot added the bot:chronographer:provided label

Collaborator Author

masenf commented Apr 4, 2023

@gaborbernat @jugmac00 do either of you have an opinion on this change?

gaborbernat approved these changes

View reviewed changes

gaborbernat closed this

gaborbernat reopened this

gaborbernat merged commit 8d8eba6 into tox-dev:main

descope bot referenced this pull request in descope/django-descope


          chore(deps): update dependency tox to v4.5.1 (#156)

e68bc2d

This PR contains the following updates:

| Package | Type | Update | Change | Pending |
|---|---|---|---|---|
| [tox](https://togithub.com/tox-dev/tox)
([changelog](https://tox.wiki/en/latest/changelog.html)) | dev | minor |
`4.4.8` -> `4.5.1` | `4.6.0` (+1) |

---

### Release Notes

<details>
<summary>tox-dev/tox</summary>

### [`v4.5.1`](https://togithub.com/tox-dev/tox/releases/tag/4.5.1):
Test Trusted Publisher

[Compare Source](https://togithub.com/tox-dev/tox/compare/4.5.0...4.5.1)

### [`v4.5.0`](https://togithub.com/tox-dev/tox/releases/tag/4.5.0)

[Compare
Source](https://togithub.com/tox-dev/tox/compare/4.4.12...4.5.0)

#### What's Changed

- Bump deps and tools by
[@&#8203;gaborbernat](https://togithub.com/gaborbernat) in
[https://github.com/tox-dev/tox/pull/2987](https://togithub.com/tox-dev/tox/pull/2987)
- git: Ignore the .lock file for demo_pkg_inline by
[@&#8203;hroncok](https://togithub.com/hroncok) in
[https://github.com/tox-dev/tox/pull/2988](https://togithub.com/tox-dev/tox/pull/2988)
- Add FAQ entry on how to test against EOL Python versions by
[@&#8203;jugmac00](https://togithub.com/jugmac00) in
[https://github.com/tox-dev/tox/pull/2991](https://togithub.com/tox-dev/tox/pull/2991)
- Feature: suppress step timings for verbosity=1
[#&#8203;2891](https://togithub.com/tox-dev/tox/issues/2891) by
[@&#8203;nedbat](https://togithub.com/nedbat) in
[https://github.com/tox-dev/tox/pull/2992](https://togithub.com/tox-dev/tox/pull/2992)

**Full Changelog**:
tox-dev/tox@4.4.12...4.5.0

### [`v4.4.12`](https://togithub.com/tox-dev/tox/releases/tag/4.4.12)

[Compare
Source](https://togithub.com/tox-dev/tox/compare/4.4.11...4.4.12)

#### What's Changed

- Avoid race conditions in tests using the demo_pkg_inline fixture by
[@&#8203;hroncok](https://togithub.com/hroncok) in
[https://github.com/tox-dev/tox/pull/2986](https://togithub.com/tox-dev/tox/pull/2986)
- Bump deps and tools by
[@&#8203;gaborbernat](https://togithub.com/gaborbernat) in
[https://github.com/tox-dev/tox/pull/2981](https://togithub.com/tox-dev/tox/pull/2981)

**Full Changelog**:
tox-dev/tox@4.4.11...4.4.12

### [`v4.4.11`](https://togithub.com/tox-dev/tox/releases/tag/4.4.11)

[Compare
Source](https://togithub.com/tox-dev/tox/compare/4.4.10...4.4.11)

#### What's Changed

- Allow plugins to set `tox_root` by
[@&#8203;kdestin](https://togithub.com/kdestin) in
[https://github.com/tox-dev/tox/pull/2978](https://togithub.com/tox-dev/tox/pull/2978)

#### New Contributors

- [@&#8203;kdestin](https://togithub.com/kdestin) made their first
contribution in
[https://github.com/tox-dev/tox/pull/2978](https://togithub.com/tox-dev/tox/pull/2978)

**Full Changelog**:
tox-dev/tox@4.4.10...4.4.11

### [`v4.4.10`](https://togithub.com/tox-dev/tox/releases/tag/4.4.10)

[Compare
Source](https://togithub.com/tox-dev/tox/compare/4.4.9...4.4.10)

#### What's Changed

- Bump deps and tools by
[@&#8203;gaborbernat](https://togithub.com/gaborbernat) in
[https://github.com/tox-dev/tox/pull/2976](https://togithub.com/tox-dev/tox/pull/2976)
- Fix `tox_root` propagation to `work_dir` by
[@&#8203;Tbruno25](https://togithub.com/Tbruno25) in
[https://github.com/tox-dev/tox/pull/2962](https://togithub.com/tox-dev/tox/pull/2962)

#### New Contributors

- [@&#8203;Tbruno25](https://togithub.com/Tbruno25) made their first
contribution in
[https://github.com/tox-dev/tox/pull/2962](https://togithub.com/tox-dev/tox/pull/2962)

**Full Changelog**:
tox-dev/tox@4.4.9...4.4.10

### [`v4.4.9`](https://togithub.com/tox-dev/tox/releases/tag/4.4.9)

[Compare Source](https://togithub.com/tox-dev/tox/compare/4.4.8...4.4.9)

#### What's Changed

- Added python 3.11 by
[@&#8203;ElBe-Plaq](https://togithub.com/ElBe-Plaq) in
[https://github.com/tox-dev/tox/pull/2964](https://togithub.com/tox-dev/tox/pull/2964)
- Document running tox within a Docker container by
[@&#8203;31z4](https://togithub.com/31z4) in
[https://github.com/tox-dev/tox/pull/2923](https://togithub.com/tox-dev/tox/pull/2923)
- Correct Docker image working dir by
[@&#8203;31z4](https://togithub.com/31z4) in
[https://github.com/tox-dev/tox/pull/2965](https://togithub.com/tox-dev/tox/pull/2965)
- Avoid UnicodeDecodeError from command output by
[@&#8203;masenf](https://togithub.com/masenf) in
[https://github.com/tox-dev/tox/pull/2970](https://togithub.com/tox-dev/tox/pull/2970)
- Bump pypa/gh-action-pypi-publish from 1.8.3 to 1.8.5 by
[@&#8203;dependabot](https://togithub.com/dependabot) in
[https://github.com/tox-dev/tox/pull/2971](https://togithub.com/tox-dev/tox/pull/2971)

#### New Contributors

- [@&#8203;ElBe-Plaq](https://togithub.com/ElBe-Plaq) made their first
contribution in
[https://github.com/tox-dev/tox/pull/2964](https://togithub.com/tox-dev/tox/pull/2964)
- [@&#8203;31z4](https://togithub.com/31z4) made their first
contribution in
[https://github.com/tox-dev/tox/pull/2923](https://togithub.com/tox-dev/tox/pull/2923)

**Full Changelog**: tox-dev/tox@4.4.8...4.4.9

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR is behind base branch, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNS4xMC4yIiwidXBkYXRlZEluVmVyIjoiMzUuMTAuMiJ9-->

Co-authored-by: descope[bot] <descope[bot]@users.noreply.github.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided