Skip to content

gh-94823: Improve coverage in tokenizer.c:valid_utf8 #94856

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Aug 16, 2022
Merged

gh-94823: Improve coverage in tokenizer.c:valid_utf8 #94856

merged 4 commits into from
Aug 16, 2022

Conversation

mdboom
Copy link
Contributor

@mdboom mdboom commented Jul 14, 2022 •
edited by miss-islington
Loading

When loading a source file from disk, there is a separate UTF-8 validator
distinct from the one in unicode_decode_utf8. This exercises that code path
with the same set of invalid inputs as we use for testing the "other" UTF-8
decoder.

Automerge-Triggered-By: GH:ericsnowcurrently

@bedevere-bot bedevere-bot added tests Tests in the Lib/test dir awaiting review labels Jul 14, 2022
@brandtbucher brandtbucher added needs backport to 3.11 only security fixes labels Jul 15, 2022
@mdboom
pythongh-94823: Improve coverage in tokenizer.c:valid_utf8
e52f328 
When loading a source file from disk, there is a separate UTF-8 validator
distinct from the one in `unicode_decode_utf8`. This exercises that code path
with the same set of invalid inputs as we use for testing the "other" UTF-8
decoder.
@mdboom mdboom force-pushed the tokenizer-valid-utf8-coverage branch from 6db7d84 to e52f328 Compare July 15, 2022 18:11
ericsnowcurrently
ericsnowcurrently requested changes Aug 12, 2022
View reviewed changes
Copy link
Member

@ericsnowcurrently ericsnowcurrently left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this! It mostly looks good. I noticed a couple of small typos.

@bedevere-bot
Copy link

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@mdboom
Copy link
Contributor Author

mdboom commented Aug 16, 2022

I have made the requested changes; please review again

@bedevere-bot
Copy link

Thanks for making the requested changes!

@ericsnowcurrently: please review the changes made to this pull request.

ericsnowcurrently
ericsnowcurrently approved these changes Aug 16, 2022
View reviewed changes
Copy link
Member

@ericsnowcurrently ericsnowcurrently left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@miss-islington miss-islington merged commit f215d7c into python:main Aug 16, 2022
@miss-islington
Copy link
Contributor

Thanks @mdboom for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 16, 2022
@mdboom @miss-islington
pythongh-94823: Improve coverage in tokenizer.c:valid_utf8 (pythonGH-…
0425b23 
…94856)

When loading a source file from disk, there is a separate UTF-8 validator
distinct from the one in `unicode_decode_utf8`. This exercises that code path
with the same set of invalid inputs as we use for testing the "other" UTF-8
decoder.
(cherry picked from commit f215d7c)

Co-authored-by: Michael Droettboom <mdboom@gmail.com>
@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label Aug 16, 2022
@bedevere-bot
Copy link

GH-96029 is a backport of this pull request to the 3.11 branch.

@ericsnowcurrently
Copy link
Member

Thanks for the test!

pablogsal pushed a commit that referenced this pull request Aug 16, 2022
@miss-islington @mdboom
[3.11] gh-94823: Improve coverage in tokenizer.c:valid_utf8 (GH-94856) (
2bb363c 
#96029)

Co-authored-by: Michael Droettboom <mdboom@gmail.com>
This was referenced Aug 22, 2022
Closed
Closed
Merged
@kumaraditya303 kumaraditya303 mentioned this pull request Sep 17, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ericsnowcurrently ericsnowcurrently ericsnowcurrently approved these changes

Assignees
No one assigned
Labels
skip news tests Tests in the Lib/test dir
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@mdboom @bedevere-bot @miss-islington @ericsnowcurrently @brandtbucher @kumaraditya303