Add check if `str(handle)` correctly converted the object, and throw py::error_already_set if not #2392

YannickJadoul · 2020-08-14T00:25:18Z

Only problematic on Python 2, I think. Discovered here: #2380 (comment)

YannickJadoul · 2020-08-14T00:30:10Z

Also note that by checking always checking the result after calling raw_ptr, we can more the if (!str_value) throw error_already_set(); inside the Python 2-only preprocessor block, and get rid of one full pointer comparison (!) on Python 3, when calling the constructor from py::object! ;-)

YannickJadoul · 2020-08-14T00:36:01Z

Of course. I tried this locally based on #2380, but detail::PyUnicode_Check_Permissive first still needs to die, such that py::str cannot contain a bytes-object anymore!

YannickJadoul · 2020-08-14T00:40:07Z

Rebased onto #2380, but waiting for #2380 to get merged before this can get rebased and merged.

tests/test_pytypes.py

bstaletic

Looks good, apart from that sys.version.major thing.

henryiii · 2020-08-16T17:44:12Z

@YannickJadoul if you used pre-commit install this couldn't happen. 😜

henryiii · 2020-08-16T17:46:39Z

This might benefit from being rebased after #2397

tests/test_pytypes.py

include/pybind11/cast.h

YannickJadoul · 2020-08-16T21:06:28Z

@YannickJadoul if you used pre-commit install this couldn't happen. stuck_out_tongue_winking_eye

OK, you win this one ;-)

…py::error_already_set if not

YannickJadoul · 2020-08-16T21:35:40Z

Fixed this. It's quite a minor PR, so a few more quick checks (@bstaletic & @henryiii, the things you mentioned should be fixed).
@rwgk, maybe you should give this a quick final check (since it involves py::str), and merge into str-bytes-cleanup at will if you're happy with it?

wjakob · 2020-08-17T08:25:23Z

LGTM.

rwgk

Looks great, thanks Yannick!
I'll hold off merging only because I'm still unclear why the current str-bytes-cleanup branch fails to pass some tests in the Google environment (I didn't get a chance to work on it at all yesterday).
(For completeness: the failing tests are NOT pybind11 unit tests.)

rwgk · 2020-08-17T17:53:05Z

Looks great, thanks Yannick!
I'll hold off merging only because I'm still unclear why the current str-bytes-cleanup branch fails to pass some tests in the Google environment (I didn't get a chance to work on it at all yesterday).
(For completeness: the failing tests are NOT pybind11 unit tests.)

For future reference: the test failure was due to accidental str vs bytes mixup, masked by the double-nature of pybind11::str.

… py::error_already_set if not. Similar to pybind#2392, but does not depend on pybind#2409. Splitting out this PR from pybind#2409 to make that PR as simple as possible. Net effects of this PR: * Adds missing test coverage. * Changes TypeError to UnicodeDecodeError for Python 2. This PR has two commits. Please do not squash, to make the behavior change obvious in the commit history.

@rwgk

…py::error_already_set if not (bis) (#2477) * Add check if `str(handle)` correctly converted the object, and throw py::error_already_set if not * Fix tests on Python 3 * Apply @rwgk's fixes to cherry-picked commits from #2392

This was referenced Aug 14, 2020

Changing pybind11::str to only hold PyUnicodeObject (NOT also bytes). #2380

Merged

Tracking PR: intermediate state of completed str/bytes cleaning up PRs #2348

Closed

YannickJadoul force-pushed the fix-python2-str-malformed_utf8 branch from 235168a to d58307d Compare August 14, 2020 00:39

YannickJadoul commented Aug 14, 2020

View reviewed changes

tests/test_pytypes.py Outdated Show resolved Hide resolved

rwgk force-pushed the str-bytes-cleanup branch 2 times, most recently from f38fafc to 6cd4552 Compare August 14, 2020 22:03

YannickJadoul force-pushed the fix-python2-str-malformed_utf8 branch 3 times, most recently from 7eee06e to 366f713 Compare August 15, 2020 21:51

YannickJadoul self-assigned this Aug 15, 2020

bstaletic reviewed Aug 16, 2020

View reviewed changes

rwgk force-pushed the str-bytes-cleanup branch from 1d553e4 to 0b49467 Compare August 16, 2020 17:24

YannickJadoul force-pushed the fix-python2-str-malformed_utf8 branch from 366f713 to 6452317 Compare August 16, 2020 17:41

YannickJadoul force-pushed the fix-python2-str-malformed_utf8 branch from 6452317 to b25eace Compare August 16, 2020 17:47

henryiii reviewed Aug 16, 2020

View reviewed changes

tests/test_pytypes.py Outdated Show resolved Hide resolved

henryiii reviewed Aug 16, 2020

View reviewed changes

include/pybind11/cast.h Show resolved Hide resolved

YannickJadoul force-pushed the fix-python2-str-malformed_utf8 branch from b25eace to f7d376c Compare August 16, 2020 21:13

YannickJadoul force-pushed the str-bytes-cleanup branch from 0b49467 to 0ab4b97 Compare August 16, 2020 21:14

Add check if str(handle) correctly converted the object, and throw …

9e900fa

…py::error_already_set if not

YannickJadoul force-pushed the fix-python2-str-malformed_utf8 branch from f7d376c to 8e53b83 Compare August 16, 2020 21:29

Fix tests on Python 3

3dbafa6

YannickJadoul force-pushed the fix-python2-str-malformed_utf8 branch from 8e53b83 to 3dbafa6 Compare August 16, 2020 21:30

bstaletic approved these changes Aug 17, 2020

View reviewed changes

rwgk approved these changes Aug 17, 2020

View reviewed changes

rwgk merged commit 7ae6bb2 into pybind:str-bytes-cleanup Aug 17, 2020

YannickJadoul deleted the fix-python2-str-malformed_utf8 branch August 17, 2020 22:08

henryiii added the bug label Aug 19, 2020

rwgk mentioned this pull request Sep 8, 2020

Adds check if str(handle) correctly converted the object, and throw py::error_already_set if not. #2473

Closed

rwgk added a commit to rwgk/pybind11 that referenced this pull request Sep 10, 2020

Backing out changes originating from pybind#2392.

6f270a3

YannickJadoul added a commit to YannickJadoul/pybind11 that referenced this pull request Sep 10, 2020

Apply @rwgk's fixes to cherry-picked commits from pybind#2392

76f3ebe

YannickJadoul mentioned this pull request Sep 10, 2020

Add check if str(handle) correctly converted the object, and throw py::error_already_set if not (bis) #2477

Merged

YannickJadoul added a commit to YannickJadoul/pybind11 that referenced this pull request Sep 11, 2020

Apply @rwgk's fixes to cherry-picked commits from pybind#2392

055f86f

rwgk added a commit to rwgk/pybind11 that referenced this pull request Sep 16, 2020

Backing out changes originating from pybind#2392.

bd1d77b

rwgk added a commit to rwgk/pybind11 that referenced this pull request Sep 16, 2020

Backing out changes originating from pybind#2392.

21376dd

rwgk added a commit to rwgk/pybind11 that referenced this pull request Sep 27, 2020

Backing out changes originating from pybind#2392.

1bdb07d

rwgk added a commit to rwgk/pybind11 that referenced this pull request Oct 13, 2020

Backing out changes originating from pybind#2392.

1152249

rwgk added a commit to rwgk/pybind11 that referenced this pull request Oct 18, 2020

Backing out changes originating from pybind#2392.

924502a

rwgk added a commit to rwgk/pybind11 that referenced this pull request Oct 29, 2020

Backing out changes originating from pybind#2392.

df6e916

rwgk mentioned this pull request Feb 10, 2023

FWD pybind11 google/pybind11clif#2392

Closed

Add check if str(handle) correctly converted the object, and throw py::error_already_set if not #2392

Add check if str(handle) correctly converted the object, and throw py::error_already_set if not #2392

Uh oh!

Conversation

YannickJadoul commented Aug 14, 2020

Uh oh!

YannickJadoul commented Aug 14, 2020

Uh oh!

YannickJadoul commented Aug 14, 2020

Uh oh!

YannickJadoul commented Aug 14, 2020

Uh oh!

Uh oh!

bstaletic left a comment

Choose a reason for hiding this comment

Uh oh!

henryiii commented Aug 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

henryiii commented Aug 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

YannickJadoul commented Aug 16, 2020

Uh oh!

YannickJadoul commented Aug 16, 2020

Uh oh!

wjakob commented Aug 17, 2020

Uh oh!

rwgk left a comment

Choose a reason for hiding this comment

Uh oh!

rwgk commented Aug 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Add check if `str(handle)` correctly converted the object, and throw py::error_already_set if not #2392

Add check if `str(handle)` correctly converted the object, and throw py::error_already_set if not #2392

henryiii commented Aug 16, 2020 •

edited

Loading

henryiii commented Aug 16, 2020 •

edited

Loading