gh-126937: ctypes: fix TypeError when a field's size is >65535 bytes #126938

Melissa0x1f992 · 2024-11-17T21:30:29Z

Used the bit_size_obj as seems to be intended, based on its usage in Lib/ctypes/_layout.py

Tested the change locally informally. Didn't write any tests.

Issue: Struct fields with size > 65535 bytes throw TypeError #126937

Fix rejection when non-bitfield fields are larger than 2^16 bytes

cpython-cla-bot · 2024-11-17T21:30:32Z

All commit authors signed the Contributor License Agreement.

bedevere-app · 2024-11-17T21:30:33Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

Melissa0x1f992 · 2024-11-17T21:38:43Z

Need some help understanding the lint error on this

Modules/_ctypes/cfield.c

terryjreedy · 2024-11-17T23:31:26Z

By somewhat random clicking of the search box and the test stage line, I got the full log. There were 6 spaces after the final \n, thus trailing spaces failed. (Note that running patchcheck as suggested somewhere would have fixed this.)

I asked to rerun just the one test, but the edit triggered all.

terryjreedy · 2024-11-17T23:36:13Z

I can't review the actual change. It might help someone else if you could connect the change to the change in 3.14 that caused the regression. To find the latter, I would first run git blame on cfield.c

picnixz · 2024-11-18T07:34:11Z

cc @encukou

Use bitsize param

Melissa0x1f992 · 2024-11-18T13:53:30Z

Added a test. It doesn't have any assertions bc the failure case is an error getting raised. Not sure if this is the appropriate way to add such a test.

Reduce test field size to (2^31 - 1) to support 32 bit systems

.pre-commit-config.yaml

Melissa0x1f992 · 2024-11-18T15:43:23Z

For the Win32, I don't have that setup. Could I get some help determining what field size should pass the < (1ULL << (8*sizeof(Py_ssize_t)-1)) / 8) test on Win32? Am I off by one? I don't want to commit (2^31 - 1 - 1) unless there's a good reason to consider that one byte smaller the limit for Win32.

Melissa0x1f992 · 2024-11-18T15:44:49Z

Is there a way for the test to set up different field sizes based on the current platform, and is that desirable? I'd like to test the actual max size, regardless of platform, rather than the smallest of the different platform maxes.

ZeroIntensity

Some initial comments :)

ZeroIntensity · 2024-11-18T15:58:52Z

Misc/NEWS.d/next/Core_and_Builtins/2024-11-17-21-35-55.gh-issue-126937.qluVM0.rst

@@ -0,0 +1 @@
+Fixed TypeError when a Structure's field's size is >65535 bytes


Suggested change

Fixed TypeError when a Structure's field's size is >65535 bytes

Fix :exc:`TypeError` when a :class:`ctypes.Structure` has a field size that doesn't fit into an unsigned 16-bit integer.

ZeroIntensity · 2024-11-18T15:59:14Z

Modules/_ctypes/cfield.c

-    Py_ssize_t bit_size = NUM_BITS(size);
-    if (bit_size) {
+    if (bit_size_obj != Py_None) {
+


No need for a newline

Suggested change

ZeroIntensity · 2024-11-18T16:00:23Z

Modules/_ctypes/cfield.c

+        Py_ssize_t bit_size;
+
+        if (PyLong_Check(bit_size_obj)) {
+            bit_size = PyLong_AsSsize_t(bit_size_obj);


PyLong_AsSsize_t can fail, you need to check for < 0 and then goto error.

ZeroIntensity · 2024-11-18T16:01:16Z

Modules/_ctypes/cfield.c

-    if (bit_size) {
+    if (bit_size_obj != Py_None) {
+
+        Py_ssize_t bit_size;


Lint is complaining about this. I'm assuming this just needs to get moved to before the if

hugovk · 2024-11-18T16:48:03Z

.pre-commit-config.yaml

@@ -29,12 +29,12 @@ repos:
      - id: black
        name: Run Black on Tools/build/check_warnings.py
        files: ^Tools/build/check_warnings.py
-        language_version: python3.12
+        language_version: python3


Let's revert these here, and deal with them in #126971:

Suggested change

language_version: python3

language_version: python3.12

hugovk · 2024-11-18T16:48:12Z

.pre-commit-config.yaml

        args: [--line-length=79]
      - id: black
        name: Run Black on Tools/jit/
        files: ^Tools/jit/
-        language_version: python3.12
+        language_version: python3


Suggested change

language_version: python3

language_version: python3.12

encukou · 2024-11-20T12:15:06Z

Thank you! That's the correct fix; there's a few details to iron out.
Here's how I'd write the start of the if (adding an explanation):

    if (bit_size_obj != Py_None) {
#ifdef Py_DEBUG
        Py_ssize_t bit_size = NUM_BITS(size);
        assert(bit_size > 0);
        assert(bit_size <= info->size * 8);
        // Currently, the bit size is specified redundantly
        // in NUM_BITS(size) and bit_size_obj.
        // Verify that they match.
        assert(PyLong_AsSsize_t(bit_size_obj) == bit_size);
#endif
        switch(info->ffi_type_pointer.type) {
            ...

(In Py_DEBUG builds, assert does nothing so a linter complains about unused bit_size. That #ifdef should fix this.)
(Putting each piece of data in one proper place is better left to another PR; I've started working on that.)

Is there a way for the test to set up different field sizes based on the current platform, and is that desirable?

Yes. sizeof(Py_ssize_t) in C is ctypes.sizeof(ctypes.c_size_t) in Python.

Added a test. It doesn't have any assertions bc the failure case is an error getting raised. Not sure if this is the appropriate way to add such a test.

It is; ideally with a comment like # this should succeed:. You could also add a with self.assertRaises block for the failing case.

@Melissa0x1f992, if at any time working on this PR stops being fun for you, let me know and I can finish it. I don't know if you're driven by curiosity or just want the bug fixed :)
Your perspective is certainly helpful.

encukou · 2024-11-29T12:37:28Z

Do you still want to work on this PR?
If not I can work on it next week.

Melissa0x1f992 · 2024-12-03T15:10:48Z

encukou, thank you! Yeah, life's gotten busy, so I won't be around to see this through.

It was a fun experience, and I hope to contribute on other things in the future 🙂

Co-Authorsd-By: Peter Bierma <zintensitydev@gmail.com>

encukou · 2024-12-04T10:37:40Z

OK. Thank you for the fix!
I hope you don't mind me pushing to your branch -- this way I can keep changes in this PR, with the discussion.

Melissa0x1f992 and others added 2 commits November 17, 2024 13:21

Fix rejection when non-bitfield fields are larger than 2^16 bytes

1962567

Merge pull request #1 from Melissa0x1f992/use-bitsize-param

1900cf5

Fix rejection when non-bitfield fields are larger than 2^16 bytes

bedevere-app bot added the awaiting review label Nov 17, 2024

bedevere-app bot mentioned this pull request Nov 17, 2024

Struct fields with size > 65535 bytes throw TypeError #126937

Open

📜🤖 Added by blurb_it.

2bd972b

terryjreedy reviewed Nov 17, 2024

View reviewed changes

Modules/_ctypes/cfield.c Outdated Show resolved Hide resolved

Update Modules/_ctypes/cfield.c

d2954cd

Melissa0x1f992 and others added 3 commits November 18, 2024 07:43

Remove specific dependency on Python 3.12 for black

0d21229

Add test case for large fields

d8a6f08

Merge pull request #2 from Melissa0x1f992/use-bitsize-param

16bfb44

Use bitsize param

Melissa0x1f992 requested review from hugovk and AlexWaygood as code owners November 18, 2024 13:51

Melissa0x1f992 and others added 4 commits November 18, 2024 08:19

Reduce test field size to (2^31 - 1) to support 32 bit systems

7402a8f

Merge branch 'python:main' into main

8994ad0

Reduce test field size to (2^31 - 1) to support 32 bit systems

db171e0

Merge pull request #3 from Melissa0x1f992/use-bitsize-param

7bc0ceb

Reduce test field size to (2^31 - 1) to support 32 bit systems

AlexWaygood reviewed Nov 18, 2024

View reviewed changes

.pre-commit-config.yaml Outdated Show resolved Hide resolved

ZeroIntensity reviewed Nov 18, 2024

View reviewed changes

ZeroIntensity added the topic-ctypes label Nov 18, 2024

hugovk reviewed Nov 18, 2024

View reviewed changes

vstinner changed the title ~~gh-126937: Fix TypeError when a field's size is >65535 bytes~~ gh-126937, ctypes: Fix TypeError when a field's size is >65535 bytes Nov 19, 2024

picnixz changed the title ~~gh-126937, ctypes: Fix TypeError when a field's size is >65535 bytes~~ gh-126937: ctypes: fix TypeError when a field's size is >65535 bytes Nov 29, 2024

serhiy-storchaka self-requested a review December 3, 2024 15:42

encukou added 9 commits December 4, 2024 11:06

Merge in the main branch

cd8c011

Remove pre-commit config (moved to python#126971)

a3bd171

Use PyLong_AsSsize_t to convert to Py_ssize_t

5e8aec0

Test maximum field size (sys.maxsize / 8, so we can count bits later)

7d3bb10

Rework the asserts.

946f261

Test naked c_char as well

19c1b3b

Better name for the test

dcf97f2

Add comment

03bc32b

Reword changelog

f61faa6

Co-Authorsd-By: Peter Bierma <zintensitydev@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-126937: ctypes: fix TypeError when a field's size is >65535 bytes #126938

gh-126937: ctypes: fix TypeError when a field's size is >65535 bytes #126938

Melissa0x1f992 commented Nov 17, 2024 •

edited by bedevere-app bot

Loading

cpython-cla-bot bot commented Nov 17, 2024 •

edited

Loading

bedevere-app bot commented Nov 17, 2024

Melissa0x1f992 commented Nov 17, 2024

terryjreedy commented Nov 17, 2024 •

edited

Loading

terryjreedy commented Nov 17, 2024

picnixz commented Nov 18, 2024

Melissa0x1f992 commented Nov 18, 2024

Melissa0x1f992 commented Nov 18, 2024

Melissa0x1f992 commented Nov 18, 2024

ZeroIntensity left a comment

ZeroIntensity Nov 18, 2024 •

edited

Loading

ZeroIntensity Nov 18, 2024

ZeroIntensity Nov 18, 2024

ZeroIntensity Nov 18, 2024

hugovk Nov 18, 2024

hugovk Nov 18, 2024

encukou commented Nov 20, 2024 •

edited

Loading

encukou commented Nov 29, 2024

Melissa0x1f992 commented Dec 3, 2024

encukou commented Dec 4, 2024

		@@ -0,0 +1 @@
		Fixed TypeError when a Structure's field's size is >65535 bytes

	Fixed TypeError when a Structure's field's size is >65535 bytes
	Fix :exc:`TypeError` when a :class:`ctypes.Structure` has a field size that doesn't fit into an unsigned 16-bit integer.

gh-126937: ctypes: fix TypeError when a field's size is >65535 bytes #126938

Are you sure you want to change the base?

gh-126937: ctypes: fix TypeError when a field's size is >65535 bytes #126938

Conversation

Melissa0x1f992 commented Nov 17, 2024 • edited by bedevere-app bot Loading

cpython-cla-bot bot commented Nov 17, 2024 • edited Loading

bedevere-app bot commented Nov 17, 2024

Melissa0x1f992 commented Nov 17, 2024

terryjreedy commented Nov 17, 2024 • edited Loading

terryjreedy commented Nov 17, 2024

picnixz commented Nov 18, 2024

Melissa0x1f992 commented Nov 18, 2024

Melissa0x1f992 commented Nov 18, 2024

Melissa0x1f992 commented Nov 18, 2024

ZeroIntensity left a comment

Choose a reason for hiding this comment

ZeroIntensity Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

ZeroIntensity Nov 18, 2024

Choose a reason for hiding this comment

ZeroIntensity Nov 18, 2024

Choose a reason for hiding this comment

ZeroIntensity Nov 18, 2024

Choose a reason for hiding this comment

hugovk Nov 18, 2024

Choose a reason for hiding this comment

hugovk Nov 18, 2024

Choose a reason for hiding this comment

encukou commented Nov 20, 2024 • edited Loading

encukou commented Nov 29, 2024

Melissa0x1f992 commented Dec 3, 2024

encukou commented Dec 4, 2024

Melissa0x1f992 commented Nov 17, 2024 •

edited by bedevere-app bot

Loading

cpython-cla-bot bot commented Nov 17, 2024 •

edited

Loading

terryjreedy commented Nov 17, 2024 •

edited

Loading

ZeroIntensity Nov 18, 2024 •

edited

Loading

encukou commented Nov 20, 2024 •

edited

Loading