Skip to content

gh-97588: Fix ctypes structs #97702

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 128 commits into from
May 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
128 commits
Select commit Hold shift + click to select a range
7922585
gh-97588: Fix ctypes structs
matthiasgoergens Oct 1, 2022
2307932
📜🤖 Added by blurb_it.
blurb-it[bot] Oct 1, 2022
47f826c
Handling pack as well
matthiasgoergens Oct 1, 2022
5a32211
Handle basedict, too
matthiasgoergens Oct 1, 2022
8d79d8f
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Oct 2, 2022
2d07375
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Oct 3, 2022
8bc8535
Merge remote-tracking branch 'matthias/fix-bitfield-clean' into fix-b…
matthiasgoergens Oct 5, 2022
c3d162b
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Oct 5, 2022
e5ed9ac
Split windows and linux
matthiasgoergens Oct 5, 2022
b0f9819
Compiles
matthiasgoergens Oct 5, 2022
2dee0e3
Abstract out proto stuff
matthiasgoergens Oct 5, 2022
79ef347
Clean align
matthiasgoergens Oct 5, 2022
6170dad
Hypothesis tests pass
matthiasgoergens Oct 5, 2022
871ca1a
Formatting
matthiasgoergens Oct 5, 2022
c162144
Clean up
matthiasgoergens Oct 5, 2022
a57cf2c
Adapt tests for Windows
matthiasgoergens Oct 5, 2022
3f7c4cc
Fix order
matthiasgoergens Oct 5, 2022
e39a271
Fix alignment test
matthiasgoergens Oct 5, 2022
359ed58
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Oct 5, 2022
52ef8d2
Avoid casting
matthiasgoergens Oct 5, 2022
e8102c4
Fixup
matthiasgoergens Oct 5, 2022
600e144
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Oct 5, 2022
9bad706
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Oct 5, 2022
5121857
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Oct 6, 2022
6cd27ea
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Oct 8, 2022
6b6fa8a
Add ability to force msvc compatibility even when not doing any packing
matthiasgoergens Oct 8, 2022
ca9d580
More tests
matthiasgoergens Oct 8, 2022
1401ee4
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Oct 9, 2022
235fa68
Big endian works
matthiasgoergens Oct 9, 2022
a534e18
Merge remote-tracking branch 'matthias/fix-bitfield-clean' into fix-b…
matthiasgoergens Oct 9, 2022
53db061
More tests
matthiasgoergens Oct 9, 2022
9c249c3
More asserts
matthiasgoergens Oct 9, 2022
3cf4747
Drop unneeded parameters
matthiasgoergens Oct 9, 2022
8beddb9
Drop unneeded parameters
matthiasgoergens Oct 9, 2022
9b64ff6
Drop unneeded parameters
matthiasgoergens Oct 9, 2022
4d0dab5
Clean up
matthiasgoergens Oct 9, 2022
bf98667
Explain
matthiasgoergens Oct 9, 2022
8223063
More cleanup
matthiasgoergens Oct 9, 2022
e80a09a
More cleanup
matthiasgoergens Oct 9, 2022
33a10fb
More cleanup
matthiasgoergens Oct 9, 2022
0f3c5a3
Move common assert
matthiasgoergens Oct 9, 2022
47eca3c
Break line
matthiasgoergens Oct 9, 2022
456afd0
Clean TODO
matthiasgoergens Oct 9, 2022
bc799ac
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Oct 9, 2022
212cf13
Fix test for Windows
matthiasgoergens Oct 9, 2022
e7e5ae9
Merge remote-tracking branch 'matthias/fix-bitfield-clean' into fix-b…
matthiasgoergens Oct 9, 2022
45e26ec
Remove warning
matthiasgoergens Oct 9, 2022
ab42187
Port test from PR-19850
matthiasgoergens Oct 9, 2022
640c062
Fix assert
matthiasgoergens Oct 9, 2022
f3e04af
Revert "Port test from PR-19850"
matthiasgoergens Oct 9, 2022
bff34a1
Support gcc's packed structs
matthiasgoergens Oct 9, 2022
5d6f0f7
Remove gcc_packed experiment
matthiasgoergens Oct 9, 2022
d9c1fca
Fix test
matthiasgoergens Oct 10, 2022
cc9ea8a
Fix ms_struct
matthiasgoergens Oct 10, 2022
1554083
Fix ms_struct
matthiasgoergens Oct 10, 2022
a86eadc
Indenting
matthiasgoergens Oct 10, 2022
eefa1ad
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Oct 10, 2022
28c7fe7
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Oct 10, 2022
f0a92b0
Fix miscounted parens
matthiasgoergens Oct 10, 2022
a3a390b
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Oct 10, 2022
e997b5f
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Mar 14, 2023
5a7ea09
make regen-all
matthiasgoergens Mar 14, 2023
0f73919
clean up
matthiasgoergens Mar 14, 2023
cdfc3c6
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Mar 15, 2023
6226e55
make regen-all
matthiasgoergens Mar 15, 2023
4d48eba
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens Mar 16, 2023
20b5582
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Mar 17, 2023
eb502d7
Explain where 999 comes from
matthiasgoergens Mar 17, 2023
0cf5f4e
fix typo
matthiasgoergens Mar 17, 2023
3457558
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Mar 17, 2023
fd3cd0b
Explain magic 999
matthiasgoergens Mar 17, 2023
2d8492e
Incorporate Sam's tests
matthiasgoergens Mar 27, 2023
a9f7f14
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens May 24, 2023
bee6c53
Tickle CI/CD
matthiasgoergens May 24, 2023
220e19e
Merge branch 'main' into fix-bitfield-clean
matthiasgoergens May 24, 2023
60cebe2
Merge branch 'main' into fix-bitfield-clean
gpshead May 24, 2023
3424a7d
Merge commit '698a0da7d440856a90b45964e9082b5a55387b80' into fix-bitf…
matthiasgoergens Apr 14, 2024
2506eea
Merge commit '0841ca7932987f30a2a23d39f3e6e141622d6fea' into fix-bitf…
matthiasgoergens Apr 14, 2024
65654b4
Merge commit '9f7176d360b5898003d5ca78bab1822ad67867c4' into fix-bitf…
matthiasgoergens Apr 14, 2024
98767da
Fix compile
matthiasgoergens Apr 14, 2024
3ca703f
Merge commit '298bcdc185d1a9709271e61a4cc529d33483add4' into fix-bitf…
matthiasgoergens Apr 14, 2024
c40ef7d
Merge commit 'dcaf33a41d5d220523d71c9b35bc08f5b8405dac' into fix-bitf…
matthiasgoergens Apr 14, 2024
d840f01
Merge commit '7e2fef865899837c47e91ef0180fa59eb03e840b' into fix-bitf…
matthiasgoergens Apr 14, 2024
8b9f0eb
Merge commit 'ea94b3b149eeadf33c2f7c46f16dcda0adc7cf4e' into fix-bitf…
matthiasgoergens Apr 14, 2024
4e8c220
Merge remote-tracking branch 'origin/main' into fix-bitfield-clean
matthiasgoergens Apr 14, 2024
0369d0d
Fix merge
matthiasgoergens Apr 14, 2024
d4dc2c0
Clean up
matthiasgoergens Apr 14, 2024
f75d7d6
Fix tests
matthiasgoergens Apr 14, 2024
cdc5cdc
Merge in the main branch
encukou Apr 22, 2024
0da36ad
Add Hypothesis test
encukou Apr 22, 2024
b6f7117
Merge in the main branch
encukou Apr 29, 2024
5e47d5f
dict -> info
encukou Apr 30, 2024
de22b39
Conditionalize ms_struct
encukou Apr 30, 2024
cdd1860
Use fixed-width types in tests
encukou Apr 30, 2024
dd84ac3
Use subTest
encukou Apr 30, 2024
8a6fb67
fixup! dict
encukou Apr 30, 2024
07ea42d
Turn `info->size != info->align` assertion into a TypeError
encukou Apr 30, 2024
3fa7f55
PEP 7
encukou Apr 30, 2024
eb2c4fb
Change _ms_struct_ to _layout_
encukou Apr 30, 2024
22cd86c
Merge in the main branch
encukou Apr 30, 2024
637b961
PEP 7
encukou Apr 30, 2024
5309b7e
TMP
encukou Apr 30, 2024
489c5c8
Add generated tests
encukou May 2, 2024
3594473
Merge in the main branch
encukou May 2, 2024
f780496
Remove ideas
encukou May 2, 2024
6ade022
Regen
encukou May 2, 2024
cfa6647
Do not set _layout_
encukou May 2, 2024
df162b0
We don't always get union alignment right
encukou May 2, 2024
836a5cc
Remove unused stuff
encukou May 2, 2024
b07ae33
Packing implies ms layout
encukou May 2, 2024
6dd7a8d
Docs & whitespace
encukou May 2, 2024
0cf1049
Use limited API again
encukou May 2, 2024
bc91549
Appease the linter
encukou May 3, 2024
70bbc26
Fix ReST typo
encukou May 3, 2024
6e23b3d
Use nicer error output for memory dumps
encukou May 3, 2024
1b90841
Regen global strings
encukou May 3, 2024
2c4874b
Allow alignment < size, as in int64_t on x86 (32-bit), GCC layout
encukou May 3, 2024
738323f
Add examples from MS docs
encukou May 4, 2024
af7487c
Use correct spelling for test skips
encukou May 5, 2024
5353352
Only use 'attribute((ms_struct))' on x86_64 & ppc; on GCC and clang
encukou May 5, 2024
fe76d45
Remove the `info->size == info->align` assertion for _layout_='ms'
encukou May 5, 2024
c4714f1
Fix silly mistake in the macro
encukou May 5, 2024
c79725d
Regen
encukou May 5, 2024
cace0c9
Include field name in error message
encukou May 5, 2024
61e92bf
Conditionally skip tests that assume sizeof(int64) == alignment(int64)
encukou May 5, 2024
ba61051
Use ms_struct only on Windows and x86/ppc GCC/clang
encukou May 5, 2024
8991444
Use proper PyArg_ParseTuple code for Py_ssize_t
encukou May 6, 2024
bc1225b
Merge in the main branch
encukou May 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 37 additions & 8 deletions Doc/library/ctypes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -661,14 +661,18 @@ for debugging because they can provide useful information::
guaranteed by the library to work in the general case. Unions and
structures with bit-fields should always be passed to functions by pointer.

Structure/union alignment and byte order
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

By default, Structure and Union fields are aligned in the same way the C
compiler does it. It is possible to override this behavior by specifying a
:attr:`~Structure._pack_` class attribute in the subclass definition.
This must be set to a positive integer and specifies the maximum alignment for the fields.
This is what ``#pragma pack(n)`` also does in MSVC.
Structure/union layout, alignment and byte order
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

By default, Structure and Union fields are laid out in the same way the C
compiler does it. It is possible to override this behavior entirely by specifying a
:attr:`~Structure._layout_` class attribute in the subclass definition; see
the attribute documentation for details.

It is possible to specify the maximum alignment for the fields by setting
the :attr:`~Structure._pack_` class attribute to a positive integer.
This matches what ``#pragma pack(n)`` does in MSVC.

It is also possible to set a minimum alignment for how the subclass itself is packed in the
same way ``#pragma align(n)`` works in MSVC.
This can be achieved by specifying a ::attr:`~Structure._align_` class attribute
Expand Down Expand Up @@ -2540,6 +2544,31 @@ fields, or any other data types containing pointer type fields.
the structure when being packed or unpacked to/from memory.
Setting this attribute to 0 is the same as not setting it at all.

.. attribute:: _layout_

An optional string naming the struct/union layout. It can currently
be set to:

- ``"ms"``: the layout used by the Microsoft compiler (MSVC).
On GCC and Clang, this layout can be selected with
``__attribute__((ms_struct))``.
- ``"gcc-sysv"``: the layout used by GCC with the System V or “SysV-like”
data model, as used on Linux and macOS.
With this layout, :attr:`~Structure._pack_` must be unset or zero.

If not set explicitly, ``ctypes`` will use a default that
matches the platform conventions. This default may change in future
Python releases (for example, when a new platform gains official support,
or when a difference between similar platforms is found).
Currently the default will be:

- On Windows: ``"ms"``
- When :attr:`~Structure._pack_` is specified: ``"ms"``
- Otherwise: ``"gcc-sysv"``

:attr:`!_layout_` must already be defined when
:attr:`~Structure._fields_` is assigned, otherwise it will have no effect.

.. attribute:: _anonymous_

An optional sequence that lists the names of unnamed (anonymous) fields.
Expand Down
12 changes: 12 additions & 0 deletions Doc/whatsnew/3.13.rst
Original file line number Diff line number Diff line change
Expand Up @@ -469,6 +469,18 @@ copy
any user classes which define the :meth:`!__replace__` method.
(Contributed by Serhiy Storchaka in :gh:`108751`.)

ctypes
------

* The layout of :ref:`bit fields <ctypes-bit-fields-in-structures-unions>` in
:class:`~ctypes.Structure` and :class:`~ctypes.Union` was improved to better
match platform defaults (GCC/Clang or MSC). In particular, fields no longer
overlap.
(Contributed by Matthias Görgens in :gh:`97702`.)
* A :attr:`ctypes.Structure._layout_` class attribute can be set
to help match a non-default ABI.
(Contributed by Petr Viktorin in :gh:`97702`.)

dbm
---

Expand Down
1 change: 1 addition & 0 deletions Include/internal/pycore_global_objects_fini_generated.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Include/internal/pycore_global_strings.h
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,7 @@ struct _Py_global_strings {
STRUCT_FOR_ID(_initializing)
STRUCT_FOR_ID(_io)
STRUCT_FOR_ID(_is_text_encoding)
STRUCT_FOR_ID(_layout_)
STRUCT_FOR_ID(_length_)
STRUCT_FOR_ID(_limbo)
STRUCT_FOR_ID(_lock_unlock_module)
Expand Down
1 change: 1 addition & 0 deletions Include/internal/pycore_runtime_init_generated.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions Include/internal/pycore_unicodeobject_generated.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading