ctypes: unions as arguments

[arigo]

BPO	16575
Nosy	@arigo, @theller, @vsajip, @vstinner, @alex, @meadori, @ammaraskar
PRs	bpo-22273: Update ctypes to correctly handle arrays in small structur… #15839 bpo-16575: Add checks for unions passed by value to functions. #16430 bpo-16575: Add checks for unions passed by value to functions. #16799 [3.8] bpo-16575: Add checks for unions passed by value to functions. … #17016 [3.7] bpo-16575: Add checks for unions passed by value to functions. (GH-16799) #17017 [3.7] bpo-16575: Fix refleak on passing unions in ctypes #17064 bpo-16575: Disabled checks for union types being passed by value. #17960 [3.8] bpo-16575: Disabled checks for union types being passed by value. (GH-17960) #17964 [3.7] bpo-16575: Disabled checks for union types being passed by value. (GH-17960) #17970
Files	test184_lib.tgz

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vsajip'
closed_at = None
created_at = <Date 2012-11-29.04:24:04.013>
labels = ['ctypes', 'type-crash', 'docs']
title = 'ctypes: unions as arguments'
updated_at = <Date 2020-01-12.20:55:57.455>
user = 'https://github.com/arigo'

bugs.python.org fields:

activity = <Date 2020-01-12.20:55:57.455>
actor = 'vinay.sajip'
assignee = 'vinay.sajip'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'ctypes']
creation = <Date 2012-11-29.04:24:04.013>
creator = 'arigo'
dependencies = []
files = ['28155']
hgrepos = []
issue_num = 16575
keywords = ['patch']
message_count = 20.0
messages = ['176627', '183461', '183464', '183817', '183819', '186689', '351454', '355741', '355751', '355754', '355993', '355994', '356053', '356141', '356147', '356230', '359834', '359836', '359844', '359876']
nosy_count = 11.0
nosy_names = ['arigo', 'theller', 'vinay.sajip', 'vstinner', 'Arfrever', 'alex', 'eli.bendersky', 'meador.inge', 'docs@python', 'python-dev', 'ammar2']
pr_nums = ['15839', '16430', '16799', '17016', '17017', '17064', '17960', '17964', '17970']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue16575'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3', 'Python 3.4']

[arigo]

ctypes pretends to support passing arguments to C functions that are unions (not pointers to unions), but that's a lie. In fact, the underlying libffi does not support it. The attached example misbehaves on Linux x86-64.

[elibendersky]

A minimal fix would be to update the documentation.

A more comprehensive fix would be to tweak ctypes to reject unions and bit-fields when running on non-x86 (does this work for ARM and other non-Intel archs?)

An even more comprehensive fix would be to make this work, but that would require forking libffi and should presumably be first implemented upstream.

[arigo]

See also http://bugs.python.org/issue16576.

[python-dev]

New changeset 0acd9408b6f1 by Eli Bendersky in branch '3.2':
Add warning in ctypes documentation for bpo-16575 and bpo-16576
http://hg.python.org/cpython/rev/0acd9408b6f1

New changeset bfc159f8e4b4 by Eli Bendersky in branch '3.3':
Add warning in ctypes documentation for bpo-16575 and bpo-16576
http://hg.python.org/cpython/rev/bfc159f8e4b4

New changeset 502624235c7b by Eli Bendersky in branch 'default':
Add warning in ctypes documentation for bpo-16575 and bpo-16576
http://hg.python.org/cpython/rev/502624235c7b

[python-dev]

New changeset eece32440a52 by Eli Bendersky in branch '2.7':
Add warning in ctypes documentation for bpo-16575 and bpo-16576
http://hg.python.org/cpython/rev/eece32440a52

[elibendersky]

I've opened a libffi issue in an attempt to get this fixed upstream:

https://github.com/atgreen/libffi/issues/33

[vsajip]

Link to issue has changed to:

libffi/libffi#33

[vsajip]

New changeset 79d4ed1 by Vinay Sajip in branch 'master':
bpo-16575: Add checks for unions passed by value to functions. (GH-16799)
79d4ed1

[vsajip]

New changeset 9528997 by Vinay Sajip in branch '3.8':
[3.8] bpo-16575: Add checks for unions passed by value to functions. (GH-16799) (GH-17016)
9528997

[vsajip]

New changeset 0118d10 by Vinay Sajip in branch '3.7':
[3.7] bpo-16575: Add checks for unions passed by value to functions. (GH-16799) (GH-17017)
0118d10

[vstinner]

This change introduced a reference leak on Windows. Example on 3.7:

https://buildbot.python.org/all/#/builders/132/builds/645

test_ctypes leaked [174, 174, 174] references, sum=522
test_ctypes leaked [76, 77, 77] memory blocks, sum=230

[vstinner]

Same issue on x86 Gentoo Refleaks 3.7:

test_ctypes leaked [174, 174, 174] references, sum=522
test_ctypes leaked [76, 78, 76] memory blocks, sum=230

https://buildbot.python.org/all/#/builders/114/builds/631

[ammaraskar]

Opened #17064 to fix this. Essentially it's a tiny little oversight in the back-porting. In the 3.7 branch, we perform an attribute lookup for from_param before the union checking code, so we must remember to DECREF it. This is unlike master where the attribute lookup happens after the union checking code.

λ win32\python_d.exe -m test -R 3:3 test_ctypes
0:00:00 Run tests sequentially
0:00:00 [1/1] test_ctypes
beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

1 test OK.

Total duration: 14.8 sec
Tests result: SUCCESS

[vsajip]

New changeset 484edbf by Vinay Sajip (Ammar Askar) in branch '3.7':
bpo-16575: Fix refleak on passing unions in ctypes (GH-17064)
484edbf

[ammaraskar]

Will close after https://buildbot.python.org/all/#/builders?tags=%2Brefleak&tags=%2B3.7 go back to green.

[vstinner]

3.7 refleaks buildbots pass again (ignoring a few warnings).

[vsajip]

It seems that notwithstanding the issues identified in the original bug report, calls by value of structs containing unions are being used out there in the wild. Examples:

The comtypes library (Windows):
#16799 (comment)

The ctypesgen library, which is used by other projects across platforms:
ctypesgen/ctypesgen#77

Linux examples:
waveform80/picamera#604
hcpl/xkbgroup#15
enkore/i3pystatus#771

So, it would seem that the simplest course of action is to disable the checks for now. The checks will be commented out and this issue and the related issue bpo-16576 will be reopened.

[vsajip]

New changeset c12440c by Vinay Sajip in branch 'master':
bpo-16575: Disabled checks for union types being passed by value. (GH-17960)
c12440c

[vsajip]

New changeset eb9ba2f by Vinay Sajip (Miss Islington (bot)) in branch '3.8':
bpo-16575: Disabled checks for union types being passed by value. (GH-17960) (GH-17964)
eb9ba2f

[vsajip]

New changeset 9dbf5d3 by Vinay Sajip in branch '3.7':
[3.7] bpo-16575: Disabled checks for union types being passed by value. (GH-17960) (GH-17970)
9dbf5d3