PY_SSIZE_T_CLEAN conflicts with Py_LIMITED_API

[dholth]

BPO	27499
Nosy	@vstinner, @dholth, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-07-12.15:59:47.480>
labels = ['extension-modules']
title = 'PY_SSIZE_T_CLEAN conflicts with Py_LIMITED_API'
updated_at = <Date 2020-06-10.16:48:48.291>
user = 'https://github.com/dholth'

bugs.python.org fields:

activity = <Date 2020-06-10.16:48:48.291>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2016-07-12.15:59:47.480>
creator = 'dholth'
dependencies = []
files = []
hgrepos = []
issue_num = 27499
keywords = []
message_count = 6.0
messages = ['270252', '270255', '270256', '270260', '371221', '371222']
nosy_count = 3.0
nosy_names = ['vstinner', 'dholth', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue27499'
versions = ['Python 3.5']

[dholth]

When compiling my cryptacular extension https://bitbucket.org/dholth/cryptacular I noticed -DPy_LIMITED_API -DPY_SSIZE_T_CLEAN creates a binary that does not actually use the limited api. This causes segfaults on Linux but does not appear to cause problems on Windows.

I found some emails suggestid PY_SSIZE_T_CLEAN was supposed to go away entirely?

[serhiy-storchaka]

First we should make PY_SSIZE_T_CLEAN mandatory for the term of at least two releases (or to the end of 2.7 support).

[dholth]

Here it is. https://mail.python.org/pipermail/python-3000/2008-November/015344.html

On Sat, Nov 22, 2008 at 06:29, Barry Warsaw <barry at python.org> wrote:

On Nov 22, 2008, at 4:05 AM, Martin v. Löwis wrote:

> I just noticed that the Python 3 C API still contains PY_SSIZE_T_CLEAN.
>
> This macro was a transition mechanism, to allow extensions to use
> Py_ssize_t in PyArg_ParseTuple, while allowing other module continue
> to use int.
>
> In Python 3, I would like the mechanism, making Py_ssize_t the only
> valid data type for size in, say, s# parsers.
>
> Is it ok to still change that?

Given that we just released the last planned candidate, I'd say it was too
late to change this for Python 3.0.

But we can at least document that the macro is a gone as soon as 3.0
final is out the door.

-Brett

[dholth]

Oh, I can avoid this problem by setting Py_LIMITED_API to 0x30300000 or greater.

[vstinner]

Oh, I can avoid this problem by setting Py_LIMITED_API to 0x30300000 or greater.

Hum, maybe the behavior is different because of the following code in Include/modsupport.h:

/* Due to a glitch in 3.2, the _SizeT versions weren't exported from the DLL. */
#if !defined(PY_SSIZE_T_CLEAN) || !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03030000

[vstinner]

See also bpo-40943: PEP-353: Drop support for PyArg_ParseTuple() format when PY_SSIZE_T_CLEAN is not defined.

[sahlberg]

SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

This change hurt me too. A python c module I depend on just started doing this when I upgraded to Fedora36.
So, how do I fix this so it works again like in python3.8 ?

This broke the ABI in a minor release and ?

[vstinner]

So, how do I fix this so it works again like in python3.8 ?

Modify the extension to use Py_ssize_t and define PY_SSIZE_T_CLEAN macro.

[vstinner]

Python 3.10 change (issue #85115):

The PY_SSIZE_T_CLEAN macro must now be defined to use PyArg_ParseTuple() and Py_BuildValue() formats which use #: es#, et#, s#, u#, y#, z#, U# and Z#. See Parsing arguments and building values and PEP 353. (Contributed by Victor Stinner in bpo-40943.)

[vstinner]

@encukou: You may be interested by this tricky issue :-) I don't know its status in Python 3.12.

[encukou]

With PY_SSIZE_T_CLEAN, the PyArg_ParseTuple name is redefined as _PyArg_ParseTuple_SizeT, which passes an extra flag to the internal routines. Handling the affected format specifiers fails without the flag.
Both _PyArg_ParseTuple_SizeT and PyArg_ParseTuple are present in the ABI since 3.3. So, yes, 3.2 was buggy – use 0x03030000.

The situation is unfortunate for non-C languages (which don't see macros).

Of course, removing the functionality was disruptive, like any removal of functionality. It wouldn't go away if I'd been in charge.
(It's not an ABI break – the module can still be loaded, it raises an exception at runtime – but that's little consolation to the affected user.)
Define PY_SSIZE_T_CLEAN, define Py_LIMITED_API as 0x3yy0000 (yy>03), recompile, and the module should again work on Python 3.yy and above.

[vstinner]

If recent versions of Python are fine, can we just close the issue?

[serhiy-storchaka]

How non-C languages work with variable-argument C functions? How they work with pointers to ssize_t?

[encukou]

How non-C languages work with variable-argument C functions?

With difficulty.
Apparently, Rust has the “ready-to-stabilize” RFC 2137 for unsafe extern "C" variadic functions. I assume there's some architecture- or ABI-specific black magic behind it.

How they work with pointers to ssize_t?

Same as any other pointer to a C integral type. It's not trivial, but if you can't do it, Python's C API won't be too useful.

[encukou]

If recent versions of Python are fine, can we just close the issue?

Defining Py_LIMITED_API to 3 is still common, and IMO we can do better there. But it's a low priority for me :(

[erlend-aasland]

Is this issue still relevant after #106315? AFAICS, we can close this as outdated, no?

[vstinner]

Is this issue still relevant after #106315? AFAICS, we can close this as outdated, no?

I close the issue. While the history is complicated, the root stable ABI issue has been fixed in Python 3.13 by commit adccff3 of gh-104922.

History:

• Python 3.8: Use of # variants of formats in parsing or building value (e.g. PyArg_ParseTuple(), Py_BuildValue(), PyObject_CallFunction(), etc.) without PY_SSIZE_T_CLEAN defined raises DeprecationWarning
• Python 3.10: The PY_SSIZE_T_CLEAN macro must now be defined to use PyArg_ParseTuple() and Py_BuildValue() formats which use #: es#, et#, s#, u#, y#, z#, U# and Z#.
• Python 3.13: You no longer have to define the PY_SSIZE_T_CLEAN macro before including Python.h when using # formats in format codes. APIs accepting the format codes always use Py_ssize_t for # formats.

At the ABI level, Python <= 3.12 calls different functions if PY_SSIZE_T_CLEAN is defined. Example:

#ifdef PY_SSIZE_T_CLEAN
#define PyArg_Parse                     _PyArg_Parse_SizeT
...

Python >= 3.13 calls the same function if PY_SSIZE_T_CLEAN macro is defined or not. The macro PY_SSIZE_T_CLEAN is ignored by Python 3.13 and newer.

If you consider Python >= 3.13, the stable ABI is fixed.

Python 3.11 and 3.12 still have the ABI issue. From what I understood, PY_SSIZE_T_CLEAN was never compatible with the stable ABI.

Python 3.10 and newer always use Py_ssize_t for size arguments, instead of int.