Backport of `PyConfig.use_system_logger` has changed ABI in a patch 3.13.1 -> 3.13.2

[davidhewitt]

Bug report

Bug description:

In #127592 a new PyConfig flag was added for apple platforms to allow use of the system logger.

This PR looks like it was backported to all branches from 3.9 through to 3.13, and has been released in 3.13.2 and 3.12.9.

Unfortunately, this has changed the public ABI (layout of the PyConfig struct) on a patch release. This means that binaries using the PyConfig API and built using 3.13.1 or 3.13.0 (or 3.12.8) will potentially misbehave if running against a newer interpreter.

I picked this up after getting a PyO3 "FFI check" failure on 3.13.2, reporting that our FFI bindings were no longer compatible with CPython.

In PEP 741 it was previously noted that backporting a flag to break the public ABI was decided against, so I believe this to be a bug.

Do I wait for CPython to revert this? Or do I make PyO3 adjust to the new ABI (and break compatibility with all existing interpreters)?

ping @vstinner as I know you have been very active on this API.

CPython versions tested on:

3.13

Operating systems tested on:

macOS

Linked PRs

• [3.13] gh-130940: Remove PyConfig.use_system_logger #131129
• gh-130940: Modify default behavior of PyConfig.use_system_logger to enable on iOS #131172

[davidhewitt]

Correction: while #127592 lists a lot of backports, it looks like the backport was only ever applied to the 3.13 branch. So it is only 3.13.1 to 3.13.2 which has an ABI break across patch versions, 3.12 and older are unaffected.

Will 3.13.3 revert this backport?

[davidhewitt]

ping @freakboy3742 as the author of the PR, maybe you have some context or opinion on the right path forward.

[ZeroIntensity]

This does look like a breaking ABI change, but note that this only applies to iOS builds, due to the __APPLE__ macro.

[davidhewitt]

I believe it to be for all apple platforms, including macOS.

https://developer.apple.com/library/archive/documentation/Porting/Conceptual/PortingUnix/compiling/compiling.html#//apple_ref/doc/uid/TP40002850-SW13 indicates __APPLE__ is for any apple system.

[freakboy3742]

Apologies for the breakage - that was definitely unintentional, and evidently slipped through the review process.

Tagging @ned-deily for the macOS perspective; @Yhg1s for the 3.13 release impact.

Confirming some other details in this thread:

• __APPLE__ will be enabled on macOS as well as iOS.

• The backports linked on gh-126925: Modify how iOS test results are gathered #127592 to 3.9-3.12 releases are on my own branch, where I back port iOS patches to support older Python releases. AFAICT, the only official release with an ABI breakage is 3.13.2.

By way of background - this patch was needed because of changes in Xcode 16 that broke the way we were capturing output for iOS testing. The core logic that is being enabled by the config flag is required for iOS testing to work. There's almost no use case for the flag to ever be disabled on iOS.

However, it's entirely optional for macOS. It's a useful feature to have for very specific use cases (GUI apps that don't have a visible console), but there are other workarounds for that use case. At the very least, any impact on macOS can be reverted without impact.

So - a complete reversion isn't possible without breaking all iOS testing; but there are some options.

Firstly, we can revert the ABI-modifying changes to PyConfig. This doesn't need to be configurable in 3.13; we can treat this as a new 3.14 feature.

That then leaves the question of how we handle not having access to the flag. On macOS, we can just ignore it - it's an optional feature, so we can just not enable the optional behavior on macOS. But on iOS, we've got a choice to make.

One option would be to leave the code to enable the system logger in place, but only activate it in the testbed app. This would mean using a private API for the 3.13 version of the testbed - but then, if you squint, the iOS testbed is "internal" Python code? Maybe? A variant on this approach would be to migrate the system log handling code wholesale into the testbed, instead of it being in pylifecyle.c. That avoids the private API problem entirely.

The other option would be to make the enabling the system logger the default behavior on iOS. This would be a change in behaviour for iOS - but it's arguably a much better default. Without enabling the system logger, stdout and stderr content are essentially swallowed at runtime. You can see stdout/stderr content in Xcode, but it won't appear in any system logs for a deployed app. This would also leave the door open for modifying 3.14 so that the default value for PyConfig.use_system_logger is 1 on iOS, making the default behavior more useful.

Of the three, the third option (change the default) would be my preferred option, as the end-state leaves iOS support in a much better place. However, I appreciate that it's a backwards incompatible change, so it's not a change that should be made lightly.

I'm happy to work up the patch to fix this as soon as we've got agreement on the approach to take here.

[ned-deily]

That's unfortunate that we didn't consider the 3.13.x ABI implications when reviewing this; sorry!

For macOS, if I understand the implications correctly, the original change made would essentially only affect programs on macOS that embed Python 3.13 using an externally-provided shared-library (or framework) build of Python 3.13? If so, I would imagine there are very few such programs: since Python 3.13 isn't provided by the OS, I would expect most such programs would supply their own private copy of Python 3.13. I suppose it could be an issue for distributors like Homebrew or MacPorts or Conda but even there I would think it would be likely that both the embedding application and the Python 3.13.x version would be updated at the same time.

So, for the next release of 3.13 (3.13.3), I guess the by-the-book thing to do would be to revert (and document the revert) the ABI change for macOS but, if my understanding is correct, the impact either way is likely to be very limited. For 3.14, an ABI change between 3.13.x and the initial release of 3.14 should be acceptable but perhaps taking another look at the changes while we are still in the alpha phase of the release cycle might be a good idea. Am I missing something? Perhaps @ronaldoussoren has an opinion?