CPython: Condense patching, update patch versions, and add 3.11/3.12 #22599

Ahajha · 2024-01-30T22:58:23Z

Specify library name and version: cpython/all

Depends on #21387

Resolves #20768

Context for some of the changes:

The MSVC project patches are extremely fragile, usually breaking on any patch release. I've switched these to an extensive replace_in_file system, though admittedly I'm not a massive fan. Ideally I'd like to use fuzzy patching, but as far as I can tell they don't work.
CPython switched to using PkgConfig for most of its dependencies in 3.12, and removed setup.py for building extensions. 3.11 is in a transition state, where maybe half the deps used PkgConfig, and the other half was "discovered" manually through setup.py.

Once this is merged, I will make a draft PR for 3.13 (yet unreleased), as there are quite a few removals from Python's stdlib that will change some patching and remove at least one dependency: python/cpython#104773

I've read the contributing guidelines.
I've used a recent Conan client version close to the currently deployed.
I've tried at least one configuration locally with the conan-center hook activated.

# Conflicts: # recipes/cpython/all/conanfile.py

Fixed modules standard library modules not being found in test_package.

valgur · 2024-04-05T09:09:41Z

Not directly related to this PR, but these pkg-config aliases need to be turned into lists:

conan-center-index/recipes/cpython/all/conanfile.py

Line 773 in 66ad791

"pkg_config_aliases", f"python{py_version.major}"

conan-center-index/recipes/cpython/all/conanfile.py

Line 785 in 66ad791

"pkg_config_aliases", f"python{py_version.major}-embed"

I'm currently seeing a .pc file for each of the letters in python3.10-embed in PkgConfigDeps output. :)

And for Conan devs: a check for pkg_config_aliases and cmake_target_aliases to not be of type str would not hurt.

Ahajha · 2024-04-05T17:42:51Z

@valgur That's the second time I've been bitten by that exact problem, yes a check for that would be wonderful. I've fixed the issue, but of course now an unrelated issue popped up :/

conan-center-bot · 2024-04-06T09:16:53Z

Conan v1 pipeline ✔️

All green in build 36 (99d30a85374fbb0b4f7d37858bb533017f2718b8):

cpython/3.11.9:
All packages built successfully! (All logs)
cpython/3.10.14:
All packages built successfully! (All logs)
cpython/3.12.2:
All packages built successfully! (All logs)
cpython/3.9.19:
All packages built successfully! (All logs)
cpython/3.8.19:
All packages built successfully! (All logs)

Conan v2 pipeline ✔️

Note: Conan v2 builds are now mandatory. Please read our discussion about it.

All green in build 38 (99d30a85374fbb0b4f7d37858bb533017f2718b8):

cpython/3.10.14:
All packages built successfully! (All logs)
cpython/3.12.2:
All packages built successfully! (All logs)
cpython/3.8.19:
All packages built successfully! (All logs)
cpython/3.11.9:
All packages built successfully! (All logs)
cpython/3.9.19:
All packages built successfully! (All logs)

AbrilRBS

Thanks a lot. Can't thank you enough :)

ericLemanissier · 2024-04-09T11:08:49Z

recipes/cpython/all/conandata.yml

+    - patch_file: "patches/3.12/3.12.1-0002-remove-module-deps.patch"
+      patch_description: "Remove section of solution file forcing projects to be built that might not be used for this recipe"
+      patch_type: "conan"
+  "3.11.8":


@Ahajha is this a typo?

Ah, yea it is. I have another PR out, I'll fix it in that one.

Thanks, great!

…nd add 3.11/3.12 * wip * cpython: migrate to Conan v2 * cpython: fix autotools install step * cpython: fix libuuid requirement * cpython: update test_package * cpython: temporarily disable FindPythonX tests * cpython: package_type should be "library" * cpython: tidy * cpython: ncurses is required transitively * cpython: fix shared test * cpython: use cpp_info.aggregated_components() for deps * cpython: bsddb has been remove in Python 3 * cpython: fix openssl support * Fix package layout * Fix new generators in test package * Misc fixes * Remove redundant test * Add FIXMEs * Fix running test package multiple times in a row * Handle conan 1/2 differences in test_package * Fix MSBuild * Misc Conan 2.0 fixes * cpython: minor tweaks, bump deps * cpython: use altinstall Fixed modules standard library modules not being found in test_package. * Comment out versions with broken patches for now * Fix Autotools build * Restore old versions, temp remove 2.7.18 * Temp remove CPython 2.7.18 * Temp remove 2.7.18 * Misc MSVC fixes * Fix Windows/static build * Readd old test_package as test_v1_package * Add conanrun env to test package self.runs * Fix Linux shared build * Shared by default, don't delete compiler in package ID * Misc Mac fixes/cleanup * libxcrypt is transitive * Static by default for now * Attempt to fix 3.7 on Mac * Remove ssl module FIXME * self.settings.arch * Try using vendored libffi for mac on <3.9 * Fix PYTHONHOME on Linux, always set PYTHONHOME for test * Try vendored libffi on mac in debug mode * Remove "dynamic" check * Remove force on versions * Use Version() when comparing versions * Use a virtualrunenv in build, revert ffi changes * Bandaid fix for Linux * Use virtualrunenvs in v1 test package also * Skip 3.7 build on M1 macs * Set platform toolset directly on <3.8 * Fix issue with pip installed cmake * Python 2 compat in test_package.py * Re-add Python 2.7 + workarounds for MSVC * Misc pre-3.8 msvc fixes * Fix python 2 on Linux * Add 2.7.18 to config.yml * Remove includedirs for all components other than the main one * Use msvc_runtime_flag * Misc MSVC fixes * Skip building _freeze_importlib, fix 3.7.12 in msvc Debug MDd * Shot in the dark attempt at fixing Mac issue * Unconditionally generate VCVars in test package * Print config.log if autotools configure fails * Skip spam module test in 2.7 (debug, MDd) as well * Remove unnecessary removals from pcbuild.sln * Inject _bz2 programmatically * Also programattically patch bz2 in python 2.7 * Dynamically patch _elementtree, _ctypes, and _decimal projects * Patch pyexpat project programmatically * Inject libdb programmatically * Inject openssl into hashlib programmatically * Inject xz_utils programattically * Programattically inject sqlite * Programmatically inject openssl into _ssl project * Programmatically inject tk into _tkinter project * Programmatically inject zlib into pythoncore project * Remove/simplify redundant patching code * Minor cleanup * Refactor into _regex_replace_in_file * Refactoring of injection of conan props files * More refactoring for props file injecting * Programmatically remove parts of _ctypes project * Programmatically remove parts of _ctypes (patches) * Patch _decimal project programmatically * Remove sqlite project programmatically * Remove sqlite project programatically (patches) * Remove lzma project include programmatically * Remove openssl props import programmatically * Programmatically patch pyexpat project * Patch _elementtree project programmatically * Allow MSVC/Debug/dynamic with newer libffi * Test package cleanup * Fix Debug MSVC test package * Use test_package/test_package.py in test_v1_package * Fix conditional _d in debug mode * Add -Wl,--as-needed * Use extra_ldflags instead * Don't use -Wl,--as-needed on Apple OSs * Programmatically patch pythoncore project * Remove pythoncore patches * Remove unnecessary(?) patch * Programmatically patch _tkinter project * Programmatically patch _lzma project * Manually inject platform toolset on 3.8.x * Programmatically patch python project * Programmatically patch _freeze_importlib * Programmatically patch _ssl project, remove empty MSVC patches * Misc python 2 fixes * Programmatically patch setup py to skip tkinter detection * Programmatically patch ini CFLAGS and CPPFLAGS * Update 3.8.12 to 3.8.18 * Don't use use_2to3 after it is deprecated * Update 3.9.7 to 3.9.18 * Update 3.10.0 to 3.10.13 * Always manually specify the platform toolset * Fix MSVC static * Fix 3.7 patching * Update 3.7.12 to 3.7.17 * Add 3.11.7 (likely only MSVC works for now) * Add 3.12.1 (likely only working on Windows for now) * Update config.yml * Remove Py_SetProgramName (deprecated in 3.11) * Fix 3.12 on Linux * Fix 3.11 on Linux * Disable GCC 9 build * Cleanup test_v1_package * Only disable 3.12.1 GCC 9 build * Use pkgconfig tool requirement * Another shot in the dark attempt at fixing MacOS sporadic failure * Update to 3.12.2 and 3.11.8 * Update dependencies * Add patch descriptions * Enable short paths * Add link to nis system library * nis -> nsl * Add final newline * Simplify pkg-config info * Check for Linux or FreeBSD * Fix disabling sqlite3 (on Linux at least) * Simplify some file removing * rm unnecessary rm * Simplify pkg-config info * Check for Linux or FreeBSD * Fix disabling sqlite3 (on Linux at least) * Simplify some file removing * rm unnecessary rm * Disable GCC 9 build on 3.12 as a whole * Misc configure variable cleanup * Point test_v1_package to test_package sources * Fix lint warning * Diff cleanup * Conditionally add --with-system-ffi * Misc test package cleanup * Bump xz_utils * Bump xz_utils * Fix pymalloc argument * Don't read SSL config file Co-authored-by: Jordan Williams <jordan@jwillikers.com> * xcrypt is not transitive in newer versions * Bandaid fix for test package overflowing Windows max path length * Fix some lints * Remove EOL versions * Remove <3.8 logic * Fix more <3.8 logic * Small fix * Remove calculated Python major version * Use autotools.install() directly Co-authored-by: Uilian Ries <uilianries@gmail.com> * Remove outdated configure flags and some hacks * Add 2.0 versions of env variables * Remove PYTHONHOME from env info for now * Bump patch versions of 3.8/9/10 * Remove unused import * Fix running test package with spaces in path * Only link to ncursesw and tinfo Co-authored-by: Martin Valgur <martin.valgur@gmail.com> * Fix typo * Add context comments to most MSVC replace_in_files * Merge branch 'master' into update-cpython * Update to 3.11.9 * Add workaround for multithreaded install bug * Remove useless MSVC patch * Convert pkgconfig aliases to list --------- Co-authored-by: memsharded <james@conan.io> Co-authored-by: Martin Valgur <martin.valgur@gmail.com> Co-authored-by: Rubén Rincón Blanco <git@rinconblanco.es> Co-authored-by: Jordan Williams <jordan@jwillikers.com> Co-authored-by: Uilian Ries <uilianries@gmail.com>

there was an typo in conan-io#22599

…nd add 3.11/3.12 * wip * cpython: migrate to Conan v2 * cpython: fix autotools install step * cpython: fix libuuid requirement * cpython: update test_package * cpython: temporarily disable FindPythonX tests * cpython: package_type should be "library" * cpython: tidy * cpython: ncurses is required transitively * cpython: fix shared test * cpython: use cpp_info.aggregated_components() for deps * cpython: bsddb has been remove in Python 3 * cpython: fix openssl support * Fix package layout * Fix new generators in test package * Misc fixes * Remove redundant test * Add FIXMEs * Fix running test package multiple times in a row * Handle conan 1/2 differences in test_package * Fix MSBuild * Misc Conan 2.0 fixes * cpython: minor tweaks, bump deps * cpython: use altinstall Fixed modules standard library modules not being found in test_package. * Comment out versions with broken patches for now * Fix Autotools build * Restore old versions, temp remove 2.7.18 * Temp remove CPython 2.7.18 * Temp remove 2.7.18 * Misc MSVC fixes * Fix Windows/static build * Readd old test_package as test_v1_package * Add conanrun env to test package self.runs * Fix Linux shared build * Shared by default, don't delete compiler in package ID * Misc Mac fixes/cleanup * libxcrypt is transitive * Static by default for now * Attempt to fix 3.7 on Mac * Remove ssl module FIXME * self.settings.arch * Try using vendored libffi for mac on <3.9 * Fix PYTHONHOME on Linux, always set PYTHONHOME for test * Try vendored libffi on mac in debug mode * Remove "dynamic" check * Remove force on versions * Use Version() when comparing versions * Use a virtualrunenv in build, revert ffi changes * Bandaid fix for Linux * Use virtualrunenvs in v1 test package also * Skip 3.7 build on M1 macs * Set platform toolset directly on <3.8 * Fix issue with pip installed cmake * Python 2 compat in test_package.py * Re-add Python 2.7 + workarounds for MSVC * Misc pre-3.8 msvc fixes * Fix python 2 on Linux * Add 2.7.18 to config.yml * Remove includedirs for all components other than the main one * Use msvc_runtime_flag * Misc MSVC fixes * Skip building _freeze_importlib, fix 3.7.12 in msvc Debug MDd * Shot in the dark attempt at fixing Mac issue * Unconditionally generate VCVars in test package * Print config.log if autotools configure fails * Skip spam module test in 2.7 (debug, MDd) as well * Remove unnecessary removals from pcbuild.sln * Inject _bz2 programmatically * Also programattically patch bz2 in python 2.7 * Dynamically patch _elementtree, _ctypes, and _decimal projects * Patch pyexpat project programmatically * Inject libdb programmatically * Inject openssl into hashlib programmatically * Inject xz_utils programattically * Programattically inject sqlite * Programmatically inject openssl into _ssl project * Programmatically inject tk into _tkinter project * Programmatically inject zlib into pythoncore project * Remove/simplify redundant patching code * Minor cleanup * Refactor into _regex_replace_in_file * Refactoring of injection of conan props files * More refactoring for props file injecting * Programmatically remove parts of _ctypes project * Programmatically remove parts of _ctypes (patches) * Patch _decimal project programmatically * Remove sqlite project programmatically * Remove sqlite project programatically (patches) * Remove lzma project include programmatically * Remove openssl props import programmatically * Programmatically patch pyexpat project * Patch _elementtree project programmatically * Allow MSVC/Debug/dynamic with newer libffi * Test package cleanup * Fix Debug MSVC test package * Use test_package/test_package.py in test_v1_package * Fix conditional _d in debug mode * Add -Wl,--as-needed * Use extra_ldflags instead * Don't use -Wl,--as-needed on Apple OSs * Programmatically patch pythoncore project * Remove pythoncore patches * Remove unnecessary(?) patch * Programmatically patch _tkinter project * Programmatically patch _lzma project * Manually inject platform toolset on 3.8.x * Programmatically patch python project * Programmatically patch _freeze_importlib * Programmatically patch _ssl project, remove empty MSVC patches * Misc python 2 fixes * Programmatically patch setup py to skip tkinter detection * Programmatically patch ini CFLAGS and CPPFLAGS * Update 3.8.12 to 3.8.18 * Don't use use_2to3 after it is deprecated * Update 3.9.7 to 3.9.18 * Update 3.10.0 to 3.10.13 * Always manually specify the platform toolset * Fix MSVC static * Fix 3.7 patching * Update 3.7.12 to 3.7.17 * Add 3.11.7 (likely only MSVC works for now) * Add 3.12.1 (likely only working on Windows for now) * Update config.yml * Remove Py_SetProgramName (deprecated in 3.11) * Fix 3.12 on Linux * Fix 3.11 on Linux * Disable GCC 9 build * Cleanup test_v1_package * Only disable 3.12.1 GCC 9 build * Use pkgconfig tool requirement * Another shot in the dark attempt at fixing MacOS sporadic failure * Update to 3.12.2 and 3.11.8 * Update dependencies * Add patch descriptions * Enable short paths * Add link to nis system library * nis -> nsl * Add final newline * Simplify pkg-config info * Check for Linux or FreeBSD * Fix disabling sqlite3 (on Linux at least) * Simplify some file removing * rm unnecessary rm * Simplify pkg-config info * Check for Linux or FreeBSD * Fix disabling sqlite3 (on Linux at least) * Simplify some file removing * rm unnecessary rm * Disable GCC 9 build on 3.12 as a whole * Misc configure variable cleanup * Point test_v1_package to test_package sources * Fix lint warning * Diff cleanup * Conditionally add --with-system-ffi * Misc test package cleanup * Bump xz_utils * Bump xz_utils * Fix pymalloc argument * Don't read SSL config file Co-authored-by: Jordan Williams <jordan@jwillikers.com> * xcrypt is not transitive in newer versions * Bandaid fix for test package overflowing Windows max path length * Fix some lints * Remove EOL versions * Remove <3.8 logic * Fix more <3.8 logic * Small fix * Remove calculated Python major version * Use autotools.install() directly Co-authored-by: Uilian Ries <uilianries@gmail.com> * Remove outdated configure flags and some hacks * Add 2.0 versions of env variables * Remove PYTHONHOME from env info for now * Bump patch versions of 3.8/9/10 * Remove unused import * Fix running test package with spaces in path * Only link to ncursesw and tinfo Co-authored-by: Martin Valgur <martin.valgur@gmail.com> * Fix typo * Add context comments to most MSVC replace_in_files * Merge branch 'master' into update-cpython * Update to 3.11.9 * Add workaround for multithreaded install bug * Remove useless MSVC patch * Convert pkgconfig aliases to list --------- Co-authored-by: memsharded <james@conan.io> Co-authored-by: Martin Valgur <martin.valgur@gmail.com> Co-authored-by: Rubén Rincón Blanco <git@rinconblanco.es> Co-authored-by: Jordan Williams <jordan@jwillikers.com> Co-authored-by: Uilian Ries <uilianries@gmail.com>

there was an typo in #22599

there was an typo in conan-io#22599

memsharded and others added 30 commits October 5, 2023 11:30

wip

a3c8eb5

cpython: migrate to Conan v2

342f926

Merge branch 'migrate/cpython' into cpython

2f838f9

# Conflicts: # recipes/cpython/all/conanfile.py

cpython: fix autotools install step

8c8671a

cpython: fix libuuid requirement

c1c9f2e

cpython: update test_package

960811e

cpython: temporarily disable FindPythonX tests

522d5d9

cpython: package_type should be "library"

9c83e18

cpython: tidy

6c496e6

cpython: ncurses is required transitively

35fac91

cpython: fix shared test

ec99e2e

cpython: use cpp_info.aggregated_components() for deps

220c472

cpython: bsddb has been remove in Python 3

2e54e11

cpython: fix openssl support

51a3e11

Fix package layout

3cc7f61

Fix new generators in test package

1840f62

Misc fixes

d0bcea7

Remove redundant test

f0797ef

Add FIXMEs

c766334

Fix running test package multiple times in a row

6a1ebd9

Handle conan 1/2 differences in test_package

1539332

Fix MSBuild

1c2d2d1

Misc Conan 2.0 fixes

a665c98

cpython: minor tweaks, bump deps

b8a0b24

cpython: use altinstall

fcfd48d

Fixed modules standard library modules not being found in test_package.

Comment out versions with broken patches for now

c929422

Fix Autotools build

2f652f8

Restore old versions, temp remove 2.7.18

593edf5

Temp remove CPython 2.7.18

f2705c5

Temp remove 2.7.18

52be52c

This comment has been minimized.

Sign in to view

Remove useless MSVC patch

1aa229b

conan-center-bot added the Unexpected Error label Apr 4, 2024

This comment has been minimized.

Sign in to view

Merge branch 'master' into update-cpython

66ad791

conan-center-bot removed the Unexpected Error label Apr 4, 2024

This comment has been minimized.

Sign in to view

Convert pkgconfig aliases to list

aea1d14

conan-center-bot added Failed Missing dependencies Build failed due missing dependencies in Conan Center labels Apr 5, 2024

This comment has been minimized.

Sign in to view

ghost mentioned this pull request Apr 6, 2024

cpython: Add native support for CMake's builtin FindPython(3) #23394

Merged

3 tasks

Merge branch 'master' into update-cpython

99d30a8

conan-center-bot removed Failed Missing dependencies Build failed due missing dependencies in Conan Center labels Apr 6, 2024

valgur approved these changes Apr 6, 2024

View reviewed changes

AbrilRBS mentioned this pull request Apr 8, 2024

Check if it makes sense to guard against mistypes in certain properties conan-io/conan#16042

Closed

AbrilRBS self-assigned this Apr 9, 2024

AbrilRBS approved these changes Apr 9, 2024

View reviewed changes

conan-center-bot merged commit 0b768ea into conan-io:master Apr 9, 2024
33 checks passed

ericLemanissier reviewed Apr 9, 2024

View reviewed changes

ericLemanissier added a commit to ericLemanissier/conan-center-index that referenced this pull request Apr 15, 2024

cpython 3.11.9: re-apply patches

b96e13a

there was an typo in conan-io#22599

ericLemanissier mentioned this pull request Apr 15, 2024

cpython 3.11.9: re-apply patches #23538

Merged

3 tasks

conan-center-bot pushed a commit that referenced this pull request Apr 16, 2024

(#23538) cpython 3.11.9: re-apply patches

252e790

there was an typo in #22599

franramirez688 pushed a commit to toge/conan-center-index that referenced this pull request Apr 23, 2024

(conan-io#23538) cpython 3.11.9: re-apply patches

2d16329

there was an typo in conan-io#22599

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CPython: Condense patching, update patch versions, and add 3.11/3.12 #22599

CPython: Condense patching, update patch versions, and add 3.11/3.12 #22599

Ahajha commented Jan 30, 2024 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

valgur commented Apr 5, 2024 •

edited

Loading

Ahajha commented Apr 5, 2024

This comment has been minimized.

conan-center-bot commented Apr 6, 2024

AbrilRBS left a comment

ericLemanissier Apr 9, 2024

Ahajha Apr 9, 2024

ericLemanissier Apr 9, 2024

CPython: Condense patching, update patch versions, and add 3.11/3.12 #22599

CPython: Condense patching, update patch versions, and add 3.11/3.12 #22599

Conversation

Ahajha commented Jan 30, 2024 • edited Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

valgur commented Apr 5, 2024 • edited Loading

Ahajha commented Apr 5, 2024

This comment has been minimized.

conan-center-bot commented Apr 6, 2024

Conan v1 pipeline ✔️

Conan v2 pipeline ✔️

AbrilRBS left a comment

Choose a reason for hiding this comment

ericLemanissier Apr 9, 2024

Choose a reason for hiding this comment

Ahajha Apr 9, 2024

Choose a reason for hiding this comment

ericLemanissier Apr 9, 2024

Choose a reason for hiding this comment

Ahajha commented Jan 30, 2024 •

edited

Loading

valgur commented Apr 5, 2024 •

edited

Loading