Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to 3.11.8 #166

Merged
merged 150 commits into from
Feb 12, 2024
Merged

Update to 3.11.8 #166

merged 150 commits into from
Feb 12, 2024

Conversation

lazka
Copy link
Member

@lazka lazka commented Feb 10, 2024

Summary:

  • simple rebase, no conflicts
  • backported two commits to fix tests with newer expat

Range diff:

  1:  fe20eea853e =   1:  a2ad3cbb958 sysconfig: make _sysconfigdata.py relocatable
  2:  f3ea27e23da =   2:  acaa4c342a6 build: add --with-nt-threads and make it default on mingw
  3:  6374874d961 =   3:  fd4219a19e8 Define `MS_WINDOWS` (and others) when compiling with MINGW
  4:  073f28a8849 =   4:  71b290bc854 configure: add `MACHDEP` and `platform` on MINGW
  5:  7f543212c2d =   5:  89f585d66a3 Add default configuration for MINGW
  6:  aacb55a802e =   6:  91f38ff7e3a configure: enable largefile support by default for Mingw
  7:  19fd8716754 =   7:  462619920c8 Add `PC/` to `CPPFLAGS` and to `SRCDIRS` on Mingw
  8:  357802d21d3 =   8:  8ea277bd020 posixmodule.c: Define necessary headers for compiling on MINGW
  9:  1a9d7c2e830 =   9:  651bdc8d367 Build winreg by default when compiling for MINGW
 10:  fa7a3f421f3 =  10:  60bb66f8abd pyport.h: Add support for MINGW
 11:  7437c3503a5 =  11:  4a8290d6394 configure: add options so that shared build is possible on MINGW
 12:  e37c5e3e538 =  12:  fac9879b9e9 Add dynamic loading support for MINGW
 13:  a1a6ca26243 =  13:  128bab0e3fc Ignore main program for frozen scripts on MINGW
 14:  763705bb731 =  14:  b2cd359bfa8 Add missing library for multiprocessing module
 15:  ef9dfd10a5a =  15:  feb446920b0 selectmodule: add support for MINGW
 16:  7751af9706d =  16:  aa52bdb29f1 setup.py: add libraries to fix compile of ctypes on MINGW
 17:  3f0202062f1 =  17:  a693d6fc118 Detect winsock2 and setup _socket module on MINGW
 18:  cd75eb028a4 !  18:  043c36effbc Enable some windows specific modules
    @@ Modules/Setup.bootstrap.in: _symtable symtablemodule.c
      ## Modules/_winapi.c ##
     @@
      
    - #define WINDOWS_LEAN_AND_MEAN
    + 
      #include "windows.h"
     +#if defined(Py_DEBUG)
      #include <crtdbg.h>
     +#endif
      #include "winreparse.h"
      
    - #if defined(MS_WIN32) && !defined(MS_WIN64)
    + #include "pycore_runtime.h"       // _Py_ID
     
      ## PC/msvcrtmodule.c ##
     @@
 19:  91d271f29a0 =  19:  cb824dbeffa sysconfig: MINGW build extensions with GCC
 20:  7c461184824 =  20:  40c137bbf60 sysconfig: treat MINGW builds as POSIX builds
 21:  baa479921eb =  21:  8ac9f0ce887 Add support for stdcall without underscore
 22:  51bad16c9c3 =  22:  c3eabfc5cec Use replace instead rename to avoid failure on windows
 23:  293fdc0f74e =  23:  a2d40f585eb Customize site for MINGW
 24:  1d46be0cac2 =  24:  f3eb6dcb2dd add python config sh
 25:  179aa9819da =  25:  657cda68400 setup.py: add export_symbols to fix compile of ctypes
 26:  57ee999cbf5 =  26:  6ceadb161bb mingw: prefer unix sep if MSYSTEM environment variable
 27:  00549ca1c7d =  27:  e085d6087b7 msys cygwin semi native build sysconfig
 28:  7f7283e7721 =  28:  b7550c22822 sysconfig: mingw sysconfig like posix
 29:  99ee8721f4b =  29:  f83e1f4f064 cross dont add multiarch paths if cross compiling
 30:  7261fea506b =  30:  cd2d17ed618 mingw use backslashes in compileall py
 31:  9b6c0817399 =  31:  654bf8e5a9c msys convert_path fix and root hack
 32:  32b764f1f79 =  32:  99318d1ae84 mingw pdcurses_ISPAD
 33:  243520b12a6 =  33:  9332fd8f6bd msys monkeypatch os system via sh exe
 34:  456cd176775 =  34:  15ea8ec05a9 msys replace slashes used in io redirection
 35:  bb086a7dcbe =  35:  43a23b72feb remove_path_max.default
 36:  841c2bad0b6 =  36:  31583b9601e dont link with gettext
 37:  0e904afcfeb =  37:  22b93f238c0 ctypes python dll
 38:  526a08f049b =  38:  afd21e99620 gdbm module includes
 39:  a35e31c35a8 =  39:  7b52a7b6646 use gnu_printf in format
 40:  81855115b6d =  40:  cede6f74a37 mingw fix ssl dont use enum_certificates
 41:  bacf3698956 =  41:  38547013f3f fix using dllhandle and winver mingw
 42:  9456dda9758 =  42:  ae904178200 Add AMD64 to sys config so msvccompiler get_build_version works
 43:  a630faa5083 =  43:  ea2fbf10d92 MINGW link with additional library
 44:  c9bf503a26d =  44:  7c739a3b206 install msilib
 45:  9cc260d1aae =  45:  d365cd8e217 fix signal module build
 46:  c0d205bb5e8 =  46:  fef2080bf75 build: build winconsoleio and _testconsole
 47:  56ebea36b00 =  47:  df8e498449e expose sem_unlink
 48:  5b7a41cb9a5 =  48:  95a83657b23 cygpty isatty
 49:  abf66be2ae7 =  49:  db9d6fa0ddc disable broken gdbm module
 50:  22dc87190c8 =  50:  2931e5e3b18 build: link win resource files and build pythonw
 51:  50ff46bc6d1 =  51:  47fe818612c disable readline
 52:  cecd418d270 =  52:  3e4922b85ab fix isselectable
 53:  99d570a5af5 =  53:  cba4c92eb9a _xxsubinterpretersmodule
 54:  267ef99ad14 =  54:  4360b847b3c configure.ac: fix inet_pton check
 55:  28b231da4e1 =  55:  f2f4a92c179 set venv activate path unix
 56:  329ff8f7f0a =  56:  d489452b826 pass gen profile ldflags
 57:  9f03dae352b =  57:  34fe0bf83ce pkg config windows must link ext with python lib
 58:  8e87b7090ae =  58:  95bae483a4a importlib bootstrap path sep
 59:  fabf68e75b1 =  59:  4fa99654cc4 pathlib path sep
 60:  eda7cac5df7 =  60:  f42398d96e6 warnings fixes
 61:  87df2d6daf3 =  61:  673f82179e8 fix build testinternalcapi
 62:  8080653a191 =  62:  f6c18e2f0f6 extend MS_WINDOWS flag
 63:  3e345b69ce6 =  63:  eff7a180fbf clang arm64
 64:  baa055d934f =  64:  0e664d3bbec configure.ac: set MINGW stack reserve
 65:  c840332948c =  65:  3ff0332a036 Don't use os.pathsep to find EOF not all distributions in win32 have them as \ instead check using sys.platform
 66:  8481e577bd6 =  66:  15c7a16884c Fix extension suffix for c-extensions on mingw
 67:  75386fda789 =  67:  08b174ac4de Change the `get_platform()` method in sysconfig
 68:  4a3002f16df =  68:  ed856e6994f distutils: compiler customize mingw cygwin compilers
 69:  433865e04b5 =  69:  4a1811f3a53 distutils: compiler enable new dtags
 70:  ee2e016ef51 =  70:  4883014ceaa distutils: MINGW build extensions with GCC
 71:  01ee3df6d1d =  71:  15ca0f93dcc distutils: use Mingw32CCompiler as default compiler for m
 72:  e409b5e6fa9 =  72:  b7ff8e7e922 distutils: find import library
 73:  e883501fe44 =  73:  30cb3c32e86 distutils: avoid circular dependency from time module dur
 74:  9a31377c3f1 =  74:  5d0c7d750f4 distutils: generalization of posix build in distutils sys
 75:  7a3ad9211f0 =  75:  906096e7322 distutils: mingw sysconfig like posix
 76:  a45c7e4baa6 =  76:  a15cff8cc5e distutils: get_versions() fixes
 77:  1aabba596a6 =  77:  34ca829abd2 distutils: install layout as posix
 78:  0602c798eaf =  78:  42075ece550 distutils: msys convert_path fix and root hack
 79:  4609ae175be =  79:  9affd212d76 distutils: mingw build optimized ext
 80:  d98c7623e60 =  80:  16809e6589e distutils: cygwinccompiler dont strip modules if pydebug
 81:  4eed41aa5bb =  81:  f1446fffc3e distutils: get compilers from env vars
 82:  1cf52d0b461 =  82:  95d2dc2c252 distutils: add windmc to cygwinccompiler
 83:  d70f9c2be18 =  83:  4d99ce14f5f distutils: fix msvc9 import
 84:  bce89b06c75 =  84:  2ce9c32b971 distutils: mingw add LIBPL to library dirs
 85:  5b6e2c201f4 =  85:  fa05bbc5477 distutils: Change the `get_platform()` method in distutils to match sysconfig
 86:  acdbc6426ce =  86:  9999f8554f5 build: Fix ncursesw include lookup
 87:  29cd5213df0 =  87:  324a0565c58 tests: fix test_bytes
 88:  c66da8a1d1f =  88:  5863c875add time: fix strftime not raising for invalid year values
 89:  db3895a673a =  89:  5dea41c9a81 ctypes: find_library('c') should return None with ucrt
 90:  b4b24970f09 =  90:  e8d4d2ee221 build: Disable checks for dlopen/dlfcn
 91:  355cfb5f8a2 =  91:  17fc98c3d4f Fix install location of the import library
 92:  f379d09ce88 =  92:  3e535dcdfa0 Set MSYS2_ARG_CONV_EXCL for the shared Python module install
 93:  2acf0cb5d1f =  93:  9bd8927ee0b build: Integrate venvlauncher build/installation into the Makefile
 94:  ad8c2f7363e =  94:  a5675f6bfba configure.ac: set _WIN32_WINNT version
 95:  b0d04bf4ee4 =  95:  b8526b7b2ea configure.ac: don't check for clock_ functions
 96:  b46f22d73d0 =  96:  74977124656 expanduser: normpath paths coming from env vars
 97:  c23c2929e23 =  97:  bea5e16fa38 CI: test the build and add some mingw specific tests
 98:  f8e92a0adee =  98:  38b67fde1b9 Prefer sysconfig.python_build
 99:  cbff90370b1 =  99:  c710f9db5b8 Define PY3_DLLNAME to fix build
100:  e9b1c205757 = 100:  59319ba70a9 distutils: remove checks for ancient gcc/binutils
101:  c2485a671c9 = 101:  3193430061b distutils: split CC env var before passing to subprocess
102:  de3bac1cdb2 = 102:  a5006d71f96 _testconsole.c: Fix casing & path sep
103:  bd96451e7e1 = 103:  75db7d09035 Return consistent architecture markers for python on mingw/armv7
104:  ddb914bb0e2 = 104:  cb77687ec0c distutils: add back gcc_version
105:  b6d28249d8e = 105:  82dba0aceb0 fix mingw cross compiling in setup.py
106:  5291f1f997f = 106:  cb6281b4e80 handle ncursesw pkg-config when cross-compiling
107:  295c5a5df78 = 107:  1f4cc0cedba mingw_smoketests: fix _UCRT condition
108:  50ee859784b = 108:  a3db60f655b Modify `sys.winver` to match upstream
109:  449580a3d26 = 109:  5169bbfb6b7 Change user site-packages path to include the environment info
110:  1bbe27b4373 = 110:  220af2c5ed2 configure: Include a header in the check for _beginthread
111:  72187a29bf8 = 111:  bc5cc7ad191 configure.ac: Default to --without-c-locale-coercion on Windows
112:  a1e3dc56f13 = 112:  302a10a36d9 Fix failing tests
113:  4ee7b56c5b3 = 113:  2b26f7bc328 distutils: add build root to libdirs when building uninstalled
114:  241ece20a8f = 114:  5690139aad8 Don't change os.sep with an empty MSYSTEM env var, not just a missing one
115:  6e2e92dd38f = 115:  cc077df4156 def VPATH when compiling `Python/sysmodule.c`
116:  86f41b1da29 = 116:  17c5ba89f2c Make `_Py_CheckPython3` extern
117:  b04f8e97e4f = 117:  31e025bfac1 link with bcrypt
118:  42abc46dc0f = 118:  2f9ce9bfa66 correctly find native python
119:  259f414d163 = 119:  39f585d2259 Add extra flags for `_bootstrap_python`
120:  d646a2163f7 = 120:  b052d3d57b9 posixmodule: undefine `HAVE_OPENDIR`
121:  1a3b03ffc91 = 121:  95dfa9be803 getpath.py: add support for mingw
122:  23135831230 = 122:  ecda02f0cd2 Don't  build _posixsubprocess on Windows.
123:  c24e7f3f3d2 = 123:  7eb4f13d255 `_ssl`: link with `ws2_32`
124:  76a66bfa4fd = 124:  c074aa0d35c Fix building `_socket` module
125:  0b382255c72 = 125:  68a709d83f7 Always normalize path in abspath
126:  de32428995b = 126:  f51799756ee Include `winsock.h` when checking for netdb function
127:  24f48fa7934 = 127:  64a80361d7c include `_multiprocessing/semaphore.c` on win32
128:  60f3f604b9f = 128:  ccaa6390d1e configure: build `mmap` module on win32
129:  497dd111b0c = 129:  913f96892a9 venv creation fixes
130:  cbb3459aee2 = 130:  9d778084eba move the `shutdown` function where `winsock.h` is included
131:  7fd6d8f55f9 = 131:  e870348e66f configure.ac: set BUILDEXEEXT and EXEEXT
132:  78b926ddf17 = 132:  99f8b4d4808 configure.ac: fix building some test modules
133:  57ec7a05bfd = 133:  44cfe487f19 Don't convert `sysconfig.get_config_var('VPATH')` to an absolute path
134:  776cae8f45c = 134:  fc8e861f2d5 Always convert `/` to `\\` before passing though pathcch functions
135:  be75cd9acbd = 135:  2fb6fa7f2c4 Build venvlauncher.exe from PC/launcher.c
136:  591b618b83d = 136:  0e9c56926f3 getpath.py: fix `dirname`
137:  b8e2990102d = 137:  6c4d3976690 getpath: use normpath on all generated paths
138:  c42816af8ba = 138:  62ff7f87738 pathconfig: normpath sys.path[0]
139:  765cca39b59 = 139:  3c6ec8cda73 smoketests: add some tests for sys/site paths
140:  c8c88382c3a = 140:  89cb8c62e6f Search DLLs only on paths added using `add_dll_directory()`.
141:  725d64aee79 = 141:  a6873f5def8 Allow picking up include/lib dirs from CFLAGS/LDFLAGS for Windows targets
142:  0043d5c9a37 = 142:  4e036ad9c28 Build and install libpython3.dll
143:  4d02ff37bfa = 143:  b1c500f47a5 setup.py: don't prepend the system library directories when building extensions
144:  1bee2249513 = 144:  5116779fd58 Port GetPythonImport() to mingw
145:  a632e9dd5c2 = 145:  b6995a54d87 LoadLibraryExW: make sure to only use backslashes for paths
146:  a9dcb699050 = 146:  8c92268ea60 Use `-shared` instead of `-mdll`
147:  8d66d4565df = 147:  f79c22b4c42 mingw_smoketests: add a test to check if sysconfig returns correct values for `LIBDIR`
148:  ebc5f3fc250 = 148:  67a024edc50 Makefile: Add a dependency on $(LDLIBRARY) for the sharedmods target
  -:  ----------- > 149:  daaa57f932e gh-115133: Fix tests for XMLPullParser with Expat 2.6.0
  -:  ----------- > 150:  dd5baba7d46 Update Lib/test/test_xml_etree.py

mingwandroid and others added 30 commits February 10, 2024 13:12
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: lovetox <8661218+lovetox@users.noreply.github.com>
Co-authored-by: Naveen M K <naveen521kk@gmail.com>
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: Christoph Reiter <reiter.christoph@gmail.com>
This is used throughout the Python code but is not defined
when compiling with Mingw-w64 compilers, so define it.

Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: Christoph Reiter <reiter.christoph@gmail.com>
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: Christoph Reiter <reiter.christoph@gmail.com>
Largefile is supported on Windows.

Co-authored-by: Naveen M K <naveen521kk@gmail.com>
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: Christoph Reiter <reiter.christoph@gmail.com>
Co-authored-by: Naveen M K <naveen521kk@gmail.com>
Also, build `nt` module instead of `posix` when compiling
with MINGW.

Co-authored-by: Naveen M K <naveen521kk@gmail.com>
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: Christoph Reiter <reiter.christoph@gmail.com>
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: Naveen M K <naveen521kk@gmail.com>
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
`ws2_32` is required when building that module
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
This is the case used in MINGW

Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: Christoph Reiter <reiter.christoph@gmail.com>
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: Christoph Reiter <reiter.christoph@gmail.com>
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: Christoph Reiter <reiter.christoph@gmail.com>
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: Christoph Reiter <reiter.christoph@gmail.com>
Co-authored-by: cat <cat@wolfgirl.org>
Co-authored-by: Naveen M K <naveen521kk@gmail.com>
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
Co-authored-by: Алексей <alexey.pawlow@gmail.com>
naveen521kk and others added 19 commits February 10, 2024 13:12
they are guarded by dlopen being present, but if module loading is enabled
is more correct.
they don't seems to handle `/` as path separator correctly
- Use function available in original msvcrt.dll
  _wdupenv_s -> _wgetenv_s
  fread_s -> fread
- Add a test for checking the new launchers
also, fix finding prefix when in a venv
Instead of just calling normpath in abspath just call normpath
on all the config results. This makes sure we don't change getpath.py
too much and still cover all outputs.

This fixes sys.exec_prefix not matching sys.prefix, see
#142
this gets added as first element in sys.path.
use normpath to make sure it uses the right path separator
This is the default behavior on upstream Python. We previously
reverted this behavior because it broke some use cases.

The old behavior can be restored by setting the environment variable
PYTHONLEGACYWINDOWSDLLLOADING to 1.

Fixes #48
Also fixes #141

Signed-off-by: Naveen M K <naveen521kk@gmail.com>
…gets

In cpython-mingw-3.10, this was part of a patch named "allow static
tcltk".

Co-authored-by: Алексей <alexey.pawlow@gmail.com>
This provides the limited ABI as a separate DLL, forwarding to the
real one. This makes linking with "-lpython3" work.

Fixes #147
…extensions

For some reason the code assumes that relative paths are build paths prepends
the system dirs after relative paths in the existing list. We only uses absolute
paths though, and I don't understand why it even tries to do that.

So just append the system dirs instead.

This fixes the build trying to link against the system Python DLL in case it happens
to be installed.
This looks for DLL names in the import table but while with MSVC the DLL
is named python311.dll in our case it is named libpython3.11.dll.
Adjust the strings and lengths accordingly.
It seems like in case the path passed to it is absolute, but contains
forward slashes then LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR does not work
and DLLs in the same directory as the extension are not considered.

This occurs in our fork because in MSYS2-mode the extension loader will
normalize to forward slashes before.

Normalize everything to backslashes again before passing it to LoadLibraryExW.

Fixes #151
When building shared modules (*.pyd) for MinGW, they link against
the import library for the main python (e.g. -lpython3.11). When
linking shared modules on Linux, this doesn't happen, but those
shared libraries are linked with undefined references which
then are fulfilled by the host process when they are loaded.

Add a dependency to make sure that $(LDLIBRARY), e.g.
libpython$(LDVERSION).dll.a, exists before starting to build
the shared modules.

Previously, if libpython$(LDVERSION).dll.a didn't exist when
the shared modules were linked, some of them failed to link,
but this wasn't reported as an error in the build, and a
later invocation of setup.py would retry building them, which
then would succeed. E.g. there was a seemingly benign race
condition in the build.

However, in some rare cases, the race condition no longer
was benign. The build also produces a corresponding static library,
$(LIBRARY), e.g. libpython$(VERSION)$(ABIFLAGS).a - which in
the end would be named similarly, libpython3.11.a, when
$(LDLIBRARY) would be libpython3.11.dll.a.

If the static library exists but the import library doesn't, then
most modules would still fail to link, but a few ones wouldn't
(select, _multiprocessing, _overlapped and _socket). The ones
that did succeed linking with the static library would crash at
runtime though.

By making sure that the right, intended library to fulfill the
linker parameter "-lpython3.11" exists before building the shared
modules, we avoid the whole race condition that in some rare
cases could produce a Python build that crashes at runtime.

For Linux targets, this extra dependency is unnecessary, but should
not cause other problems ($(LDLIBRARY) is set to the same as
$(LIBRARY) if not building any shared library).

This fixes MSYS2 cpython-mingw issue #162.
@lazka
Copy link
Member Author

lazka commented Feb 10, 2024

Tests fail due to python#115133

Feeding the parser by too small chunks defers parsing to prevent
CVE-2023-52425. Future versions of Expat may be more reactive.
@lazka lazka marked this pull request as ready for review February 11, 2024 08:39
Copy link
Member

@naveen521kk naveen521kk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@lazka lazka merged commit 9d73243 into mingw-v3.11.8 Feb 12, 2024
26 checks passed
@lazka lazka deleted the wip-mingw-v3.11.8 branch February 12, 2024 17:23
@lazka
Copy link
Member Author

lazka commented Feb 12, 2024

thanks!

lazka added a commit to lazka/MINGW-packages that referenced this pull request Feb 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.