Upgrade python to 3.11 #33842

mkoeppe · 2022-05-12T02:48:43Z

Issues

Depends on #32423
Depends on #33530
Depends on #33878
Depends on #34795

CC: @kiwifb @antonio-rojas @nbruin @dimpase @jhpalmieri

Component: packages: standard

Author: Gonzalo Tornaría, Matthias Koeppe, Andrey Belgorodski

Branch/Commit: ac0105e

Reviewer: Matthias Koeppe, Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/33842

The text was updated successfully, but these errors were encountered:

mkoeppe · 2022-05-12T03:00:31Z

Branch: u/mkoeppe/test_ticket__python_3_11

mkoeppe · 2022-05-12T03:06:00Z

comment:2

It builds. First failure: building cython - we need to wait for a version with 3.11 support - https://github.com/cython/cython/commits/0.29.x

New commits:

`3d3a5b7`	`build/pkgs/python3: Update to 3.11.0b1`
`a2b955e`	`build/pkgs/python3/patches/cygwin-socket-tcpnodelay-21649.patch: Remove, upstreamed`

mkoeppe · 2022-05-12T03:06:00Z

Commit: a2b955e

mkoeppe · 2022-05-18T00:20:01Z

comment:3

Cython update in #33864

mkoeppe · 2022-05-18T00:20:01Z

Dependencies: #33864

sagetrac-git · 2022-05-18T04:19:41Z

Changed commit from a2b955e to 694cab5

sagetrac-git · 2022-05-18T04:19:41Z

Branch pushed to git repo; I updated commit sha1. New commits:

`92e9cff`	`build/pkgs/cython: Update to 0.29.30`
`694cab5`	`Merge #33864`

mkoeppe · 2022-05-18T04:44:17Z

comment:6

  [pyzmq-22.3.0]       #include "longintrepr.h"
  [pyzmq-22.3.0]                ^~~~~~~~~~~~~~~
  [pyzmq-22.3.0]     1 error generated.

same error as in memory_allocator

mkoeppe · 2022-05-18T04:46:28Z

comment:7

Fix is in 23.0.0 (still beta) https://pyzmq.readthedocs.io/en/latest/changelog.html
https://pypi.org/project/pyzmq/#history

mkoeppe · 2022-05-18T04:47:30Z

comment:8

cypari2:

  gcc -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -g -O2 -g -O2 -I./cypari2 -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/site-packages/cysignals -I/usr/local/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/include/python3.11 -c cypari2/convert.c -o build/temp.macosx-12.4-x86_64-3.11/cypari2/convert.o
  cypari2/convert.c:3347:21: error: cannot take the address of an rvalue of type 'Py_ssize_t' (aka 'long')
    __pyx_v_sizeptr = &Py_SIZE(((PyObject *)__pyx_v_x));
                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1 error generated.

mkoeppe · 2022-05-18T04:50:46Z

Changed dependencies from #33864 to #33864, #32423

mkoeppe · 2022-05-18T04:57:47Z

comment:10

    File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/var/tmp/sage/build/jupyter_jsmol-0.2.4/src/setupbase.py", line 630, in _compile_pattern
      return re.compile(res, flags=flags).match
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/__init__.py", line 225, in compile
      return _compile(pattern, flags)
             ^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/__init__.py", line 273, in _compile
      p = _compiler.compile(pattern, flags)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/_compiler.py", line 759, in compile
      p = _parser.parse(p, flags)
          ^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/_parser.py", line 979, in parse
      p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/_parser.py", line 454, in _parse_sub
      itemsappend(_parse(source, state, verbose, nested + 1,
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/_parser.py", line 840, in _parse
      raise source.error('global flags not at the start '
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  re.error: global flags not at the start of the expression at position 52
  error: subprocess-exited-with-error

mkoeppe · 2022-05-18T05:49:55Z

Changed dependencies from #33864, #32423 to #33864, #32423, #33866

mkoeppe · 2022-05-18T05:51:30Z

Changed dependencies from #33864, #32423, #33866 to #33864, #32423, #33866, #33566

malb · 2022-05-18T10:28:46Z

comment:15

This should be fixed in FPyLLL upstream. I can cut new packages?

mkoeppe · 2022-05-18T15:20:05Z

comment:16

Yes please!

kliem · 2022-05-21T08:11:32Z

Changed dependencies from #33864, #32423, #33866, #33566 to #33864, #32423, #33866, #33566, #33872

sagetrac-git · 2022-05-27T05:03:55Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`7619755`	`Merge #32937`
`253a8f2`	`Merge tag '9.6.rc4' into t/32423/update_numpy_to_1_22_x__scipy_1_8_x___requires_dropping_python_3_7`
`d188705`	`Merge #33782`
`be053cb`	`Merge #32423`
`ce9a905`	`build/pkgs/jupyter_jsmol: Update to 2022.1.0`
`cea777c`	`build/pkgs/jupyter_packaging: Make it a normal standard package`
`731424b`	`Merge #33866`
`b75fd6e`	`test memory_allocator 0.1.3a1`
`542326c`	`Merge #33872`
`e61481e`	`build/pkgs/fpylll: Update to 0.5.7`

sagetrac-git · 2022-05-27T05:03:55Z

Changed commit from 694cab5 to e61481e

malb · 2022-05-27T15:40:58Z

comment:19

See #33913 for FP(y)LLL

mkoeppe · 2022-05-27T20:08:48Z

Changed dependencies from #33864, #32423, #33866, #33566, #33872 to #33864, #32423, #33866, #33566, #33872, #33878

mkoeppe · 2022-05-27T20:09:18Z

Changed dependencies from #33864, #32423, #33866, #33566, #33872, #33878 to #33864, #32423, #33866, #33566, #33872, #33878, #33913

sagetrac-git · 2022-05-27T21:30:54Z

Changed commit from e61481e to 39a71bc

vbraun · 2022-12-19T16:27:45Z

comment:139

I still get the

DeprecationWarning: Use setlocale(), getencoding() and getlocale() instead

on macOS Big Sur, so I guess it wasn't fixed in #34795

PS: Indeed the offending line is here site-packages/docutils/io.py:30

     _locale_encoding = locale.getlocale()[1] or locale.getdefaultlocale()[1]

PPS: Buildbot locale is:

buildbot-sage@bigsur-osx build % locale
LANG=""
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=
buildbot-sage@bigsur-osx build % ./sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.8.beta5, Release Date: 2022-12-11               │
│ Using Python 3.11.1. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: import locale
sage: locale.getlocale()
(None, 'UTF-8')

mkoeppe · 2022-12-19T19:20:34Z

comment:140

Thanks for the details. I think we'll have to suppress the warning. I'll also open an issue with docutils

mkoeppe · 2022-12-19T19:22:19Z

comment:141

locale.getlocale()[1] is UTF-8, so getdefaultlocale shouldn't be called there

mkoeppe · 2022-12-19T20:46:00Z

Changed branch from u/tornaria/py311 to u/mkoeppe/py311

mkoeppe · 2022-12-19T20:56:23Z

Changed commit from 3e86d91 to ac0105e

mkoeppe · 2022-12-19T20:56:23Z

New commits:

`ac0105e`	`Filter out locale DeprecationWarning`

mkoeppe · 2022-12-19T21:11:46Z

comment:144

Replying to Matthias Köppe:

I'll also open an issue with docutils

That's now https://sourceforge.net/p/docutils/bugs/464/

mkoeppe · 2022-12-19T23:09:38Z

comment:145

$ LC_ALL=C ./sage -tp --all --long 
...
All tests passed!

vbraun · 2023-01-05T23:16:59Z

Changed branch from u/mkoeppe/py311 to ac0105e

This is needed for python 3.11 support (sagemath#33842) and to make cypari compatible with current pari version https://groups.google.com/g/sage-devel/c/jqmr6bDjDrk/m/XE2GlB_GBgAJ Necessary follow ups: - make cypari thread safe: see [[https://github.com/sagemath/cypari2/pull/116|cypari2 sagemath#116]] - remove optional build time dependency of cysignals on pari: see [[https://github.com/sagemath/cypari2/pull/130|cypari2 sagemath#130]] URL: https://trac.sagemath.org/33878 Reported by: gh-kliem Ticket author(s): Matthias Koeppe Reviewer(s): Dima Pasechnik

This affects 32 bit architectures, where the representation of python integers changed in cpython 3.11. As part of sagemath#33842, the function `sage.arith.long.integer_check_long_py()` was rewritten to support the new representation. Unfortunately a bug remains for the conversion of integers between 2^60 and 2^63-1. This manifests in lots of doctests failing, but a quick way to demonstrate the issue is sage: ZZ ( int(1152921504606847018) ) # 2^60 + 42 42 The function `integer_check_long_py()` has good unit testing, checking values around the word size, but this range was missing. This commit adds a simple fix and new test cases for a few integers in this range.

This affects 32 bit architectures, where the representation of python integers changed in cpython 3.11, when compiled with gcc12. As part of sagemath#33842, the function `sage.arith.long.integer_check_long_py()` was rewritten to support the new representation. Unfortunately a bug remained that triggers UB for the conversion of integers between 2^60 and 2^63-1. Alas, the undesired behaviour does not happen with gcc10; it only started when I switched to gcc12. The bug manifests in lots of doctests failing, but a quick way to demonstrate the issue is sage: ZZ ( int(1152921504606847018) ) # 2^60 + 42 42 The function `integer_check_long_py()` has good unit testing, checking values around the word size, but this range was missing. This commit adds a simple fix and new test cases for a few integers in this range. Technical explanation: The UB is in the line cdef long lead_3_overflow = (<long>1) << (BITS_IN_LONG - 2 * PyLong_SHIFT) In our case we have `BITS_IN_LONG == 31` and `PyLong_SHIFT == 30` so the computed value is `<long>1 << -29` which is UB and it happens to evaluate to 0 with gcc10 but 8 with gcc12. The solution is to set the value to 0 when `BITS_IN_LONG < 2 * PyLong_SHIFT`.

….11, 32 bit) This affects 32 bit architectures, where the representation of python integers changed in cpython 3.11, when compiled with gcc12. As part of sagemath#33842, the function `sage.arith.long.integer_check_long_py()` was rewritten to support the new representation. Unfortunately a bug remained that triggers UB for the conversion of integers between 2^60 and 2^63-1. Alas, the undesired behaviour does not happen with gcc10; it only started when I switched to gcc12. The bug manifests in lots of doctests failing, but a quick way to demonstrate the issue is sage: ZZ ( int(1152921504606847018) ) # 2^60 + 42 42 The function `integer_check_long_py()` has good unit testing, checking values around the word size, but this range was missing. This commit adds a simple fix and new test cases for a few integers in this range. Technical explanation: The UB is in the line cdef long lead_3_overflow = (<long>1) << (BITS_IN_LONG - 2 * PyLong_SHIFT) In our case we have `BITS_IN_LONG == 31` and `PyLong_SHIFT == 30` so the computed value is `<long>1 << -29` which is UB and it happens to evaluate to 0 with gcc10 but 8 with gcc12. The solution is to set the value to 0 when `BITS_IN_LONG < 2 * PyLong_SHIFT`.

…on 3.11, 32 bit, gcc12) This affects 32 bit architectures, where the representation of python integers changed in cpython 3.11, when compiled with gcc12. As part of #33842, the function `sage.arith.long.integer_check_long_py()` was rewritten to support the new representation. Unfortunately a bug remained that triggers UB for the conversion of integers between 2^60 and 2^63-1. Alas, the undesired behaviour does not happen with gcc10; it only started when I switched to gcc12. The bug manifests in lots of doctests failing, but a quick way to demonstrate the issue is sage: ZZ ( int(1152921504606847018) ) # 2^60 + 42 42 The function `integer_check_long_py()` has good unit testing, checking values around the word size, but this range was missing. This commit adds a simple fix and new test cases for a few integers in this range. Technical explanation: The UB is in the line cdef long lead_3_overflow = (<long>1) << (BITS_IN_LONG - 2 * PyLong_SHIFT) In our case we have `BITS_IN_LONG == 31` and `PyLong_SHIFT == 30` so the computed value is `<long>1 << -29` which is UB and it happens to evaluate to 0 with gcc10 but 8 with gcc12. The solution is to set the value to 0 when `BITS_IN_LONG < 2 * PyLong_SHIFT` (which only happens for 32 bit python 3.11) --- TESTING: - With gcc10 the fix in #33842 was extensively tested (e.g. in void-linux/void-packages#41085) and everything seemed ok, that's why we missed this bug. - When using gcc 12, without this PR around 200 tests fail on 32 bit. - With this PR all tests pass as shown in https://github.com/void- linux/void-packages/pull/42048 (I'm actually testing sagemath-9.7 with python 3.11 support backported since that's easier for me, but it shouldn't make a difference). URL: #34997 Reported by: Gonzalo Tornaría Reviewer(s): Gonzalo Tornaría, Matthias Köppe

mkoeppe added this to the sage-9.7 milestone May 12, 2022

mkoeppe added c: packages: standard labels May 12, 2022

This comment has been minimized.

Sign in to view

vbraun added s: needs work and removed s: positive review labels Dec 19, 2022

mkoeppe added s: needs review and removed s: needs work labels Dec 19, 2022

mkoeppe added s: positive review and removed s: needs review labels Dec 19, 2022

vbraun removed the s: positive review label Jan 5, 2023

vbraun closed this as completed in 08aa2f8 Jan 5, 2023

This was referenced Jan 29, 2023

Migrate from setup.cfg to pyproject.toml #33577

Open

Update cython to 0.29.30 #33864

Closed

kliem mentioned this issue Nov 15, 2022

Upgrade cypari2 to 2.1.3 #33878

Closed

sheerluck mentioned this issue Jan 1, 2023

tox.ini: Add gentoo-python3.11 #34715

Closed

mezzarobba mentioned this issue Dec 15, 2022

sage-loaded libraries are slowing down cypari2 a lot (on x86_64 glibc) #34850

Closed

tornaria mentioned this issue Feb 7, 2023

Fix bug due to UB in conversion from python int to ZZ (python 3.11, 32 bit, gcc12) #34997

Merged

balejk mentioned this issue Sep 19, 2023

func_persist fails with Python 3.11 #36302

Closed

Upgrade python to 3.11 #33842

Upgrade python to 3.11 #33842

Comments

mkoeppe commented May 12, 2022

mkoeppe commented May 12, 2022

mkoeppe commented May 12, 2022

mkoeppe commented May 12, 2022

mkoeppe commented May 18, 2022

mkoeppe commented May 18, 2022

sagetrac-git mannequin commented May 18, 2022

sagetrac-git mannequin commented May 18, 2022

This comment has been minimized.

mkoeppe commented May 18, 2022

mkoeppe commented May 18, 2022

mkoeppe commented May 18, 2022

mkoeppe commented May 18, 2022

mkoeppe commented May 18, 2022

This comment has been minimized.

This comment has been minimized.

mkoeppe commented May 18, 2022

mkoeppe commented May 18, 2022

malb commented May 18, 2022

mkoeppe commented May 18, 2022

kliem commented May 21, 2022

sagetrac-git mannequin commented May 27, 2022

sagetrac-git mannequin commented May 27, 2022

malb commented May 27, 2022

mkoeppe commented May 27, 2022

mkoeppe commented May 27, 2022

sagetrac-git mannequin commented May 27, 2022

vbraun commented Dec 19, 2022

mkoeppe commented Dec 19, 2022

mkoeppe commented Dec 19, 2022

mkoeppe commented Dec 19, 2022

mkoeppe commented Dec 19, 2022

mkoeppe commented Dec 19, 2022

mkoeppe commented Dec 19, 2022

mkoeppe commented Dec 19, 2022

vbraun commented Jan 5, 2023