Meta-ticket: Cython 3.0.0 (released July 2023) #29863

mkoeppe · 2020-06-15T00:21:25Z

This ticket is for experimenting with an update to Cython 3, which entered the beta stage on 2023-02-25 after several years in alpha. (No release schedule has been published.)

https://mail.python.org/pipermail/cython-devel/2019-November/thread.html (2019 - no response)
https://groups.google.com/g/cython-users/c/WTHmGJ44aB0 (2022-01 "no concrete time plan")
3.0.0 roadmap? cython/cython#4022 (still no road map)

https://github.com/cython/cython/blob/master/CHANGES.rst

Add missing cdef method declarations (for Cython 3) #34257 Add missing cdef method declarations (for Cython 3)
Use Cython directive binding=True to get signatures for cython methods #26254 Use Cython directive binding=True to get signatures in help for cythonized built-in methods
cython: Eliminate use of custom patches #32832 cython: Eliminate use of custom patches
Type hints (__annotations__) and coercion/categories/Cython #30511 Type hints (__annotations__) and coercion/categories/Cython

See also:

Backport PEP420 namespace package support from Cython 3 #34221 / Backport to 0.29.x: Add PEP420 namespace support cython/cython#4918 (Backport to 0.29.x: Add PEP420 namespace support)

Depends on #26254
Depends on #32832
Depends on #34257

CC: @kwankyu @williamstein @videlec @tscrim @tobiasdiez

Component: packages: standard

Keywords: upgrade

Branch/Commit: u/mkoeppe/meta_ticket__cython_3_0_0__unreleased_ @ 6c338f8

Issue created by migration from https://trac.sagemath.org/ticket/29863

The text was updated successfully, but these errors were encountered:

slel · 2020-06-18T02:57:30Z

Changed keywords from none to upgrade

mkoeppe · 2021-02-13T20:51:01Z

comment:3

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

mkoeppe · 2022-07-25T02:23:20Z

Dependencies: #26254, #32832

mkoeppe · 2022-07-25T02:37:41Z

Branch: u/mkoeppe/meta_ticket__cython_3_0_0__unreleased_

mkoeppe · 2022-07-25T02:38:35Z

Commit: 02ce997

mkoeppe · 2022-07-25T02:38:35Z

comment:13

pplpy build fails:

    tree = Parsing.p_module(s, pxd, full_module_name)

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
  from ppl_decl cimport *
  ^
  ------------------------------------------------------------

  ppl/bit_arrays.pxd:1:0: 'ppl_decl.pxd' not found

  Error compiling Cython file:

Last 10 new commits:

`bea228b`	`Try with localized import`
`89b8081`	`Inline all other imports from expression`
`9bbf574`	`Fix a few tests`
`803804f`	`and one more test`
`a23ffe9`	`Tiny edits`
`570058c`	`Merge remote-tracking branch 'origin/u/mkoeppe/update_cython_to_0_29_29' into public/26254`
`02d6282`	`Merge remote-tracking branch 'origin/develop' into public/26254`
`bca67b0`	`Merge #26254`
`f537175`	`build/pkgs/cython: Use 3.0.0a10`
`02ce997`	`build/pkgs/cython/patches/trashcan.patch: Remove`

mkoeppe · 2022-07-25T02:59:21Z

comment:14

[sagelib-9.7.beta6]   Error compiling Cython file:
[sagelib-9.7.beta6]   ------------------------------------------------------------
[sagelib-9.7.beta6]   ...
[sagelib-9.7.beta6]               sage: p = next_prime(2^32)
[sagelib-9.7.beta6]               sage: GF(p)(int(p+1))
[sagelib-9.7.beta6]               1
[sagelib-9.7.beta6]           """
[sagelib-9.7.beta6]           mpz_set_si(self.value, value)
[sagelib-9.7.beta6]           mpz_mod(self.value, self.value, self.__modulus.sageInteger.value)
[sagelib-9.7.beta6]                                                                    ^
[sagelib-9.7.beta6]   ------------------------------------------------------------
[sagelib-9.7.beta6] 
[sagelib-9.7.beta6]   sage/rings/finite_rings/integer_mod.pyx:1986:66: Cannot convert Python object to '__mpz_struct *'

mkoeppe · 2022-07-25T03:00:00Z

comment:15

[sagelib-9.7.beta6]   Error compiling Cython file:
[sagelib-9.7.beta6]   ------------------------------------------------------------
[sagelib-9.7.beta6]   ...
[sagelib-9.7.beta6]           """
[sagelib-9.7.beta6]           Lfunction.__init__(self, name, what_type_L, dirichlet_coefficient,  period, Q, OMEGA, gamma,lambd, pole,residue)
[sagelib-9.7.beta6]           self._repr += " with integer Dirichlet coefficients"
[sagelib-9.7.beta6] 
[sagelib-9.7.beta6]       # override
[sagelib-9.7.beta6]       cdef void __init_fun(self, char *NAME, int what_type, dirichlet_coeff, long long Period, double q,  c_Complex w, int A, double *g, c_Complex *l, int n_poles, c_Complex *p, c_Complex *r):
[sagelib-9.7.beta6]           ^
[sagelib-9.7.beta6]   ------------------------------------------------------------
[sagelib-9.7.beta6] 
[sagelib-9.7.beta6]   sage/libs/lcalc/lcalc_Lfunction.pyx:500:9: C method '_Lfunction_I__init_fun' not previously declared in definition part of extension type 'Lfunction_I'

mkoeppe · 2022-07-25T03:01:49Z

comment:16

[sagelib-9.7.beta6]   Error compiling Cython file:
[sagelib-9.7.beta6]   ------------------------------------------------------------
[sagelib-9.7.beta6]   ...
[sagelib-9.7.beta6]           """
[sagelib-9.7.beta6]           return Matroid._repr_(self) + " with " + str(self.bases_count()) + " bases"
[sagelib-9.7.beta6] 
[sagelib-9.7.beta6]       # support for parent BasisExchangeMatroid
[sagelib-9.7.beta6] 
[sagelib-9.7.beta6]       cdef bint __is_exchange_pair(self, long x, long y) except -1:      # test if current_basis-x + y is a basis
[sagelib-9.7.beta6]           ^
[sagelib-9.7.beta6]   ------------------------------------------------------------
[sagelib-9.7.beta6] 
[sagelib-9.7.beta6]   sage/matroids/basis_matroid.pyx:268:9: C method '_BasisMatroid__is_exchange_pair' not previously declared in definition part of extension type 'BasisMatroid'

[sagelib-9.7.beta6]   Error compiling Cython file:
[sagelib-9.7.beta6]   ------------------------------------------------------------
[sagelib-9.7.beta6]   ...
[sagelib-9.7.beta6]                   b = frozenset(B)
[sagelib-9.7.beta6]                   if len(b) != self._matroid_rank:
[sagelib-9.7.beta6]                       raise ValueError("basis has wrong cardinality.")
[sagelib-9.7.beta6]                   if not b.issubset(self._groundset):
[sagelib-9.7.beta6]                       raise ValueError("basis is not a subset of the groundset")
[sagelib-9.7.beta6]                   self.__pack(self._b, b)
[sagelib-9.7.beta6]                                  ^
[sagelib-9.7.beta6]   ------------------------------------------------------------
[sagelib-9.7.beta6] 
[sagelib-9.7.beta6]   sage/matroids/basis_matroid.pyx:224:32: Cannot convert 'bitset_t' to Python object
[sagelib-9.7.beta6]

mkoeppe · 2022-07-25T03:02:18Z

comment:17

[sagelib-9.7.beta6]   Error compiling Cython file:
[sagelib-9.7.beta6]   ------------------------------------------------------------
[sagelib-9.7.beta6]   ...
[sagelib-9.7.beta6] 
[sagelib-9.7.beta6]       AUTHOR:
[sagelib-9.7.beta6] 
[sagelib-9.7.beta6]       - Xavier Caruso (2013-03)
[sagelib-9.7.beta6]       """
[sagelib-9.7.beta6]       cdef int __normalize(self) except -1:
[sagelib-9.7.beta6]           ^
[sagelib-9.7.beta6]   ------------------------------------------------------------
[sagelib-9.7.beta6] 
[sagelib-9.7.beta6]   sage/rings/polynomial/polynomial_element.pyx:11912:9: C method '_Polynomial_generic_dense_inexact__normalize' not previously declared in definition part of extension type 'Polynomial_generic_dense_inexact'

sagetrac-git · 2022-07-25T04:32:21Z

Changed commit from 4e14632 to 46af90e

sagetrac-git · 2022-07-25T04:32:21Z

Branch pushed to git repo; I updated commit sha1. New commits:

`46af90e`	`src/sage/rings/polynomial/polynomial_element.pxd: Add missing cdef method declaration`

mkoeppe · 2022-07-25T04:58:15Z

comment:21

Help wanted with Cannot convert 'bitset_t' to Python object and Cannot convert Python object to '__mpz_struct *'

mkoeppe · 2022-07-31T19:33:57Z

Changed dependencies from #26254, #32832 to #26254, #32832, #34257

sagetrac-git · 2022-07-31T19:54:01Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`1f9ee80`	`src/sage/libs/lcalc/lcalc_Lfunction.pxd: Add missing cdef method declarations`
`034094c`	`src/sage/matroids/{basis,linear}_matroid.pxd: Add missing cdef method declarations`
`9d89bdf`	`src/sage/rings/polynomial/polynomial_element.pxd: Add missing cdef method declaration`
`727d101`	`Merge #34257`
`6c338f8`	`build/pkgs/pplpy/patches: Add https://gitlab.com/videlec/pplpy/-/merge_requests/20`

sagetrac-git · 2022-07-31T19:54:01Z

Changed commit from 46af90e to 6c338f8

mkoeppe · 2022-07-31T19:54:44Z

comment:25

Replying to @mkoeppe:

pplpy build fails:

    tree = Parsing.p_module(s, pxd, full_module_name)

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
  from ppl_decl cimport *
  ^
  ------------------------------------------------------------

  ppl/bit_arrays.pxd:1:0: 'ppl_decl.pxd' not found

  Error compiling Cython file:

PR: https://gitlab.com/videlec/pplpy/-/merge_requests/20

tornaria · 2023-07-17T21:30:16Z

Now that 3.0.0 is released, could we have this rebased and in a PR, please?

tornaria · 2023-07-21T15:14:13Z

Related: cython/cython#5552

tornaria · 2023-07-21T19:33:06Z

Cf: https://github.com/tornaria/sage/tree/cython3-legacy, I've got as far as being able to cythonize everything, but some C files still fail to compile.

First five commits are git cherry-pick f36275a7869 36ec4930832 1f9ee80cded 034094c849e 9d89bdf776 from the sagetrac mirror on top of 10.1.beta7.

Note that cython/cython#5552 means cythonize fails if python package lxml is available. Can use SAGE_DEBUG=no to avoid the issue which only happens with debug enabled.

Also: void-linux/void-packages#45086 includes patches for cysignals, cypari2, fpylll and pplpy that might be useful.

tornaria · 2023-07-22T20:28:58Z

After adding a commit fixing inheritance in lcalc_Lfunction (see 10.1.beta7...tornaria:sage:d0ad68ed9d4) I've got the whole of sagelib to compile.

Now it fails at import sage.structure.element as follows:

$ PYTHONPATH=pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-311 python
Python 3.11.4 (main, Jun  8 2023, 02:02:15) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sage.structure.element
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-311/sage/structure/__init__.py", line 2, in <module>
    import sage.structure.element
  File "sage/structure/element.pyx", line 1, in init sage.structure.element (build/cythonized/sage/structure/element.c:44218)
TypeError: PyMethodDescr_CallSelf requires a method without arguments

Please note: I'll probably keep rebasing and/or changing commits in my branch https://github.com/tornaria/sage/tree/cython3-legacy so don't take them as public commits unless you are willing to rebase.

tornaria · 2023-07-22T20:37:52Z

Branch pushed to git repo; I updated commit sha1. New commits:
fd2e122 src/sage/libs/lcalc/lcalc_Lfunction.pxd: Add missing cdef method declarations

This is incorrect and it stems from a fundamental change in cython 3 that will probably bite us all over the place (and this one cannot be switched off afaict).

I better link to the relevant section of the migration guide: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#class-private-name-mangling

This means sagelib has been (mis)using __name members (methods and attributes) that (a) in python are class-private via name mangling (b) in cython before 3.0.0 they are inherited due to mangling not implemented (c) in cython 3.0.0 they are class-private via the exact same name mangling as in python.

So the way to fix all these "missing cdef ..." errors is not by adding the missing methods but renaming the members to use a single underscore which is the python way to declare a "private" member that is still inherited by subclasses (think "protected" in c++).

Here are the changes needed to get Sage compiled with Cython 3. This PR contains changes that do not break compatibility with old Cython, so they can be merged without actually upgrading Cython. Part of sagemath#29863. ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. ### ⌛ Dependencies   URL: sagemath#36109 Reported by: Yutao Yuan Reviewer(s): Gonzalo Tornaría, Yutao Yuan

mkoeppe added this to the sage-9.2 milestone Jun 15, 2020

mkoeppe added c: packages: standard labels Jun 15, 2020

slel added the c: cython label Jun 18, 2020

mkoeppe modified the milestones: sage-9.2, sage-9.3 Aug 13, 2020

mkoeppe modified the milestones: sage-9.3, sage-9.4 Feb 13, 2021

mkoeppe modified the milestones: sage-9.4, sage-9.5 Jul 19, 2021

mkoeppe modified the milestones: sage-9.5, sage-9.6 Dec 14, 2021

mkoeppe modified the milestones: sage-9.6, sage-9.7 Mar 5, 2022

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~Upgrade Cython to 3.0.0 (unreleased)~~ Meta-ticket: Cython 3.0.0 (unreleased) Jul 24, 2022

This comment has been minimized.

Sign in to view

mkoeppe modified the milestones: sage-9.7, sage-9.8 Aug 31, 2022

mkoeppe modified the milestones: sage-9.8, sage-9.9 Jan 7, 2023

mkoeppe changed the title ~~Meta-ticket: Cython 3.0.0 (unreleased)~~ Meta-ticket: Cython 3.0.0 (entered beta stage) Feb 26, 2023

mkoeppe removed this from the sage-10.0 milestone Feb 26, 2023

mkoeppe changed the title ~~Meta-ticket: Cython 3.0.0 (entered beta stage)~~ Meta-ticket: Cython 3.0.0 (entered rc stage) Jul 14, 2023

tornaria mentioned this issue Jul 17, 2023

python3-Cython: update to 3.0.1. void-linux/void-packages#45086

Closed

30 tasks

mkoeppe changed the title ~~Meta-ticket: Cython 3.0.0 (entered rc stage)~~ Meta-ticket: Cython 3.0.0 (released July 2023) Jul 17, 2023

infmagic2047 mentioned this issue Aug 9, 2023

Update Cython to 3.0.0 #36054

Closed

3 tasks

This was referenced Aug 21, 2023

Prepare for updating Cython to 3.0.0 #36109

Merged

Update Cython to 3.0.2 #36110

Merged

vbraun closed this as completed in 05d7d1d Sep 27, 2023

mkoeppe added this to the sage-10.2 milestone Sep 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Meta-ticket: Cython 3.0.0 (released July 2023) #29863

Meta-ticket: Cython 3.0.0 (released July 2023) #29863

mkoeppe commented Jun 15, 2020 •

edited

Loading

slel commented Jun 18, 2020

mkoeppe commented Feb 13, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

mkoeppe commented Jul 25, 2022

mkoeppe commented Jul 25, 2022

mkoeppe commented Jul 25, 2022

mkoeppe commented Jul 25, 2022

mkoeppe commented Jul 25, 2022

mkoeppe commented Jul 25, 2022

mkoeppe commented Jul 25, 2022

mkoeppe commented Jul 25, 2022

sagetrac-git mannequin commented Jul 25, 2022

sagetrac-git mannequin commented Jul 25, 2022

mkoeppe commented Jul 25, 2022

This comment has been minimized.

mkoeppe commented Jul 31, 2022

This comment has been minimized.

sagetrac-git mannequin commented Jul 31, 2022

sagetrac-git mannequin commented Jul 31, 2022

mkoeppe commented Jul 31, 2022

tornaria commented Jul 17, 2023

tornaria commented Jul 21, 2023

tornaria commented Jul 21, 2023

tornaria commented Jul 22, 2023

tornaria commented Jul 22, 2023

Meta-ticket: Cython 3.0.0 (released July 2023) #29863

Meta-ticket: Cython 3.0.0 (released July 2023) #29863

Comments

mkoeppe commented Jun 15, 2020 • edited Loading

slel commented Jun 18, 2020

mkoeppe commented Feb 13, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

mkoeppe commented Jul 25, 2022

mkoeppe commented Jul 25, 2022

mkoeppe commented Jul 25, 2022

mkoeppe commented Jul 25, 2022

mkoeppe commented Jul 25, 2022

mkoeppe commented Jul 25, 2022

mkoeppe commented Jul 25, 2022

mkoeppe commented Jul 25, 2022

sagetrac-git mannequin commented Jul 25, 2022

sagetrac-git mannequin commented Jul 25, 2022

mkoeppe commented Jul 25, 2022

This comment has been minimized.

mkoeppe commented Jul 31, 2022

This comment has been minimized.

sagetrac-git mannequin commented Jul 31, 2022

sagetrac-git mannequin commented Jul 31, 2022

mkoeppe commented Jul 31, 2022

tornaria commented Jul 17, 2023

tornaria commented Jul 21, 2023

tornaria commented Jul 21, 2023

tornaria commented Jul 22, 2023

tornaria commented Jul 22, 2023

mkoeppe commented Jun 15, 2020 •

edited

Loading