Modularization of sagelib: Break out separate packages sagemath-objects, sagemath-categories #29865

mkoeppe · 2020-06-15T04:26:08Z

This ticket creates two experimental packages https://pypi.org/project/sagemath-objects/ and https://pypi.org/project/sagemath-categories/, outlined in Meta-ticket #29705, that consist of small subsets of the modules of sagelib, making Sage objects, the element/parent framework, categories, the coercion system and the related metaclasses available.

To build an sdist:

./sage -sh build/pkgs/sagemath_objects/spkg-src
./sage -sh build/pkgs/sagemath_categories/spkg-src

This creates a pip-installable source distribution sagemath-objects-VERSION.tar.gz (or sagemath-categories-VERSION.tar.gz) in upstream/.

The distributions depend only on Cython, pkgconfig, cysignals, gmpy2. The latter, of course, requires various headers and libraries, so they are not quite "purely" pip-installable packages.

The goal is for all modules of sagemath-objects and sagemath-categories to be importable even when sagelib is not installed. This can be tested (with tox) as follows:

./bootstrap && ./sage -sh -c '(cd build/pkgs/sagemath_objects/src && tox -v -v -e py39)'
./bootstrap && ./sage -sh -c '(cd build/pkgs/sagemath_categories/src && tox -v -v -e py39)'

sagemath-categories is intended to have sagemath-objects as a prerequisite. However, currently, sagemath-categories is an independent distribution that provides a superset of the modules of sagemath-objects.

Recombining it with the rest of sagelib via namespace package magic as outlined in #29705 is not implemented on this ticket. This will be done in follow-up tickets.

Depends on #29039
Depends on #32933

CC: @videlec @tscrim @orlitzky @nthiery @timokau @fchapoton @tobiasdiez @kwankyu @vbraun

Component: refactoring

Keywords: sd111

Author: Matthias Koeppe

Branch/Commit: 36d6b09

Reviewer: Kwankyu Lee

Issue created by migration from https://trac.sagemath.org/ticket/29865

The text was updated successfully, but these errors were encountered:

mkoeppe · 2020-06-15T06:12:50Z

Branch: u/mkoeppe/t/29865-modularizarion-sage-objects

sagetrac-git · 2020-06-15T07:09:49Z

Commit: dd3dc15

sagetrac-git · 2020-06-15T07:09:49Z

Branch pushed to git repo; I updated commit sha1. New commits:

`889f0c6`	`sage.env.sage_include_directories: Do not fail if numpy cannot be imported`
`dd3dc15`	`build/pkgs/sage_objects: Add dependencies, get minimal tox working`

sagetrac-git · 2020-06-15T07:47:14Z

Branch pushed to git repo; I updated commit sha1. New commits:

`4a99c1c`	`Add comment`
`b72264e`	`WIP: More stuff`

sagetrac-git · 2020-06-15T07:47:14Z

Changed commit from dd3dc15 to b72264e

sagetrac-git · 2020-06-15T16:17:11Z

Changed commit from b72264e to 23aaec1

sagetrac-git · 2020-06-15T16:17:11Z

Branch pushed to git repo; I updated commit sha1. New commits:

`23aaec1`	`Version with sage.structure, sage.categories that builds OK; uses gmpy2`

mkoeppe · 2020-06-15T18:40:28Z

Changed dependencies from #29864 to #29864, #29869

sagetrac-git · 2020-06-15T19:24:21Z

Changed commit from 23aaec1 to 29e30a4

sagetrac-git · 2020-06-15T19:24:21Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`cbc5555`	`sage.misc.misc, sage.misc.persist, sage.structure.formal_sum: Make some imports local to functions`
`0b7e11f`	`On the way to get sage.structure.all importing`
`19f86dc`	`sage.env: Do not die if DOT_SAGE is None`
`a795d56`	`sage.structure.factorization: Make import of sage.rings.integer.Integer local to methods`
`0ce9916`	`sage.structure.sequence: Make import of sage.misc.latex local to the latex method`
`fa22e19`	`sage.structure.all: Import .formal_sum lazily to avoid pulling in sage.modules`
`acf1ba3`	`sage.structure.factorization: Make import of sage.misc.misc_c.prod local to methods`
`14b1e55`	`sage.structure.formal_sum, .parent_gens: Fixup: Remove imports`
`a5453bf`	`Fixup: Add src/sage/misc/call.py`
`29e30a4`	`Merge branch 't/29869/move_attrcall_and_friends_from_sage_misc_misc_to_new_module_sage_misc_call' into t/29865-modularizarion-sage-objects`

mkoeppe · 2020-06-15T19:25:28Z

comment:11

import sage.structure.all succeeds now

sagetrac-git · 2020-06-15T19:50:00Z

Branch pushed to git repo; I updated commit sha1. New commits:

`c4a68e3`	`build/pkgs/sage_objects/src/tox.ini: Add comments, test more`

sagetrac-git · 2020-06-15T19:50:00Z

Changed commit from 29e30a4 to c4a68e3

mkoeppe · 2020-06-15T19:52:43Z

comment:13

Some of the Python modules in sage.categories seem to be sensitive to import order.

$ build/pkgs/sage_objects/src/.tox/python/bin/python -c 'import sage.categories.map'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "sage/categories/map.pyx", line 1, in init sage.categories.map (build/cythonized/sage/categories/map.c:16980)
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sage_objects/src/.tox/python/lib/python3.7/site-packages/sage/structure/__init__.py", line 2, in <module>
    import sage.structure.element
  File "sage/structure/parent.pxd", line 12, in init sage.structure.element (build/cythonized/sage/structure/element.c:34583)
  File "sage/categories/map.pxd", line 4, in init sage.structure.parent (build/cythonized/sage/structure/parent.c:27689)
AttributeError: module 'sage.categories.map' has no attribute 'Map'

vs.

$ build/pkgs/sage_objects/src/.tox/python/bin/python -c 'import sage.structure.all, sage.categories.map'
# succeeds

sagetrac-git · 2020-06-15T20:24:51Z

Changed commit from c4a68e3 to 7cca294

sagetrac-git · 2020-06-15T20:24:51Z

Branch pushed to git repo; I updated commit sha1. New commits:

`1963567`	`sage.categories.crystals: Make import of sage.misc.latex local to a method`
`c557714`	`sage.categories: Make imports from sage.rings, .sets, .combinat, .plot, .matrix local to methods`
`7cca294`	`Now we can import sage.categories.all`

mkoeppe · 2020-06-15T20:26:20Z

comment:15

$ build/pkgs/sage_objects/src/.tox/python/bin/python -c 'import sage.structure.all, sage.categories.all'
# succeeds

but does not succeed without importing sage.structure.all first...

mkoeppe · 2020-06-15T20:39:40Z

comment:16

build/pkgs/sage_objects/src/.tox/python/bin/python -c 'import sage.structure.all, sage.categories.all, sage.categories.facade_sets; sage.categories.facade_sets.FacadeSets().example()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "sage/misc/classcall_metaclass.pyx", line 322, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1717)
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sage_objects/src/.tox/python/lib/python3.7/site-packages/sage/categories/category_with_axiom.py", line 1993, in __classcall__
    (base_category_class, axiom) = cls._base_category_class_and_axiom
  File "sage/misc/lazy_attribute.pyx", line 699, in sage.misc.lazy_attribute.lazy_class_attribute.__get__ (build/cythonized/sage/misc/lazy_attribute.c:2894)
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sage_objects/src/.tox/python/lib/python3.7/site-packages/sage/categories/category_with_axiom.py", line 1933, in _base_category_class_and_axiom
    base_category_class, axiom = base_category_class_and_axiom(cls)
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sage_objects/src/.tox/python/lib/python3.7/site-packages/sage/categories/category_with_axiom.py", line 1790, in base_category_class_and_axiom
    assert getattr(base_category_class, axiom, None) is cls, \
  File "sage/misc/lazy_import.pyx", line 498, in sage.misc.lazy_import.LazyImport.__get__ (build/cythonized/sage/misc/lazy_import.c:4236)
  File "sage/misc/lazy_import.pyx", line 188, in sage.misc.lazy_import.LazyImport.get_object (build/cythonized/sage/misc/lazy_import.c:2340)
  File "sage/misc/lazy_import.pyx", line 217, in sage.misc.lazy_import.LazyImport._get_object (build/cythonized/sage/misc/lazy_import.c:2482)
RuntimeError: resolving lazy import FacadeSets during startup

sagetrac-git · 2020-06-15T21:43:03Z

Changed commit from 7cca294 to 2c2ddb7

sagetrac-git · 2020-06-15T21:43:03Z

Branch pushed to git repo; I updated commit sha1. New commits:

`2c2ddb7`	`build/pkgs/sage_objects/src/setup.py: Remove unused imports`

slel · 2021-11-26T19:27:39Z

comment:167

Never mind then! Sorry for the noise.

kwankyu · 2021-11-27T10:11:53Z

comment:168

Okay. Let's get this in.

mkoeppe · 2021-11-27T16:50:55Z

comment:169

Thanks!

kwankyu · 2022-01-03T02:33:58Z

comment:170

Why this is not yet merged?

vbraun · 2022-01-31T23:31:48Z

Changed branch from u/mkoeppe/t/29865-modularizarion-sage-objects to 36d6b09

…e relative by absolute imports   We restructure the `all.py` files for modularization. Guided by the technical constraints outlined in https://groups.google.com/g/sage-devel/c/ozh-7ZZ848s, sagemath#35095 defines distribution packages (pip-installable packages) **sagemath-{brial, combinat, eclib, flint, gap, giac, glpk, graphs, groups, homfly, lcalc, libbraiding, libecm, linbox, modules, mpmath, ntl, pari, plot, polyhedra, schemes, singular, standard-no-symbolics, symbolics}**. When a namespace package such as `sage.misc` is filled by modules from several distribution packages, we create modules named: - `src/sage/misc/all__sagemath_environment.py` - `src/sage/misc/all__sagemath_objects.py` - `src/sage/misc/all__sagemath_repl.py` Import statements are moved from `src/sage/misc/all.py` to these files as appropriate, and `src/sage/misc/all.py` imports `*` from there. Also some imports are replaced by lazy imports. The new files provide the top level namespaces for the modularized distribution packages, thus [enabling modularized testing](https://doc.s agemath.org/html/en/developer/packaging_sage_library.html#testing- distribution-packages). This design was introduced in sagemath#29865 (merged in Jan 2022, early in the Sage 9.6 development cycle).   - Copied from sagemath#35095. - Part of sagemath#29705 Moreover, applied a one-line command to replace relative by absolute imports, thus complementing sagemath#36666, which does not touch `.all*` files. The changes to other files in `sage.modules` etc. come from the PRs sagemath#36597, sagemath#36572, sagemath#36588, sagemath#36589 merged here and do not need review.  ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies  - Depends on sagemath#36597 (merged here to resolve merge conflict) - Depends on sagemath#36572 (merged here to resolve merge conflict) - Depends on sagemath#36588 (merged here to resolve merge conflict) - Depends on sagemath#36589 (merged here to resolve merge conflict) - Depends on sagemath#36449 (merged here to resolve merge conflict)  URL: sagemath#36676 Reported by: Matthias Köppe Reviewer(s): David Coudert, John H. Palmieri, Matthias Köppe, Tobias Diez

…e relative by absolute imports   We restructure the `all.py` files for modularization. Guided by the technical constraints outlined in https://groups.google.com/g/sage-devel/c/ozh-7ZZ848s, sagemath#35095 defines distribution packages (pip-installable packages) **sagemath-{brial, combinat, eclib, flint, gap, giac, glpk, graphs, groups, homfly, lcalc, libbraiding, libecm, linbox, modules, mpmath, ntl, pari, plot, polyhedra, schemes, singular, standard-no-symbolics, symbolics}**. When a namespace package such as `sage.misc` is filled by modules from several distribution packages, we create modules named: - `src/sage/misc/all__sagemath_environment.py` - `src/sage/misc/all__sagemath_objects.py` - `src/sage/misc/all__sagemath_repl.py` Import statements are moved from `src/sage/misc/all.py` to these files as appropriate, and `src/sage/misc/all.py` imports `*` from there. Also some imports are replaced by lazy imports. The new files provide the top level namespaces for the modularized distribution packages, thus [enabling modularized testing](https://doc.s agemath.org/html/en/developer/packaging_sage_library.html#testing- distribution-packages). This design was introduced in sagemath#29865 (merged in Jan 2022, early in the Sage 9.6 development cycle).   - Copied from sagemath#35095. - Part of sagemath#29705 Moreover, applied a one-line command to replace relative by absolute imports, thus complementing sagemath#36666, which does not touch `.all*` files. The changes to other files in `sage.modules` etc. come from the PRs sagemath#36597, sagemath#36572, sagemath#36588, sagemath#36589 merged here and do not need review.  ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies  - Depends on sagemath#36597 (merged here to resolve merge conflict) - Depends on sagemath#36572 (merged here to resolve merge conflict) - Depends on sagemath#36588 (merged here to resolve merge conflict) - Depends on sagemath#36589 (merged here to resolve merge conflict) - Depends on sagemath#36449 (merged here to resolve merge conflict)  URL: sagemath#36676 Reported by: Matthias Köppe Reviewer(s): David Coudert, John H. Palmieri, Kwankyu Lee, Matthias Köppe, Tobias Diez

mkoeppe added this to the sage-9.2 milestone Jun 15, 2020

mkoeppe added t: refactoring labels Jun 15, 2020

mkoeppe modified the milestones: sage-9.2, sage-9.3 Jun 15, 2020

This comment has been minimized.

Sign in to view

kwankyu added s: positive review and removed s: needs review labels Nov 27, 2021

mkoeppe modified the milestones: sage-9.5, sage-9.6 Jan 28, 2022

vbraun removed the s: positive review label Jan 31, 2022

vbraun closed this as completed in 25080fa Jan 31, 2022

embray mentioned this issue Mar 29, 2021

Add gappy as a standard package and use for sage's libgap interface #31297

Closed

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 12, 2023

Merge sagemath#29865

919f6a7

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 12, 2023

Merge sagemath#29865

aa60ae0

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 12, 2023

Merge sagemath#29865

4c9a9dd

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 12, 2023

sagemath#29865: src/sage/all__sagemath_objects.py: Add more from all.py

0fe8035

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 12, 2023

Merge sagemath#29865

98a3f39

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 12, 2023

Merge sagemath#29865

00c742d

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 12, 2023

Merge sagemath#29865

862261b

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 12, 2023

Merge sagemath#29865

88baa5a

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 12, 2023

Merge sagemath#29865

c1b9054

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 12, 2023

Merge sagemath#29865

433cb33

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 12, 2023

Merge sagemath#29865

d39e12c

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 17, 2023

Merge sagemath#29865

2c1366c

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 17, 2023

Merge sagemath#29865

c60d2d4

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 17, 2023

Merge sagemath#29865

41a5f50

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 17, 2023

Merge sagemath#29865

531063a

saraedum mentioned this issue Apr 29, 2024

Restructure sage.*.all for modularization, replace relative by absolute imports #37900

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modularization of sagelib: Break out separate packages sagemath-objects, sagemath-categories #29865

Modularization of sagelib: Break out separate packages sagemath-objects, sagemath-categories #29865

mkoeppe commented Jun 15, 2020

This comment has been minimized.

This comment has been minimized.

mkoeppe commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

This comment has been minimized.

sagetrac-git mannequin commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

mkoeppe commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

mkoeppe commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

mkoeppe commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

mkoeppe commented Jun 15, 2020

mkoeppe commented Jun 15, 2020

This comment has been minimized.

sagetrac-git mannequin commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

slel commented Nov 26, 2021

kwankyu commented Nov 27, 2021

mkoeppe commented Nov 27, 2021

kwankyu commented Jan 3, 2022

vbraun commented Jan 31, 2022

Modularization of sagelib: Break out separate packages sagemath-objects, sagemath-categories #29865

Modularization of sagelib: Break out separate packages sagemath-objects, sagemath-categories #29865

Comments

mkoeppe commented Jun 15, 2020

This comment has been minimized.

This comment has been minimized.

mkoeppe commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

This comment has been minimized.

sagetrac-git mannequin commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

mkoeppe commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

mkoeppe commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

mkoeppe commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

mkoeppe commented Jun 15, 2020

mkoeppe commented Jun 15, 2020

This comment has been minimized.

sagetrac-git mannequin commented Jun 15, 2020

sagetrac-git mannequin commented Jun 15, 2020

slel commented Nov 26, 2021

kwankyu commented Nov 27, 2021

mkoeppe commented Nov 27, 2021

kwankyu commented Jan 3, 2022

vbraun commented Jan 31, 2022