Modify find_python_sources to support modularization of sagelib by native namespace packages (PEP 420) #28925

mkoeppe · 2019-12-30T01:31:06Z

This ticket prepares the modularization of sagelib into separate distributions (pip-installable packages) by adding support for native namespace packages (PEP 420) to sagelib's build system.

Ordinary packages (directories of modules) can be turned into namespace packages by removing the empty __init__.py file.
Then several distribution package can share the same namespace.

Because our source discovery mechanism in sagelib's build system (setup.py + sage_setup) distinguishes package directories from other directories containing data files by the presence of the __init__.py file, the present ticket makes several adjustments to the build system - in particular to the functions find_python_sources and find_extra_files.

Per convention in the Sage library, namespace packages are recognized by the presence of all.py or all__*.py files (#33033).

In this ticket, we apply it to turn sagemath-objects and sagemath-categories (from #29865) into namespace packages,
in which __init__.py are removed from the MANIFEST.in files.

Because the actual files are not removed from the source tree, the normal build of the Sage distribution is not affected.

To test these two distributions:

./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-objects && SAGE_NUM_THREADS=16 tox -r -v -v -v -e py39)'

./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-categories && SAGE_NUM_THREADS=16 tox -r -v -v -v -e py39)'

Follow-up tickets:

Remove __init__.py files for subpackages designated to be namespace packages #33011 Remove the __init__.py files for packages designated to be namespace packages
Wishlist item: Warn or give errors if the sage_setup: distribution directives, namespace and nonamespace files indicate an inconsistent structure such as when it was forgotten to remove __init__.py.
sage_setup: Modify clean_stale_files to support out-of-tree namespace packages #30152 will make further changes to the cleaner that will allow out-of-tree namespace packages

See also:

Meta-ticket: Modularize sagelib into separate distributions (pip-installable packages) sagemath-... for Sage 10.x #29705: Meta-ticket: Modularize sagelib into separate distutils packages
Initial discussion of namespace packages for Sage in Move optional sage optimization backends (COIN, CPLEX, Gurobi) to separate Cython packages to remove OptionalExtension problems #28175 and Use native namespace package sage-numerical-backends-coin#4
src/setup.py: Disentangle cleaning of stale installed files in build directory and in install directory #21654: src/setup.py: Disentangle cleaning of stale installed files in build directory and in install directory

Depends on #33803

CC: @isuruf @embray @jdemeyer @dimpase @dcoudert @videlec @vbraun @tobiasdiez @kiwifb

Component: build

Keywords: sd111

Author: Matthias Koeppe

Branch/Commit: f9df1ae

Reviewer: Kwankyu Lee, Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/28925

The text was updated successfully, but these errors were encountered:

mkoeppe · 2020-04-14T06:38:06Z

comment:1

pushing these forward to 9.2

mkoeppe · 2020-05-20T04:49:18Z

Branch: u/mkoeppe/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_

mkoeppe · 2020-05-20T04:49:35Z

Commit: 8bc5662

mkoeppe · 2020-05-20T04:49:35Z

New commits:

`8e50bbe`	`sage.graphs.graph_decompositions: Make it a namespace package by removing __init__.py (which had unnecessary reimports only)`
`dff6e62`	`Mark src/sage/ext_data as directories that are not namespace packages`
`8bc5662`	`sage_setup.find.find_python_sources: Handle native namespace packages and 'nonamespace' files`

mkoeppe · 2020-05-20T04:49:35Z

Author: Matthias Koeppe

sagetrac-git · 2020-05-20T05:11:46Z

Changed commit from 8bc5662 to 606a509

sagetrac-git · 2020-05-20T05:11:46Z

Branch pushed to git repo; I updated commit sha1. New commits:

`606a509`	`sage.numerical.backends: Make it a namespace package by removing __init__.py (which was empty)`

mkoeppe · 2020-05-20T05:51:31Z

comment:18

Next steps: adjust the cleaner, find Cython sources

mkoeppe · 2020-05-20T05:55:14Z

comment:19

The Cython source finder could perhaps also parse a new directive that controls OptionalExtension behavior (until we get rid of OptionalExtension altogether in #29701) so that we can get rid of module_list.py completely (see #29706)

sagetrac-git · 2020-05-20T22:36:21Z

Changed commit from 606a509 to a7b2287

kwankyu · 2022-07-01T01:25:38Z

comment:155

Thanks.

Another comment: clean_install_dir and find_extra_files have a new argument distributions, but the docstrings do not explain it.

sagetrac-git · 2022-07-01T01:49:57Z

Changed commit from cd4d8e9 to 9ab3200

sagetrac-git · 2022-07-01T01:49:57Z

Branch pushed to git repo; I updated commit sha1. New commits:

`af9ac78`	`src/sage_setup/find.py (find_extra_files): Document new arg 'distributions'`
`13e66a3`	`src/sage_setup/find.py (find_extra_files): Fix indentation`
`9ab3200`	`src/sage_setup/clean.py (clean_install_dir): Document new arg 'distributions'`

kwankyu · 2022-07-04T02:47:24Z

Reviewer: Kwankyu Lee, ...

kwankyu · 2022-07-04T02:47:24Z

comment:157

Thank you for the update.

I have been testing (using) this ticket and and the sequel #33011 for a while. They works well. I checked the code as far as I can (not very far), and it looks good to me.

If we don't have further comments on this ticket, then it would better be tested by more developers in the wild by merging this now, so that we can fix problems before the next release.

I will wait for your (not Matthias) meta comments, and proceed to "positive review" in a couple of days.

kwankyu · 2022-07-06T06:39:56Z

comment:158

Is this

./sage -sh -c '(cd pkgs/sagemath-categories && tox -v -v -v)'

broken? It doesn't work with (and without) this ticket.

dimpase · 2022-07-06T09:00:54Z

comment:159

Replying to @kwankyu:

Is this
./sage -sh -c '(cd pkgs/sagemath-categories && tox -v -v -v)'  
broken? It doesn't work with (and without) this ticket.

works for me (after make build from a clean state done)

kwankyu · 2022-07-06T09:06:36Z

comment:160

Replying to @dimpase:

Replying to @kwankyu:
Is this
./sage -sh -c '(cd pkgs/sagemath-categories && tox -v -v -v)'  
broken? It doesn't work with (and without) this ticket.
works for me (after make build from a clean state done)

I did make distclean; make -j8; ./sage -sh -c '(cd pkgs/sagemath-categories && tox -v -v -v)'. So this is clean.

This is the initial part that goes wrong:

sage -t --random-seed=251639358214975434990568781743714015030 --environment=sage.all__sagemath_categories /Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/structure/coerce_actions.pyx
**********************************************************************
File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/structure/coerce_actions.pyx", line 58, in sage.structure.coerce_actions.GenericAction.__init__
Failed example:
    M = MatrixSpace(ZZ,2)
Exception raised:
    Traceback (most recent call last):
      File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.structure.coerce_actions.GenericAction.__init__[0]>", line 1, in <module>
        M = MatrixSpace(ZZ,Integer(2))
    NameError: name 'MatrixSpace' is not defined

Any clue? I am on Mac.

dimpase · 2022-07-06T10:28:40Z

Changed reviewer from Kwankyu Lee, ... to Kwankyu Lee, Dima Pasechnik

dimpase · 2022-07-06T10:28:40Z

comment:161

I start from git clean -fdx - which cleans much more.

You'll probably want to move upstream/ outside and after git clean make a symbolic link to it, otherwise it will re-download things.

So it's positive review from me.

kwankyu · 2022-07-06T12:57:46Z

comment:162

Replying to @dimpase:

I start from git clean -fdx - which cleans much more.

Thanks. But it didn't help. As far as I understand, the doctest failures are expected. The real problem is:

$ pkgs/sagemath-categories/.tox/py39/bin/python3.9 
Python 3.9.13 (main, May 24 2022, 21:28:31) 
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from sage.categories.category import Category
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/py39/lib/python3.9/site-packages/sage/categories/category.py", line 107, in <module>
    from sage.misc.c3_controlled import _cmp_key, _cmp_key_named, C3_sorted_merge
  File "sage/misc/c3_controlled.pyx", line 364, in init sage.misc.c3_controlled (build/cythonized/sage/misc/c3_controlled.c:11819)
  File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/py39/lib/python3.9/site-packages/sage/structure/__init__.py", line 2, in <module>
    import sage.structure.element
  File "sage/structure/element.pyx", line 1, in init sage.structure.element (build/cythonized/sage/structure/element.c:36204)
  File "sage/structure/category_object.pyx", line 60, in init sage.structure.category_object (build/cythonized/sage/structure/category_object.c:9982)
ImportError: cannot import name Category

Strangely this works

Hera:sage-dev$ pkgs/sagemath-categories/.tox/py39/bin/python3.9 
Python 3.9.13 (main, May 24 2022, 21:28:31) 
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from sage.structure.sage_object import SageObject
>>> from sage.categories.category import Category
>>>

Anyway, this is not a problem of this ticket. The situation is the same without this ticket.

Hence my positive review to this ticket!

dimpase · 2022-07-06T14:10:28Z

comment:163

hmm, do you have the current directory, i.e. ., in your PATH? IMHO this might mess things up.

kwankyu · 2022-07-06T14:56:25Z

comment:164

Replying to @dimpase:

hmm, do you have the current directory, i.e. ., in your PATH? IMHO this might mess things up.

Yes. But even after removing it from the path, it's the same.

./configure shows python3-3.10.3: using system package; SPKG will not be installed, and

$ pkgs/sagemath-categories/.tox/py39/bin/python3.9 
Python 3.9.13 (main, May 24 2022, 21:28:31) 
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python39.zip', '/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9', '/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload', '/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/py39/lib/python3.9/site-packages']

$ pkgs/sagemath-categories/.tox/python/bin/python 
Python 3.10.4 (main, Apr 26 2022, 19:42:59) [Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/local/Cellar/python@3.10/3.10.4/Frameworks/Python.framework/Versions/3.10/lib/python310.zip', '/usr/local/Cellar/python@3.10/3.10.4/Frameworks/Python.framework/Versions/3.10/lib/python3.10', '/usr/local/Cellar/python@3.10/3.10.4/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload', '/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages']

mkoeppe · 2022-07-06T18:04:16Z

comment:165

Replying to @dimpase:

Replying to @kwankyu:
Is this
./sage -sh -c '(cd pkgs/sagemath-categories && tox -v -v -v)'  
broken? It doesn't work with (and without) this ticket.
works for me (after make build from a clean state done)

sagemath-categories (in contrast to sagemath-standard) does not need make build because all of its build and runtime dependencies can be found on PyPI.

mkoeppe · 2022-07-06T18:10:27Z

comment:166

Replying to @kwankyu:

This is the initial part that goes wrong:

sage -t --random-seed=251639358214975434990568781743714015030 --environment=sage.all__sagemath_categories /Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/structure/coerce_actions.pyx
**********************************************************************
File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/structure/coerce_actions.pyx", line 58, in sage.structure.coerce_actions.GenericAction.__init__
Failed example:
    M = MatrixSpace(ZZ,2)
Exception raised:
    Traceback (most recent call last):
      File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/python/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.structure.coerce_actions.GenericAction.__init__[0]>", line 1, in <module>
        M = MatrixSpace(ZZ,Integer(2))
    NameError: name 'MatrixSpace' is not defined

sagemath-categories as of this ticket does not contain src/sage/matrix/matrix_space.py (see pkgs/sagemath-categories/MANIFEST.in), so MatrixSpace is not available in sage.all__sagemath_categories.

Note that the tox run ends with the message "lots of doctest failures are expected". This is one of them.

mkoeppe · 2022-07-06T18:13:13Z

comment:167

Replying to @kwankyu:

The real problem is:

$ pkgs/sagemath-categories/.tox/py39/bin/python3.9 
Python 3.9.13 (main, May 24 2022, 21:28:31) 
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from sage.categories.category import Category
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/py39/lib/python3.9/site-packages/sage/categories/category.py", line 107, in <module>
    from sage.misc.c3_controlled import _cmp_key, _cmp_key_named, C3_sorted_merge
  File "sage/misc/c3_controlled.pyx", line 364, in init sage.misc.c3_controlled (build/cythonized/sage/misc/c3_controlled.c:11819)
  File "/Users/kwankyu/GitHub/sage-dev/pkgs/sagemath-categories/.tox/py39/lib/python3.9/site-packages/sage/structure/__init__.py", line 2, in <module>
    import sage.structure.element
  File "sage/structure/element.pyx", line 1, in init sage.structure.element (build/cythonized/sage/structure/element.c:36204)
  File "sage/structure/category_object.pyx", line 60, in init sage.structure.category_object (build/cythonized/sage/structure/category_object.c:9982)
ImportError: cannot import name Category

Strangely this works

Hera:sage-dev$ pkgs/sagemath-categories/.tox/py39/bin/python3.9 
Python 3.9.13 (main, May 24 2022, 21:28:31) 
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from sage.structure.sage_object import SageObject
>>> from sage.categories.category import Category
>>>

The Sage library contains many cyclic imports. If one starts at the wrong place in an import cycle, one can run into problems with partially initialized modules. See #33580 (Meta-ticket: make all modules importable without sage.all)

mkoeppe · 2022-07-06T18:13:36Z

comment:168

Thanks both for the review!

kwankyu · 2022-07-07T04:45:28Z

comment:169

Replying to @mkoeppe:

Note that the tox run ends with the message "lots of doctest failures are expected".

Yes, it's there. It would perhaps better be colored red or green...

vbraun · 2022-07-08T21:12:46Z

comment:170

Merge failure on top of:

843eb03 Updated SageMath version to 9.7.beta4

merge was not clean: conflicts in pkgs/sagemath-categories/tox.ini

sagetrac-git · 2022-07-08T23:35:31Z

Branch pushed to git repo; I updated commit sha1. New commits:

`f9df1ae`	`Merge tag '9.7.beta4' into t/28925/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_`

sagetrac-git · 2022-07-08T23:35:31Z

Changed commit from 9ab3200 to f9df1ae

vbraun · 2022-07-10T22:35:02Z

Changed branch from u/mkoeppe/modify_find_python_sources__clean_stale_files_to_support_modularization_of_sagelib_by_native_namespace_packages__pep_420_ to f9df1ae

mkoeppe added this to the sage-9.1 milestone Dec 30, 2019

mkoeppe added c: build labels Dec 30, 2019

mkoeppe modified the milestones: sage-9.1, sage-9.2 Apr 14, 2020

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~Modify clean_stale_files to support modularization of sagelib by namespace packages~~ Modify clean_stale_files to support modularization of sagelib by native namespace packages (PEP 420) May 17, 2020

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~Modify clean_stale_files to support modularization of sagelib by native namespace packages (PEP 420)~~ Modify find_python_sources, clean_stale_files to support modularization of sagelib by native namespace packages (PEP 420) May 18, 2020

This comment has been minimized.

Sign in to view

dimpase added s: positive review and removed s: needs review labels Jul 6, 2022

vbraun added s: needs work and removed s: positive review labels Jul 8, 2022

mkoeppe added s: positive review and removed s: needs work labels Jul 8, 2022

vbraun removed the s: positive review label Jul 10, 2022

vbraun closed this as completed in 6294f36 Jul 10, 2022

Modify find_python_sources to support modularization of sagelib by native namespace packages (PEP 420) #28925

Modify find_python_sources to support modularization of sagelib by native namespace packages (PEP 420) #28925

Comments

mkoeppe commented Dec 30, 2019

mkoeppe commented Apr 14, 2020

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

mkoeppe commented May 20, 2020

mkoeppe commented May 20, 2020

mkoeppe commented May 20, 2020

mkoeppe commented May 20, 2020

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sagetrac-git mannequin commented May 20, 2020

sagetrac-git mannequin commented May 20, 2020

This comment has been minimized.

mkoeppe commented May 20, 2020

mkoeppe commented May 20, 2020

sagetrac-git mannequin commented May 20, 2020

kwankyu commented Jul 1, 2022

sagetrac-git mannequin commented Jul 1, 2022

sagetrac-git mannequin commented Jul 1, 2022

kwankyu commented Jul 4, 2022

kwankyu commented Jul 4, 2022

kwankyu commented Jul 6, 2022

dimpase commented Jul 6, 2022

kwankyu commented Jul 6, 2022

dimpase commented Jul 6, 2022

dimpase commented Jul 6, 2022

kwankyu commented Jul 6, 2022

dimpase commented Jul 6, 2022

kwankyu commented Jul 6, 2022

mkoeppe commented Jul 6, 2022

mkoeppe commented Jul 6, 2022

mkoeppe commented Jul 6, 2022

mkoeppe commented Jul 6, 2022

kwankyu commented Jul 7, 2022

vbraun commented Jul 8, 2022

sagetrac-git mannequin commented Jul 8, 2022

sagetrac-git mannequin commented Jul 8, 2022

vbraun commented Jul 10, 2022