Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Many more namespace packages – follow up #35366

Merged
merged 16 commits into from
Apr 6, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
3fe393c
Remove many empty __init__.py files when all.py files are present
mkoeppe Mar 21, 2023
915dc24
Remove some more empty / almost empty __init__.py files when all.py f…
mkoeppe Mar 21, 2023
9957360
sage.misc.package_dir.walk_packages: New, use in sage.misc.dev_tools
mkoeppe Mar 21, 2023
8a9ff3f
Remove most remaining empty / almost empty __init__.py files when all…
mkoeppe Mar 22, 2023
8dcea98
src/sage/symbolic/integration/all.py: New
mkoeppe Mar 22, 2023
7f8eb6a
is_package_or_sage_namespace_package_dir: Update doctest after namesp…
mkoeppe Mar 22, 2023
f13d6e9
src/sage/symbolic/pynac.pxi: Work around https://github.com/cython/cy…
tornaria Mar 25, 2023
5ee730d
src/sage/misc/cython.py: Add doctests for PEP 420 support, https://gi…
tornaria Mar 25, 2023
dc8b3c5
is_package_or_sage_namespace_package_dir: Update more doctests after …
tornaria Mar 25, 2023
a3f9255
Merge branch 'many_more_namespace_packages' into many_more_namespace_…
mkoeppe Mar 27, 2023
884b18f
src/sage/combinat/words: Move documentation from __init__.py to all.py
mkoeppe Mar 27, 2023
8c4ea79
src/sage/modules/with_basis: Move documentation from __init__.py to a…
mkoeppe Mar 27, 2023
4837e0b
src/sage/groups/perm_gps/partn_ref*: Change to namespace packages
mkoeppe Mar 27, 2023
9e8afdd
src/sage_setup/autogen/interpreters/__init__.py: Generate sage.ext.in…
mkoeppe Mar 27, 2023
cb5577a
src/doc/en/reference: Update module lists
mkoeppe Mar 28, 2023
4b397a7
src/sage/combinat/all.py: Update reference
mkoeppe Mar 28, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/doc/en/reference/combinat/module_list.rst
Original file line number Diff line number Diff line change
Expand Up @@ -367,7 +367,7 @@ Comprehensive Module List
sage/combinat/tutorial
sage/combinat/vector_partition
sage/combinat/words/abstract_word
sage/combinat/words
sage/combinat/words/all
sage/combinat/words/alphabet
sage/combinat/words/finite_word
sage/combinat/words/infinite_word
Expand Down
2 changes: 1 addition & 1 deletion src/doc/en/reference/modules/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Modules with basis
.. toctree::
:maxdepth: 1

sage/modules/with_basis/__init__
sage/modules/with_basis/all
sage/modules/with_basis/cell_module
sage/modules/with_basis/indexed_element
sage/modules/with_basis/invariant
Expand Down
File renamed without changes.
2 changes: 1 addition & 1 deletion src/sage/combinat/all.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
- :ref:`sage.combinat.species.all`
- :ref:`sage.combinat.designs.all`
- :ref:`sage.combinat.posets.all`
- :ref:`sage.combinat.words`
- :ref:`sage.combinat.words.all`
- :ref:`sage.combinat.bijectionist`

Utilities
Expand Down
Empty file removed src/sage/combinat/sf/__init__.py
Empty file.
Empty file.
41 changes: 0 additions & 41 deletions src/sage/combinat/words/__init__.py

This file was deleted.

45 changes: 45 additions & 0 deletions src/sage/combinat/words/all.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,48 @@
r"""
Combinatorics on words

**Main modules and their methods:**

- :ref:`sage.combinat.words.abstract_word`
- :ref:`sage.combinat.words.finite_word`
- :ref:`sage.combinat.words.infinite_word`
- :ref:`sage.combinat.words.alphabet`
- :ref:`sage.combinat.words.words`
- :ref:`sage.combinat.words.paths`
- :ref:`sage.combinat.words.morphism`
- :ref:`sage.combinat.words.shuffle_product`
- :ref:`sage.combinat.words.suffix_trees`

Main classes and functions meant to be used by the user:

:func:`~sage.combinat.words.word.Word`,
:class:`~sage.combinat.words.words.FiniteWords`,
:class:`~sage.combinat.words.words.InfiniteWords`,
:func:`~sage.combinat.words.words.Words`,
:func:`~sage.combinat.words.alphabet.Alphabet`,
:class:`~sage.combinat.words.morphism.WordMorphism`,
:class:`~sage.combinat.words.paths.WordPaths`.

A list of common words can be accessed through ``words.<tab>`` and are listed in
the :ref:`words catalog <sage.combinat.words.word_generators>`.

**Internal representation of words:**

- :ref:`sage.combinat.words.word`
- :ref:`sage.combinat.words.word_char`
- :ref:`sage.combinat.words.word_datatypes`
- :ref:`sage.combinat.words.word_infinite_datatypes`

**Options:**

- :ref:`sage.combinat.words.word_options`

See :func:`~sage.combinat.words.word_options.WordOptions`.
"""
# install the docstring of this module to the containing package
from sage.misc.namespace_package import install_doc
install_doc(__package__, __doc__)

from .alphabet import Alphabet, build_alphabet
from .morphism import WordMorphism
from .paths import WordPaths
Expand Down
Empty file.
Empty file.
Empty file.
Empty file removed src/sage/databases/__init__.py
Empty file.
Empty file removed src/sage/dynamics/__init__.py
Empty file.
Empty file.
3 changes: 0 additions & 3 deletions src/sage/dynamics/cellular_automata/__init__.py

This file was deleted.

Empty file.
Empty file removed src/sage/functions/__init__.py
Empty file.
Empty file removed src/sage/game_theory/__init__.py
Empty file.
Empty file removed src/sage/games/__init__.py
Empty file.
Empty file removed src/sage/geometry/__init__.py
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file removed src/sage/graphs/base/__init__.py
Empty file.
Empty file.
Empty file removed src/sage/groups/__init__.py
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
File renamed without changes.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
File renamed without changes.
Empty file.
Empty file removed src/sage/homology/__init__.py
Empty file.
Empty file removed src/sage/interacts/__init__.py
Empty file.
Empty file removed src/sage/knots/__init__.py
Empty file.
Empty file removed src/sage/lfunctions/__init__.py
Empty file.
1 change: 0 additions & 1 deletion src/sage/libs/gap/__init__.py

This file was deleted.

Empty file removed src/sage/logic/__init__.py
Empty file.
Empty file removed src/sage/manifolds/__init__.py
Empty file.
Empty file.
Empty file.
Empty file.
File renamed without changes.
2 changes: 0 additions & 2 deletions src/sage/matroids/__init__.py

This file was deleted.

22 changes: 22 additions & 0 deletions src/sage/misc/cython.py
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,28 @@ def cython(filename, verbose=0, compile_message=False,
sage: cython('''
....: cdef size_t foo = 3/2
....: ''')

Check that Cython supports PEP 420 packages::

sage: cython('''
....: cimport sage.misc.cachefunc
....: ''')

sage: cython('''
....: from sage.misc.cachefunc cimport cache_key
....: ''')

In Cython 0.29.33 using `from PACKAGE cimport MODULE` is broken
when `PACKAGE` is a namespace package, see :trac:`35322`::

sage: cython('''
....: from sage.misc cimport cachefunc
....: ''')
Traceback (most recent call last):
...
RuntimeError: Error compiling Cython file:
...
...: 'sage/misc.pxd' not found
"""
if not filename.endswith('pyx'):
print("Warning: file (={}) should have extension .pyx".format(filename), file=sys.stderr)
Expand Down
4 changes: 2 additions & 2 deletions src/sage/misc/dev_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ def load_submodules(module=None, exclude_pattern=None):
load sage.geometry.polyhedron.palp_database... succeeded
load sage.geometry.polyhedron.ppl_lattice_polygon... succeeded
"""
import pkgutil
from .package_dir import walk_packages

if module is None:
import sage
Expand All @@ -181,7 +181,7 @@ def load_submodules(module=None, exclude_pattern=None):
else:
exclude = None

for importer, module_name, ispkg in pkgutil.walk_packages(module.__path__, module.__name__ + '.'):
for importer, module_name, ispkg in walk_packages(module.__path__, module.__name__ + '.'):
if ispkg or module_name in sys.modules:
continue

Expand Down
130 changes: 126 additions & 4 deletions src/sage/misc/package_dir.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@

import os
import glob
import sys
from contextlib import contextmanager


Expand Down Expand Up @@ -154,29 +155,29 @@ def is_package_or_sage_namespace_package_dir(path, *, distribution_filter=None):
:mod:`sage.cpython` is an ordinary package::

sage: from sage.misc.package_dir import is_package_or_sage_namespace_package_dir
sage: directory = os.path.dirname(sage.cpython.__file__); directory
sage: directory = sage.cpython.__path__[0]; directory
'.../sage/cpython'
sage: is_package_or_sage_namespace_package_dir(directory)
True

:mod:`sage.libs.mpfr` only has an ``__init__.pxd`` file, but we consider
it a package directory for consistency with Cython::

sage: directory = os.path.join(os.path.dirname(sage.libs.all.__file__), 'mpfr'); directory
sage: directory = os.path.join(sage.libs.__path__[0], 'mpfr'); directory
'.../sage/libs/mpfr'
sage: is_package_or_sage_namespace_package_dir(directory)
True

:mod:`sage` is designated to become an implicit namespace package::

sage: directory = os.path.dirname(sage.env.__file__); directory
sage: directory = sage.__path__[0]; directory
'.../sage'
sage: is_package_or_sage_namespace_package_dir(directory)
True

Not a package::

sage: directory = os.path.join(os.path.dirname(sage.symbolic.__file__), 'ginac'); directory
sage: directory = os.path.join(sage.symbolic.__path__[0], 'ginac'); directory
'.../sage/symbolic/ginac'
sage: is_package_or_sage_namespace_package_dir(directory)
False
Expand Down Expand Up @@ -211,3 +212,124 @@ def cython_namespace_package_support():
yield
finally:
Cython.Utils.is_package_dir = Cython.Build.Cythonize.is_package_dir = Cython.Build.Dependencies.is_package_dir = orig_is_package_dir


def walk_packages(path=None, prefix='', onerror=None):
r"""
Yield :class:`pkgutil.ModuleInfo` for all modules recursively on ``path``.

This version of the standard library function :func:`pkgutil.walk_packages`
addresses https://github.com/python/cpython/issues/73444 by handling
the implicit namespace packages in the package layout used by Sage;
see :func:`is_package_or_sage_namespace_package_dir`.

INPUT:

- ``path`` -- a list of paths to look for modules in or
``None`` (all accessible modules).

- ``prefix`` -- a string to output on the front of every module name
on output.

- ``onerror`` -- a function which gets called with one argument (the
name of the package which was being imported) if any exception
occurs while trying to import a package. If ``None``, ignore
:class:`ImportError` but propagate all other exceptions.

EXAMPLES::

sage: sorted(sage.misc.package_dir.walk_packages(sage.misc.__path__)) # a namespace package
[..., ModuleInfo(module_finder=FileFinder('.../sage/misc'), name='package_dir', ispkg=False), ...]
"""
# Adapted from https://github.com/python/cpython/blob/3.11/Lib/pkgutil.py

def iter_modules(path=None, prefix=''):
"""
Yield :class:`ModuleInfo` for all submodules on ``path``.
"""
from pkgutil import get_importer, iter_importers, ModuleInfo

if path is None:
importers = iter_importers()
elif isinstance(path, str):
raise ValueError("path must be None or list of paths to look for modules in")
else:
importers = map(get_importer, path)

yielded = {}
for i in importers:
for name, ispkg in iter_importer_modules(i, prefix):
if name not in yielded:
yielded[name] = 1
yield ModuleInfo(i, name, ispkg)

def iter_importer_modules(importer, prefix=''):
r"""
Yield :class:`ModuleInfo` for all modules of ``importer``.
"""
from importlib.machinery import FileFinder

if isinstance(importer, FileFinder):
if importer.path is None or not os.path.isdir(importer.path):
return

yielded = {}
import inspect
try:
filenames = os.listdir(importer.path)
except OSError:
# ignore unreadable directories like import does
filenames = []
filenames.sort() # handle packages before same-named modules

for fn in filenames:
modname = inspect.getmodulename(fn)
if modname and (modname in ['__init__', 'all']
or modname.startswith('all__')
or modname in yielded):
continue

path = os.path.join(importer.path, fn)
ispkg = False

if not modname and os.path.isdir(path) and '.' not in fn:
modname = fn
if not (ispkg := is_package_or_sage_namespace_package_dir(path)):
continue

if modname and '.' not in modname:
yielded[modname] = 1
yield prefix + modname, ispkg

elif not hasattr(importer, 'iter_modules'):
yield from []

else:
yield from importer.iter_modules(prefix)

def seen(p, m={}):
if p in m:
return True
m[p] = True

for info in iter_modules(path, prefix):
yield info

if info.ispkg:
try:
__import__(info.name)
except ImportError:
if onerror is not None:
onerror(info.name)
except Exception:
if onerror is not None:
onerror(info.name)
else:
raise
else:
path = getattr(sys.modules[info.name], '__path__', None) or []

# don't traverse path items we've seen before
path = [p for p in path if not seen(p)]

yield from walk_packages(path, info.name + '.', onerror)
Empty file removed src/sage/modular/__init__.py
Empty file.
Empty file removed src/sage/modular/abvar/__init__.py
Empty file.
Empty file.
Empty file.
Empty file removed src/sage/modular/hecke/__init__.py
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
1 change: 0 additions & 1 deletion src/sage/modular/quasimodform/__init__.py

This file was deleted.

Empty file.
Empty file removed src/sage/modular/ssmod/__init__.py
Empty file.
Empty file removed src/sage/modules/__init__.py
Empty file.
2 changes: 2 additions & 0 deletions src/sage/modules/all.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@

from .vector_space_morphism import linear_transformation

from .with_basis.all import *

from sage.misc.lazy_import import lazy_import

lazy_import('sage.modules.filtered_vector_space', 'FilteredVectorSpace')
Expand Down
Empty file.
File renamed without changes.
Empty file.
File renamed without changes.
Empty file.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,6 @@

.. SEEALSO:: The category :class:`ModulesWithBasis`
"""
# install the docstring of this module to the containing package
from sage.misc.namespace_package import install_doc
install_doc(__package__, __doc__)
Empty file removed src/sage/monoids/__init__.py
Empty file.
Empty file removed src/sage/parallel/__init__.py
Empty file.
Empty file removed src/sage/plot/__init__.py
Empty file.
Empty file removed src/sage/plot/plot3d/__init__.py
Empty file.
Empty file removed src/sage/probability/__init__.py
Empty file.
Empty file.
Empty file.
Empty file removed src/sage/quivers/__init__.py
Empty file.
File renamed without changes.
Empty file.
Empty file removed src/sage/rings/convert/__init__.py
Empty file.
File renamed without changes.
Empty file.
Empty file.
Empty file.
Empty file removed src/sage/rings/padics/__init__.py
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file removed src/sage/sandpiles/__init__.py
Empty file.
Empty file removed src/sage/sat/__init__.py
Empty file.
Empty file removed src/sage/schemes/__init__.py
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file.
Empty file removed src/sage/schemes/toric/__init__.py
Empty file.
Empty file.
Empty file removed src/sage/server/__init__.py
Empty file.
Empty file removed src/sage/stats/__init__.py
Empty file.
Empty file.
Empty file removed src/sage/stats/hmm/__init__.py
Empty file.
Empty file removed src/sage/symbolic/__init__.py
Empty file.
Empty file.
4 changes: 2 additions & 2 deletions src/sage/symbolic/pynac.pxi
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ Declarations for pynac, a Python frontend for ginac

Check that we can externally cimport this (:trac:`18825`)::

sage: cython( # long time; random compiler warnings # optional - sage.misc.cython
sage: cython( # optional - sage.misc.cython
....: '''
....: from sage.symbolic cimport expression
....: cimport sage.symbolic.expression
....: ''')
"""

Expand Down
Empty file removed src/sage/tensor/__init__.py
Empty file.
Empty file.
Empty file removed src/sage/topology/__init__.py
Empty file.
Empty file removed src/sage/typeset/__init__.py
Empty file.
9 changes: 7 additions & 2 deletions src/sage_setup/autogen/interpreters/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,11 @@ def rebuild(dirname, force=False):
except OSError:
if not os.path.isdir(dirname):
raise
# Remove leftover file from before move to namespace packages
try:
os.remove(os.path.join(dirname, '__init__.py'))
except FileNotFoundError:
pass

# Although multiple files are generated by this function, since
# they are all generated at once it suffices to make sure if just
Expand All @@ -208,7 +213,7 @@ class NeedToRebuild(Exception):
try:
if force:
raise NeedToRebuild("-> Force rebuilding interpreters")
gen_file = os.path.join(dirname, '__init__.py')
gen_file = os.path.join(dirname, 'all.py')
if not os.path.isfile(gen_file):
raise NeedToRebuild("-> First build of interpreters")

Expand All @@ -230,5 +235,5 @@ class NeedToRebuild(Exception):
for interp in _INTERPRETERS:
build_interp(interp(), dirname)

with open(os.path.join(dirname, '__init__.py'), 'w') as f:
with open(os.path.join(dirname, 'all.py'), 'w') as f:
f.write("# " + AUTOGEN_WARN)