Skip to content

Commit

Permalink
Trac #32899: Developer's guide: Add section on modularization
Browse files Browse the repository at this point in the history 
We add a section to the developer manual, explaining packaging portions
of the Sage library to distributions packages.

Related discussion in the sage devel:

https://groups.google.com/g/sage-devel/c/cTifWZOAfH8

See also: #32925 Improve documentation of `sage.features`

URL: https://trac.sagemath.org/32899
Reported by: mkoeppe
Ticket author(s): Matthias Koeppe, Kwankyu Lee
Reviewer(s): Kwankyu Lee, Matthias Koeppe
  • Loading branch information
Release Manager committed Dec 16, 2021
2 parents c137dea + b555735 commit 44ce36b
Show file tree
Hide file tree
Showing 6 changed files with 614 additions and 60 deletions.
46 changes: 32 additions & 14 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -349,54 +349,72 @@ Simplified directory layout (only essential files/directories):
SAGE_ROOT Root directory (sage-x.y.z in Sage tarball)
├── build
│ └── pkgs Every package is a subdirectory here
│ ├── 4ti2
│ ├── 4ti2/
│ …
│ └── zn_poly
│ └── zn_poly/
├── configure Top-level configure script
├── COPYING.txt Copyright information
├── pkgs Source trees of Python distribution packages
│ ├── sage-conf
│ │ ├── sage_conf.py
│ │ └── setup.py
│ ├── sage-docbuild
│ │ ├── sage_docbuild
│ │ ├── sage_docbuild/
│ │ └── setup.py
│ ├── sage-setup
│ │ ├── sage_setup/
│ │ └── setup.py
│ ├── sage-sws2rst
│ │ ├── sage_sws2rst
│ │ ├── sage_sws2rst/
│ │ └── setup.py
│ └── sagemath-standard
│ ├── bin
│ ├── sage
│ ├── bin/
│ ├── sage -> ../../src/sage
│ └── setup.py
├── local (SAGE_LOCAL) Compiled packages are installed here
├── local (SAGE_LOCAL) Installation hierarchy for non-Python packages
│ ├── bin Executables
│ ├── include C/C++ headers
│ ├── lib Shared libraries
│ ├── lib Shared libraries, architecture-dependent data
│ ├── share Databases, architecture-independent data, docs
│ │ └── doc Viewable docs of Sage and of some components
│ └── var
│ ├── lib/sage List of installed packages
│ └── tmp/sage Temporary files when building Sage
│ ├── lib/sage
│ │ ├── installed/
│ │ │ Records of installed non-Python packages
│ │ ├── scripts/ Scripts for uninstalling installed packages
│ │ └── venv-python3.9 (SAGE_VENV)
│ │ │ Installation hierarchy (virtual environment)
│ │ │ for Python packages
│ │ ├── bin/ Executables and installed scripts
│ │ ├── lib/python3.9/site-packages/
│ │ │ Python modules/packages are installed here
│ │ └── var/lib/sage/
│ │ └── wheels/
│ │ Python wheels for all installed Python packages
│ │
│ └── tmp/sage/ Temporary files when building Sage
├── logs
│ ├── dochtml.log Log of the documentation build
│ ├── install.log Full install log
│ └── pkgs Build logs of individual packages
│ ├── alabaster-0.7.12.log
│ …
│ └── zn_poly-0.9.2.log
├── m4 M4 macros for configure
├── m4 M4 macros for generating the configure script
│ └── *.m4
├── Makefile Running "make" uses this file
├── prefix -> SAGE_LOCAL Convenience symlink to the installation tree
├── README.md This file
├── sage Script to start Sage
├── src Monolithic Sage library source tree
│ ├── bin Scripts that Sage uses internally
│ ├── doc Sage documentation sources
│ └── sage The Sage library source code
│ ├── bin/ Scripts that Sage uses internally
│ ├── doc/ Sage documentation sources
│ └── sage/ The Sage library source code
├── upstream Source tarballs of packages
│ ├── Babel-2.9.1.tar.gz
│ …
│ └── zn_poly-0.9.2.tar.gz
├── venv -> SAGE_VENV Convenience symlink to the virtual environment
└── VERSION.txt
```
For more details see [our Developer's Guide](https://doc.sagemath.org/html/en/developer/coding_basics.html#files-and-directory-structure).
Expand Down
65 changes: 33 additions & 32 deletions pkgs/sagemath-standard/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,21 +28,34 @@
sage.env.SAGE_SRC = os.getcwd()
from sage.env import *

from sage_setup.excepthook import excepthook
sys.excepthook = excepthook

from sage_setup.setenv import setenv
setenv()

#########################################################
### Configuration
#########################################################

if len(sys.argv) > 1 and sys.argv[1] == "sdist":
if len(sys.argv) > 1 and (sys.argv[1] == "sdist" or sys.argv[1] == "egg_info"):
sdist = True
else:
sdist = False

if sdist:
cmdclass = {}
else:
from sage_setup.excepthook import excepthook
sys.excepthook = excepthook

from sage_setup.setenv import setenv
setenv()

from sage_setup.command.sage_build import sage_build
from sage_setup.command.sage_build_cython import sage_build_cython
from sage_setup.command.sage_build_ext import sage_build_ext
from sage_setup.command.sage_install import sage_install_and_clean

cmdclass = dict(build=sage_build,
build_cython=sage_build_cython,
build_ext=sage_build_ext,
install=sage_install_and_clean)

#########################################################
### Testing related stuff
#########################################################
Expand All @@ -52,45 +65,36 @@
if os.path.exists(sage.misc.lazy_import_cache.get_cache_file()):
os.unlink(sage.misc.lazy_import_cache.get_cache_file())


from sage_setup.command.sage_build import sage_build
from sage_setup.command.sage_build_cython import sage_build_cython
from sage_setup.command.sage_build_ext import sage_build_ext


#########################################################
### Discovering Sources
#########################################################

# TODO: This should be quiet by default
print("Discovering Python/Cython source code....")
t = time.time()

from sage_setup.optional_extension import is_package_installed_and_updated

if sdist:
# No need to compute distributions. This avoids a dependency on Cython
# just to make an sdist.
distributions = None
python_packages = []
python_modules = []
cython_modules = []
else:
# TODO: This should be quiet by default
print("Discovering Python/Cython source code....")
t = time.time()
from sage_setup.optional_extension import is_package_installed_and_updated
distributions = ['']
optional_packages_with_extensions = ['mcqd', 'bliss', 'tdlib', 'primecount',
'coxeter3', 'fes', 'sirocco', 'meataxe']
distributions += ['sagemath-{}'.format(pkg)
for pkg in optional_packages_with_extensions
if is_package_installed_and_updated(pkg)]
log.warn('distributions = {0}'.format(distributions))
from sage_setup.find import find_python_sources
python_packages, python_modules, cython_modules = find_python_sources(
SAGE_SRC, ['sage'], distributions=distributions)

from sage_setup.find import find_python_sources
python_packages, python_modules, cython_modules = find_python_sources(
SAGE_SRC, ['sage'], distributions=distributions)

log.debug('python_packages = {0}'.format(python_packages))

print("Discovered Python/Cython sources, time: %.2f seconds." % (time.time() - t))

log.debug('python_packages = {0}'.format(python_packages))
print("Discovered Python/Cython sources, time: %.2f seconds." % (time.time() - t))

from sage_setup.command.sage_install import sage_install_and_clean

#########################################################
### Distutils
Expand Down Expand Up @@ -175,8 +179,5 @@
'bin/sage-update-src',
'bin/sage-update-version',
],
cmdclass = dict(build=sage_build,
build_cython=sage_build_cython,
build_ext=sage_build_ext,
install=sage_install_and_clean),
cmdclass = cmdclass,
ext_modules = cython_modules)
12 changes: 6 additions & 6 deletions pkgs/sagemath-standard/tox.ini
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,15 @@ envlist =
# Build dependencies according to requirements.txt (all versions fixed).
# Use ONLY the wheels built and stored by the Sage distribution (no PyPI):
#
# ./sage -sh -c '(cd pkgs/sagelib && tox -v -v -v -e python-sagewheels-nopypi)'
# ./sage -sh -c '(cd pkgs/sagemath-standard && tox -v -v -v -e python-sagewheels-nopypi)'
#
python-sagewheels-nopypi,
#
# Build and test without using the concrete dependencies specified by requirements.txt,
# using the dependencies declared in pyproject.toml and setup.cfg (install-requires) only:
# Still use ONLY the wheels built and stored by the Sage distribution (no PyPI).
#
# ./sage -sh -c '(cd pkgs/sagelib && tox -v -v -v -e python-sagewheels-nopypi-norequirements)'
# ./sage -sh -c '(cd pkgs/sagemath-standard && tox -v -v -v -e python-sagewheels-nopypi-norequirements)'
#
python-sagewheels-nopypi-norequirements,
#
Expand All @@ -44,7 +44,7 @@ envlist =
# and additionally allow packages from PyPI.
# Because all versions are fixed, we "should" end up using the prebuilt wheels.
#
# ./sage -sh -c '(cd pkgs/sagelib && tox -v -v -v -e python-sagewheels)'
# ./sage -sh -c '(cd pkgs/sagemath-standard && tox -v -v -v -e python-sagewheels)'
#
python-sagewheels,
#
Expand Down Expand Up @@ -81,8 +81,8 @@ passenv =
PKG_CONFIG_PATH
# Parallel build
SAGE_NUM_THREADS
# SAGE_LOCAL only for finding the wheels
sagewheels: SAGE_LOCAL
# SAGE_VENV only for finding the wheels
sagewheels: SAGE_VENV
# Location of the wheels (needs to include a PEP 503 compliant
# Simple Repository index, i.e., a subdirectory "simple")
sagewheels: SAGE_SPKG_WHEELS
Expand All @@ -92,7 +92,7 @@ setenv =
HOME={envdir}
# We supply pip options by environment variables so that they
# apply both to the installation of the dependencies and of the package
sagewheels: PIP_FIND_LINKS=file://{env:SAGE_SPKG_WHEELS:{env:SAGE_LOCAL:{toxinidir}/../../../../local}/var/lib/sage/wheels}
sagewheels: PIP_FIND_LINKS=file://{env:SAGE_SPKG_WHEELS:{env:SAGE_VENV:{toxinidir}/../../../../venv}/var/lib/sage/wheels}
nopypi: PIP_NO_INDEX=true
# No build isolation for PEP 517 packages - use what is already in the environment
# Note that this pip env "NO" variable uses inverted logic:
Expand Down
8 changes: 8 additions & 0 deletions src/doc/en/developer/index.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,14 @@ Sage Coding Details
coding_in_cython
coding_in_other

Packaging the Sage Library
--------------------------

.. toctree::
:maxdepth: 3

packaging_sage_library

Packaging Third-Party Code
--------------------------

Expand Down
20 changes: 12 additions & 8 deletions src/doc/en/developer/packaging.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -545,7 +545,12 @@ would simply contain:
Python-based packages
---------------------
The best way to install a Python-based package is to use pip, in which
Python-based packages should declare ``$(PYTHON)`` as a dependency,
and most Python-based packages will also have ``$(PYTHON_TOOLCHAIN)`` as
an order-only dependency, which will ensure that fundamental packages such
as ``pip`` and ``setuptools`` are available at the time of building the package.
The best way to install a Python-based package is to use ``pip``, in which
case the ``spkg-install.in`` script template might just consist of
.. CODE-BLOCK:: bash
Expand All @@ -556,19 +561,18 @@ Where ``sdh_pip_install`` is a function provided by ``sage-dist-helpers`` that
points to the correct ``pip`` for the Python used by Sage, and includes some
default flags needed for correct installation into Sage.
If pip will not work but a command like ``python3 setup.py install``
If ``pip`` will not work for a package but a command like ``python3 setup.py install``
will, you may use ``sdh_setup_bdist_wheel``, followed by
``sdh_store_and_pip_install_wheel .``.
For ``spkg-check.in`` script templates, make sure to call
``sage-python23`` rather than ``python``. This will ensure that the
correct version of Python is used to check the package.
The same holds for ; for example, the ``scipy`` ``spkg-check.in``
file contains the line
For ``spkg-check.in`` script templates, use ``python3`` rather
than just ``python``. The paths are set by the Sage build system
so that this runs the correct version of Python.
For example, the ``scipy`` ``spkg-check.in`` file contains the line
.. CODE-BLOCK:: bash
exec sage-python23 spkg-check.py
exec python3 spkg-check.py
All normal Python packages must have a file ``install-requires.txt``.
If a Python package is available on PyPI, this file must contain the
Expand Down
Loading

0 comments on commit 44ce36b

Please sign in to comment.