Skip to content

Commit

Permalink
Trac #21524: configure.ac: write build/make/Makefile within an AC_CON…
Browse files Browse the repository at this point in the history
…FIG_FILE, not during main configure

`build/make/Makefile` should be written by `config.status` within an
`AC_CONFIG_FILE` template, not at `configure` time.

URL: https://trac.sagemath.org/21524
Reported by: mkoeppe
Ticket author(s): Erik Bray
Reviewer(s): Julian Rüth
  • Loading branch information
Release Manager authored and vbraun committed Mar 6, 2018
2 parents 2972b5a + 2a99185 commit 15fbaa6
Show file tree
Hide file tree
Showing 7 changed files with 570 additions and 316 deletions.
56 changes: 56 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -344,6 +344,62 @@ For more details, see:
http://doc.sagemath.org/html/en/developer/coding_basics.html#files-and-directory-structure


Build System
------------

This is a brief summary of the Sage software distribution's build system.
There are two components to the full Sage system--the Sage Python library
and its associated user interfaces, and the larger software distribution of
Sage's main dependencies (for those dependencies not supplied by the user's
system).

Sage's Python library is built and installed using a `setup.py` script as is
standard for Python packages (Sage's `setup.py` is non-trivial, but not
unusual).

Most of the rest of the build system is concerned with building all of Sage's
dependencies in the correct order in relation to each other. The dependencies
included by Sage are referred to as SPKGs (i.e. "Sage Packages") and are listed
under `build/pkgs`.

The main entrypoint to Sage's build system is the top-level `Makefile` at the
root of the source tree. Unlike most normal projects that use autoconf (Sage
does as well, as described below), this `Makefile` is not generated. Instead,
it contains a few high-level targets and targets related to bootstrapping the
system. Nonetheless, we still run `make <target>` from the root of the source
tree--targets not explicitly defined in the top-level `Makefile` are passed
through to another Makefile under `build/make/Makefile`.

The latter `build/make/Makefile` *is* generated by an autoconf-generated
`configure` script, using the template in `build/make/Makefile.in`. This
includes rules for building the Sage library itself (`make sagelib`), and for
building and installing each of Sage's dependencies (e.g. `make python2`).

Although it's possible to manually run Sage's `configure` script if one wants
to provide some customizations (e.g. it is possible to select which BLAS
implementation to use), the top-level `Makefile` will run `configure` for you,
in order to build `build/make/Makefile` since it's a prerequisite for most of
Sage's make targets.

The `configure` script itself, if it is not already built, can be generated by
running the `bootstrap` script. The top-level `Makefile` also takes care of
this automatically.

To summarize, running a command like `make python2` at the top-level of the
source tree goes something like this:

1. `make python2`
2. run `./bootstrap` if `configure` does not exist
3. run `./configure` if `build/make/Makefile` doe not exist
4. `cd` into `build/make` and run the `install` script--this is little more
than a front-end to running `make -f build/make/Makefile python2`, which
sets some necessary environment variables and logs some information
5. `build/make/Makefile` contains the actual rule for building `python2`; this
includes building all of `python2`'s dependencies first (and their
dependencies, recursively); the actual package installation is performed
with the `sage-spkg` program


Relocation
----------

Expand Down
258 changes: 258 additions & 0 deletions build/make/Makefile.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,258 @@
# Makefile template for Sage packages: This Makefile is filled by the
# ./configure script with information all of Sage's dependent packages (SPKGs),
# including their names, their current versions, their dependencies, and some
# classifications according to their installation priority ("standard",
# "optional") and installation method ("normal", "pip", "script").
#
# Finally, install and clean rules for each package are generated from the
# templates at the end of this file. Because the templates may slightly
# obscure the substance of the actual rules, this file can be debugged by
# running:
#
# $ make -f build/make/Makefile -n DEBUG_RULES=1
#
# This will not actually run any rules (the -n flag) but will print all the
# rules generated from the templates.

# Always use bash for make rules
SHELL = @SHELL@

ifndef SAGE_SPKG_INST
ifndef DEBUG_RULES
$(error This Makefile needs to be invoked by build/make/install)
else
# A dummy value for this variable for debugging purposes
SAGE_SPKG_INST=installed
endif
endif

# Directory to keep track of which packages are installed
INST = $(SAGE_SPKG_INST)

# Aliases for optional packages selected at configure time
TOOLCHAIN = @SAGE_TOOLCHAIN@
PYTHON = python@SAGE_PYTHON_VERSION@
MP_LIBRARY = @SAGE_MP_LIBRARY@
BLAS = @SAGE_BLAS@

# Files to track installation of packages
BUILT_PACKAGES = @SAGE_BUILT_PACKAGES@
DUMMY_PACKAGES = @SAGE_DUMMY_PACKAGES@

# Versions of all the packages, in the format
#
# vers_<pkgname> = <pkgvers>

@SAGE_PACKAGE_VERSIONS@

# Dependencies for all packages, in the format
#
# deps_<pkgname> = <dep1> <dep2> etc...

@SAGE_PACKAGE_DEPENDENCIES@

# All standard packages
STANDARD_PACKAGES = @SAGE_STANDARD_PACKAGES@
STANDARD_PACKAGE_INSTS = \
$(foreach pkgname,$(STANDARD_PACKAGES),$(inst_$(pkgname)))

# All optional installed packages (triggers the auto-update)
OPTIONAL_INSTALLED_PACKAGES = @SAGE_OPTIONAL_INSTALLED_PACKAGES@
OPTIONAL_INSTALLED_PACKAGE_INSTS = \
$(foreach pkgname,$(OPTIONAL_INSTALLED_PACKAGES),$(inst_$(pkgname)))

# All packages which should be downloaded
SDIST_PACKAGES = @SAGE_SDIST_PACKAGES@
SDIST_PACKAGE_INSTS = $(foreach pkgname,$(SDIST_PACKAGES),$(inst_$(pkgname)))

SCRIPTS = @SAGE_SCRIPTS@

EXTCODE = @SAGE_EXTCODE@

# Packages that use the 'normal' build rules
NORMAL_PACKAGES = @SAGE_NORMAL_PACKAGES@

# Packages that use the 'pip' package build rules
PIP_PACKAGES = @SAGE_PIP_PACKAGES@

# Packages that use the 'script' package build rules (not to be confused with
# the $(SCRIPTS) list)
SCRIPT_PACKAGES = @SAGE_SCRIPT_PACKAGES@



# Generate the actual inst_<pkgname> variables; for each package that is
# actually built this generates a line like:
#
# inst_<pkgname> = $(INST)/<pkgname>-<pkgvers>
#
# And for 'dummy' package that are not actually built/installed (e.g. because
# configure determined we can use the package from the system):
#
# inst_<pkgname> = $(INST)/.dummy
#
# For example:
#
# inst_python2 = $(INST)/python2-$(vers_python2)
#
# inst_git = $(INST)/.dummy

$(foreach pkgname,$(BUILT_PACKAGES),\
$(eval inst_$(pkgname) = $$(INST)/$(pkgname)-$(vers_$(pkgname))))

$(foreach pkgname,$(DUMMY_PACKAGES),\
$(eval inst_$(pkgname) = $$(INST)/.dummy))


# Dummy target for packages which are not installed
$(INST)/.dummy:
touch $@


@SAGE_MAKE_DEPS@


#==============================================================================
# Rules generated from pkgs/<package>/dependencies files
#==============================================================================

# Define a function for generating the list of a package's dependencies
# as $(inst_<pkgname>) variables. For example, takes:
#
# deps_cysignals = python2 cython pari | pip
#
# to:
#
# $(inst_python2) $(inst_cython) $(inst_pari) | $(inst_pip)
#
# Positional arguments:
# $(1): package name
pkg_deps = \
$(foreach dep,$(deps_$(1)),\
$(if $(findstring |,$(dep)),|,$$(inst_$(dep))))

# ============================= normal packages ==============================
# Generate build rules for 'normal' packages; this template is used to generate
# three rules in the form:
#
# $(INST)/<pkgname>-<pkgvers>: <dependencies>
# +$(AM_V_at)sage-logger -p '$(SAGE_SPKG) <pkgname>-<pkgvers>' '$(SAGE_LOGS)/<pkgname>-<pkgvers>.log'
#
# <pkgname>: $(INST)/<pkgname>-<pkgvers>
#
# <pkgname>-clean:
# rm -rf $(INST)/<pkgname>-<pkgvers>
#
# For example, for python2 this will expand to:
#
# $(INST)/python2-2.7.14: $(inst_zlib) $(inst_readline) $(inst_sqlite) $(inst_libpng) $(inst_bzip2)
# +$(AM_V_at)sage-logger -p '$(SAGE_SPKG) python2-2.7.14' '$(SAGE_LOGS)/python2-2.7.14.log'
#
# python2: $(INST)/python2-2.7.14
#
# python2-clean:
# rm -rf $(INST)/python2-2.7.14
#
# Note: In these rules the $(INST)/<pkgname>-<pkgvers> target is used
# explicitly, rather than expanding the $(inst_<pkgname>) variable, since
# it may expand to $(INST)/.dummy for packages that were not configured
# for installation by default. However, we wish to be able to manually
# install those packages later.

# Positional arguments:
# $(1): package name
# $(2): package version
# $(3): package dependencies
define NORMAL_PACKAGE_templ
$$(INST)/$(1)-$(2): $(3)
+$(AM_V_at)sage-logger -p '$$(SAGE_SPKG) $(1)-$(2)' '$$(SAGE_LOGS)/$(1)-$(2).log'

$(1): $$(INST)/$(1)-$(2)

$(1)-clean:
rm -rf $$(INST)/$(1)-$(2)

endef

$(foreach pkgname, $(NORMAL_PACKAGES),\
$(eval $(call NORMAL_PACKAGE_templ,$(pkgname),$(vers_$(pkgname)),\
$(call pkg_deps,$(pkgname)))))

ifdef DEBUG_RULES
$(info # Rules for standard packages)
$(foreach pkgname, $(NORMAL_PACKAGES),\
$(info $(call NORMAL_PACKAGE_templ,$(pkgname),$(vers_$(pkgname)),\
$(call pkg_deps,$(pkgname)))))
endif

# ================================ pip packages ===============================
# Generate build rules for 'pip' packages; this template is used to generate
# two rules in the form:
#
# <pkgname>: <dependencies>
# $(AM_V_at)sage-logger -p 'sage --pip install <pkgname>' '$(SAGE_LOGS)/<pkgname>.log'
#
# <pkgname>-clean:
# -sage --pip uninstall -y <pkgname>

# Positional arguments:
# $(1): package name
# $(2): package dependencies
define PIP_PACKAGE_templ
$(1): $(2)
$(AM_V_at)sage-logger -p 'sage --pip install $(1)' '$$(SAGE_LOGS)/$(1).log'

$(1)-clean:
-sage --pip uninstall -y $(1)
endef

$(foreach pkgname,$(PIP_PACKAGES),\
$(eval $(call PIP_PACKAGE_templ,$(pkgname),$(call pkg_deps,$(pkgname)))))

ifdef DEBUG_RULES
$(info # Rules for pip packages)
$(foreach pkgname,$(PIP_PACKAGES),\
$(info $(call PIP_PACKAGE_templ,$(pkgname),$(call pkg_deps,$(pkgname)))))
endif

# ============================= script packages ==============================
# Generate build rules for 'script' packages; this template is used to generate
# two rules in the form:
#
# <pkgname>: <dependencies>
# $(AM_V_at)cd '$SAGE_ROOT' && \\
# source '$SAGE_ROOT/src/bin/sage-env' && \\
# sage-logger -p '$SAGE_ROOT/build/pkgs/<pkgname>/spkg-install' '$(SAGE_LOGS)/<pkgname>.log'
#
# <pkgname>-clean:
# -$(AM_V_at)cd '$SAGE_ROOT' && \\
# source '$SAGE_ROOT/src/bin/sage-env' && \\
# '$SAGE_ROOT/build/pkgs/$PKG_NAME/spkg-uninstall'

# Positional arguments:
# $(1): package name
# $(2): package dependencies
define SCRIPT_PACKAGE_templ
$(1): $(2)
$(AM_V_at)cd '$$(SAGE_ROOT)' && \
source '$$(SAGE_ROOT)/src/bin/sage-env' && \
sage-logger -p '$$(SAGE_ROOT)/build/pkgs/$(1)/spkg-install' '$$(SAGE_LOGS)/$(1).log'

$(1)-clean:
-$(AM_V_at)cd '$$(SAGE_ROOT)' && \
source '$$(SAGE_ROOT)/src/bin/sage-env' && \
'$$(SAGE_ROOT)/build/pkgs/$(1)/spkg-uninstall'
endef

$(foreach pkgname,$(SCRIPT_PACKAGES),\
$(eval $(call SCRIPT_PACKAGE_templ,$(pkgname),$(call pkg_deps,$(pkgname)))))

ifdef DEBUG_RULES
$(info # Rules for script packages)
$(foreach pkgname,$(SCRIPT_PACKAGES),\
$(info $(call SCRIPT_PACKAGE_templ,$(pkgname),$(call pkg_deps,$(pkgname)))))
endif

.PHONY: $(NORMAL_PACKAGES) $(addsuffix -clean,$(NORMAL_PACKAGES)) \
$(PIP_PACKAGES) $(addsuffix -clean,$(PIP_PACKAGES)) \
$(SCRIPT_PACKAGES) $(addsuffix -clean,$(SCRIPT_PACKAGES))
12 changes: 6 additions & 6 deletions build/make/deps
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## -*- Makefile -*- ###########################################################
# This file ($SAGE_ROOT/build/make/deps) will be copied into
# $SAGE_ROOT/build/make/Makefile by $SAGE_ROOT/build/make/install
# $SAGE_ROOT/build/make/Makefile by config.status
###############################################################################

# Silent rules
Expand Down Expand Up @@ -49,20 +49,20 @@ all-toolchain:
# All targets except for the base packages
all-sage: \
sagelib \
$(STANDARD_PACKAGES) \
$(OPTIONAL_INSTALLED_PACKAGES) \
$(STANDARD_PACKAGE_INSTS) \
$(OPTIONAL_INSTALLED_PACKAGE_INSTS) \
$(EXTCODE) \
$(SCRIPTS)

# Download all packages which should be inside an sdist tarball (the -B
# option to make forces all targets to be built unconditionally)
download-for-sdist: base
env SAGE_INSTALL_FETCH_ONLY=yes $(MAKE) -B SAGERUNTIME= \
$(SDIST_PACKAGES)
$(SDIST_PACKAGE_INSTS)

# TOOLCHAIN consists of dependencies determined by configure.
# These are built after the "base" target but before anything else.
toolchain: $(TOOLCHAIN)
toolchain: $(foreach pkgname,$(TOOLCHAIN),$(inst_$(pkgname)))

# Build all packages that GCC links against serially, otherwise this
# leads to race conditions where some library which is used by GCC gets
Expand Down Expand Up @@ -93,7 +93,7 @@ start: all-build
# We make this depend on all standard packages because running
# sage-starts runs sage-location, which should be run after installing
# any package.
$(STARTED): $(STANDARD_PACKAGES)
$(STARTED): $(STANDARD_PACKAGE_INSTS)
$(AM_V_at)"$(SAGE_LOCAL)/bin/sage-starts"


Expand Down
2 changes: 1 addition & 1 deletion build/pkgs/gcc/spkg-install
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ $MAKE install

# Mark gfortran as not installed in case gfortran was installed earlier
cd "$SAGE_SPKG_INST"
rm -f gmp-* mpir-* mpfr-* mpc-* gfortran-*
rm -f gfortran-*

# Force re-configuration: the next time that "make" is run, we need to
# rebuild all packages (#24703) but we should not rebuild gcc (#19324)
Expand Down
Loading

0 comments on commit 15fbaa6

Please sign in to comment.