Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ninja uses more jobs than specified by makeflags #38950

Open
user202729 opened this issue Nov 10, 2024 · 1 comment
Open

ninja uses more jobs than specified by makeflags #38950

user202729 opened this issue Nov 10, 2024 · 1 comment
Labels
c: build

Comments

@user202729
Copy link
Contributor

user202729 commented Nov 10, 2024 •
edited
Loading

Currently, if SageMath is built, it will use up more than number of parallel jobs specified by MAKEFLAGS, because make and ninja has no communication.

I notice that there are already tools to parse number of threads (./src/bin/sage-num-threads.py). At the very least we can throw -jN to ninja, so it would use at most 2N jobs.

Partial work:

A process tree might look like the following.

 0.0  0.1             make build
 0.0  0.1               make all-build
 0.0  0.0                 bash build/bin/sage-logger cd build/make && ./install 'all-build' logs/install.log
 0.0  0.0                   bash build/bin/sage-logger cd build/make && ./install 'all-build' logs/install.log
 0.0  0.0                     bash build/bin/sage-logger cd build/make && ./install 'all-build' logs/install.log
 0.0  0.0                       bash build/bin/sage-logger cd build/make && ./install 'all-build' logs/install.log
 0.0  0.0                         bash ./install all-build
 0.0  0.0                           make all-build
 0.0  0.1                             make --no-print-directory all-sage
 0.0  0.0                               make --no-print-directory scipy-SAGE_VENV-no-deps
 0.0  0.0                                 /bin/sh -c if [ -z '/sage/local/var/lib/sage/venv-python3.12' ]; then echo "Error: The installation tree SAGE_VENV has been disabled" 2>&1; echo "" 2>&1; exit 1; else sage-logger -p 'SAGE_CHECK=no PATH=/sage/src/bin:/sage/local/var/lib/sage/venv-python3.12/bin:$PATH sage-spkg -o  scipy-1.12.0 /sage/local/var/lib/sage/venv-python3.12' '/sage/logs/pkgs/scipy-1.12.0.log'; fi
 0.0  0.0                                   bash /sage/build/bin/sage-logger -p SAGE_CHECK=no PATH=/sage/src/bin:/sage/local/var/lib/sage/venv-python3.12/bin:$PATH sage-spkg -o  scipy-1.12.0 /sage/local/var/lib/sage/venv-python3.12 /sage/logs/pkgs/scipy-1.12.0.log
 0.0  0.0                                     bash /sage/build/bin/sage-logger -p SAGE_CHECK=no PATH=/sage/src/bin:/sage/local/var/lib/sage/venv-python3.12/bin:$PATH sage-spkg -o  scipy-1.12.0 /sage/local/var/lib/sage/venv-python3.12 /sage/logs/pkgs/scipy-1.12.0.log
 0.0  0.0                                       bash /sage/build/bin/sage-logger -p SAGE_CHECK=no PATH=/sage/src/bin:/sage/local/var/lib/sage/venv-python3.12/bin:$PATH sage-spkg -o  scipy-1.12.0 /sage/local/var/lib/sage/venv-python3.12 /sage/logs/pkgs/scipy-1.12.0.log
 0.0  0.0                                         bash /sage/build/bin/sage-logger -p SAGE_CHECK=no PATH=/sage/src/bin:/sage/local/var/lib/sage/venv-python3.12/bin:$PATH sage-spkg -o  scipy-1.12.0 /sage/local/var/lib/sage/venv-python3.12 /sage/logs/pkgs/scipy-1.12.0.log
 0.0  0.0                                           bash /sage/build/bin/sage-spkg -o scipy-1.12.0 /sage/local/var/lib/sage/venv-python3.12
 0.0  0.0                                             bash /sage/build/bin/sage-logger -P spkg-install ./spkg-install
 0.0  0.0                                               bash ./spkg-install
 0.0  0.1                                                 python3 -m build --wheel --outdir=dist --no-isolation --skip-dependency-check .
 0.0  0.2                                                   /sage/local/var/lib/sage/venv-python3.12/bin/python3 /sage/local/var/lib/sage/venv-python3.12/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpbaglu4nd
 0.0  0.1                                                     /usr/bin/ninja
 0.0  0.0                                                       g++ -Iscipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p -Iscipy/sparse/sparsetools -I../scipy/sparse/sparsetools -I../scipy/_build_utils/src -I../../../../../../../lib/python3.12/site-packages/numpy/core/include -I/usr/include/python3.12 -fvisibility=hidden -fvisibility-inlines-hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c++14 -O3 -g -O2 -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_9_API_VERSION -MD -MQ scipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p/bsr.cxx.o -MF scipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p/bsr.cxx.o.d -o scipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p/bsr.cxx.o -c ../scipy/sparse/sparsetools/bsr.cxx
34.8 11.0                                                         /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/cc1plus -quiet -I scipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p -I scipy/sparse/sparsetools -I ../scipy/sparse/sparsetools -I ../scipy/_build_utils/src -I ../../../../../../../lib/python3.12/site-packages/numpy/core/include -I /usr/include/python3.12 -MD scipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p/bsr.cxx.d -MF scipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p/bsr.cxx.o.d -MQ scipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p/bsr.cxx.o -D_GNU_SOURCE -D NDEBUG -D _FILE_OFFSET_BITS=64 -D NPY_NO_DEPRECATED_API=NPY_1_9_API_VERSION ../scipy/sparse/sparsetools/bsr.cxx -quiet -dumpdir scipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p/ -dumpbase bsr.cxx.cxx -dumpbase-ext .cxx -mtune=generic -march=x86-64 -g -O3 -O2 -Wall -Winvalid-pch -std=c++14 -fdiagnostics-color=always -fvisibility=hidden -fvisibility-inlines-hidden -fPIC -o /tmp/ccoTUpvy.s
 0.0  0.0                                                       g++ -Iscipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p -Iscipy/sparse/sparsetools -I../scipy/sparse/sparsetools -I../scipy/_build_utils/src -I../../../../../../../lib/python3.12/site-packages/numpy/core/include -I/usr/include/python3.12 -fvisibility=hidden -fvisibility-inlines-hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c++14 -O3 -g -O2 -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_9_API_VERSION -MD -MQ scipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p/csr.cxx.o -MF scipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p/csr.cxx.o.d -o scipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p/csr.cxx.o -c ../scipy/sparse/sparsetools/csr.cxx
33.1 12.5                                                         /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/cc1plus -quiet -I scipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p -I scipy/sparse/sparsetools -I ../scipy/sparse/sparsetools -I ../scipy/_build_utils/src -I ../../../../../../../lib/python3.12/site-packages/numpy/core/include -I /usr/include/python3.12 -MD scipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p/csr.cxx.d -MF scipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p/csr.cxx.o.d -MQ scipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p/csr.cxx.o -D_GNU_SOURCE -D NDEBUG -D _FILE_OFFSET_BITS=64 -D NPY_NO_DEPRECATED_API=NPY_1_9_API_VERSION ../scipy/sparse/sparsetools/csr.cxx -quiet -dumpdir scipy/sparse/sparsetools/_sparsetools.cpython-312-x86_64-linux-gnu.so.p/ -dumpbase csr.cxx.cxx -dumpbase-ext .cxx -mtune=generic -march=x86-64 -g -O3 -O2 -Wall -Winvalid-pch -std=c++14 -fdiagnostics-color=always -fvisibility=hidden -fvisibility-inlines-hidden -fPIC -o /tmp/ccAZ1GBR.s
 0.0  0.0                                                       gcc -Iscipy/sparse/csgraph/_traversal.cpython-312-x86_64-linux-gnu.so.p -Iscipy/sparse/csgraph -I../scipy/sparse/csgraph -I../../../../../../../lib/python3.12/site-packages/numpy/core/include -I/usr/include/python3.12 -fvisibility=hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c99 -O3 -Wno-unused-but-set-variable -Wno-unused-function -Wno-conversion -Wno-misleading-indentation -g -O2 -fPIC -DCYTHON_CCOMPLEX=0 -Wno-cpp -MD -MQ scipy/sparse/csgraph/_traversal.cpython-312-x86_64-linux-gnu.so.p/meson-generated__traversal.c.o -MF scipy/sparse/csgraph/_traversal.cpython-312-x86_64-linux-gnu.so.p/meson-generated__traversal.c.o.d -o scipy/sparse/csgraph/_traversal.cpython-312-x86_64-linux-gnu.so.p/meson-generated__traversal.c.o -c scipy/sparse/csgraph/_traversal.cpython-312-x86_64-linux-gnu.so.p/_traversal.c
32.1  2.2                                                         /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/cc1 -quiet -I scipy/sparse/csgraph/_traversal.cpython-312-x86_64-linux-gnu.so.p -I scipy/sparse/csgraph -I ../scipy/sparse/csgraph -I ../../../../../../../lib/python3.12/site-packages/numpy/core/include -I /usr/include/python3.12 -MD scipy/sparse/csgraph/_traversal.cpython-312-x86_64-linux-gnu.so.p/meson-generated__traversal.c.d -MF scipy/sparse/csgraph/_traversal.cpython-312-x86_64-linux-gnu.so.p/meson-generated__traversal.c.o.d -MQ scipy/sparse/csgraph/_traversal.cpython-312-x86_64-linux-gnu.so.p/meson-generated__traversal.c.o -D NDEBUG -D _FILE_OFFSET_BITS=64 -D CYTHON_CCOMPLEX=0 scipy/sparse/csgraph/_traversal.cpython-312-x86_64-linux-gnu.so.p/_traversal.c -quiet -dumpdir scipy/sparse/csgraph/_traversal.cpython-312-x86_64-linux-gnu.so.p/ -dumpbase meson-generated__traversal.c.c -dumpbase-ext .c -mtune=generic -march=x86-64 -g -O3 -O2 -Wall -Winvalid-pch -Wno-unused-but-set-variable -Wno-unused-function -Wno-conversion -Wno-misleading-indentation -Wno-cpp -std=c99 -fdiagnostics-color=always -fvisibility=hidden -fPIC -o /tmp/ccnaDcBb.s
 0.0  0.0                                                       gcc -Iscipy/sparse/linalg/_propack/_zpropack.cpython-312-x86_64-linux-gnu.so.p -Iscipy/sparse/linalg/_propack -I../scipy/sparse/linalg/_propack -I../../../../../../../lib/python3.12/site-packages/numpy/core/include -I../../../../../../../lib/python3.12/site-packages/numpy/f2py/src -I/usr/include/openblas -I/usr/include/python3.12 -fvisibility=hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c99 -O3 -Wno-unused-but-set-variable -Wno-unused-function -Wno-conversion -Wno-misleading-indentation -g -O2 -fPIC -U_OPENMP -Wno-cpp -MD -MQ scipy/sparse/linalg/_propack/_zpropack.cpython-312-x86_64-linux-gnu.so.p/meson-generated_..__zpropackmodule.c.o -MF scipy/sparse/linalg/_propack/_zpropack.cpython-312-x86_64-linux-gnu.so.p/meson-generated_..__zpropackmodule.c.o.d -o scipy/sparse/linalg/_propack/_zpropack.cpython-312-x86_64-linux-gnu.so.p/meson-generated_..__zpropackmodule.c.o -c scipy/sparse/linalg/_propack/_zpropackmodule.c
36.3  0.8                                                         /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/cc1 -quiet -I scipy/sparse/linalg/_propack/_zpropack.cpython-312-x86_64-linux-gnu.so.p -I scipy/sparse/linalg/_propack -I ../scipy/sparse/linalg/_propack -I ../../../../../../../lib/python3.12/site-packages/numpy/core/include -I ../../../../../../../lib/python3.12/site-packages/numpy/f2py/src -I /usr/include/openblas -I /usr/include/python3.12 -MD scipy/sparse/linalg/_propack/_zpropack.cpython-312-x86_64-linux-gnu.so.p/meson-generated_..__zpropackmodule.c.d -MF scipy/sparse/linalg/_propack/_zpropack.cpython-312-x86_64-linux-gnu.so.p/meson-generated_..__zpropackmodule.c.o.d -MQ scipy/sparse/linalg/_propack/_zpropack.cpython-312-x86_64-linux-gnu.so.p/meson-generated_..__zpropackmodule.c.o -D NDEBUG -D _FILE_OFFSET_BITS=64 -U _OPENMP scipy/sparse/linalg/_propack/_zpropackmodule.c -quiet -dumpdir scipy/sparse/linalg/_propack/_zpropack.cpython-312-x86_64-linux-gnu.so.p/ -dumpbase meson-generated_..__zpropackmodule.c.c -dumpbase-ext .c -mtune=generic -march=x86-64 -g -O3 -O2 -Wall -Winvalid-pch -Wno-unused-but-set-variable -Wno-unused-function -Wno-conversion -Wno-misleading-indentation -Wno-cpp -std=c99 -fdiagnostics-color=always -fvisibility=hidden -fPIC -o /tmp/cc6x5G3Y.s
 0.0  0.0                                                       gcc -Iscipy/sparse/linalg/_dsolve/_superlu.cpython-312-x86_64-linux-gnu.so.p -Iscipy/sparse/linalg/_dsolve -I../scipy/sparse/linalg/_dsolve -I../scipy/sparse/linalg/_dsolve/SuperLU/SRC -I../../../../../../../lib/python3.12/site-packages/numpy/core/include -I/usr/include/openblas -I/usr/include/python3.12 -fvisibility=hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c99 -O3 -Wno-unused-but-set-variable -Wno-unused-function -Wno-conversion -Wno-misleading-indentation -g -O2 -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_9_API_VERSION -MD -MQ scipy/sparse/linalg/_dsolve/_superlu.cpython-312-x86_64-linux-gnu.so.p/_superlumodule.c.o -MF scipy/sparse/linalg/_dsolve/_superlu.cpython-312-x86_64-linux-gnu.so.p/_superlumodule.c.o.d -o scipy/sparse/linalg/_dsolve/_superlu.cpython-312-x86_64-linux-gnu.so.p/_superlumodule.c.o -c ../scipy/sparse/linalg/_dsolve/_superlumodule.c
22.5  0.3                                                         /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/cc1 -quiet -I scipy/sparse/linalg/_dsolve/_superlu.cpython-312-x86_64-linux-gnu.so.p -I scipy/sparse/linalg/_dsolve -I ../scipy/sparse/linalg/_dsolve -I ../scipy/sparse/linalg/_dsolve/SuperLU/SRC -I ../../../../../../../lib/python3.12/site-packages/numpy/core/include -I /usr/include/openblas -I /usr/include/python3.12 -MD scipy/sparse/linalg/_dsolve/_superlu.cpython-312-x86_64-linux-gnu.so.p/_superlumodule.c.d -MF scipy/sparse/linalg/_dsolve/_superlu.cpython-312-x86_64-linux-gnu.so.p/_superlumodule.c.o.d -MQ scipy/sparse/linalg/_dsolve/_superlu.cpython-312-x86_64-linux-gnu.so.p/_superlumodule.c.o -D NDEBUG -D _FILE_OFFSET_BITS=64 -D NPY_NO_DEPRECATED_API=NPY_1_9_API_VERSION ../scipy/sparse/linalg/_dsolve/_superlumodule.c -quiet -dumpdir scipy/sparse/linalg/_dsolve/_superlu.cpython-312-x86_64-linux-gnu.so.p/ -dumpbase _superlumodule.c.c -dumpbase-ext .c -mtune=generic -march=x86-64 -g -O3 -O2 -Wall -Winvalid-pch -Wno-unused-but-set-variable -Wno-unused-function -Wno-conversion -Wno-misleading-indentation -std=c99 -fdiagnostics-color=always -fvisibility=hidden -fPIC -o /tmp/ccK98bzw.s
 0.0  0.0                                               bash /sage/build/bin/sage-logger -P spkg-install ./spkg-install
 0.0  0.0                                                 sed --unbuffered s/^/[spkg-install] /
 0.0  0.0                                     bash /sage/build/bin/sage-logger -p SAGE_CHECK=no PATH=/sage/src/bin:/sage/local/var/lib/sage/venv-python3.12/bin:$PATH sage-spkg -o  scipy-1.12.0 /sage/local/var/lib/sage/venv-python3.12 /sage/logs/pkgs/scipy-1.12.0.log
 0.0  0.0                                       tee -a /sage/logs/pkgs/scipy-1.12.0.log
 0.0  0.0                                       bash /sage/build/bin/sage-logger -p SAGE_CHECK=no PATH=/sage/src/bin:/sage/local/var/lib/sage/venv-python3.12/bin:$PATH sage-spkg -o  scipy-1.12.0 /sage/local/var/lib/sage/venv-python3.12 /sage/logs/pkgs/scipy-1.12.0.log
 0.0  0.0                                         sed --unbuffered s/^/[scipy-1.12.0] /
 0.0  0.0                               make --no-print-directory sagelib-SAGE_VENV-no-deps
 0.0  0.0                                 /bin/sh -c if [ -z '/sage/local/var/lib/sage/venv-python3.12' ]; then echo "Error: The installation tree SAGE_VENV has been disabled" 2>&1; echo "" 2>&1; exit 1; else rm -rf '/sage/local/var/lib/sage/venv-python3.12/var/lib/sage/scripts/sagelib'; cd '/sage/build/pkgs/sagelib' && . '/sage/src/bin/sage-src-env-config' && . '/sage/src/bin/sage-env-config' && . '/sage/src/bin/sage-env' && . '/sage/build/bin/sage-build-env-config' && . '/sage/build/bin/sage-build-env' && SAGE_CHECK=no sage-logger -p 'SAGE_CHECK=no PATH=/sage/src/bin:/sage/local/var/lib/sage/venv-python3.12/bin:$PATH sage-spkg -o  sagelib-10.5.beta9 /sage/local/var/lib/sage/venv-python3.12' '/sage/logs/pkgs/sagelib-10.5.beta9.log'; fi
 0.0  0.0                                   bash /sage/build/bin/sage-logger -p SAGE_CHECK=no PATH=/sage/src/bin:/sage/local/var/lib/sage/venv-python3.12/bin:$PATH sage-spkg -o  sagelib-10.5.beta9 /sage/local/var/lib/sage/venv-python3.12 /sage/logs/pkgs/sagelib-10.5.beta9.log
 0.0  0.0                                     bash /sage/build/bin/sage-logger -p SAGE_CHECK=no PATH=/sage/src/bin:/sage/local/var/lib/sage/venv-python3.12/bin:$PATH sage-spkg -o  sagelib-10.5.beta9 /sage/local/var/lib/sage/venv-python3.12 /sage/logs/pkgs/sagelib-10.5.beta9.log
 0.0  0.0                                       bash /sage/build/bin/sage-logger -p SAGE_CHECK=no PATH=/sage/src/bin:/sage/local/var/lib/sage/venv-python3.12/bin:$PATH sage-spkg -o  sagelib-10.5.beta9 /sage/local/var/lib/sage/venv-python3.12 /sage/logs/pkgs/sagelib-10.5.beta9.log
 0.0  0.0                                         bash /sage/build/bin/sage-logger -p SAGE_CHECK=no PATH=/sage/src/bin:/sage/local/var/lib/sage/venv-python3.12/bin:$PATH sage-spkg -o  sagelib-10.5.beta9 /sage/local/var/lib/sage/venv-python3.12 /sage/logs/pkgs/sagelib-10.5.beta9.log
 0.0  0.0                                           bash /sage/build/bin/sage-spkg -o sagelib-10.5.beta9 /sage/local/var/lib/sage/venv-python3.12
 0.0  0.0                                             bash /sage/build/bin/sage-logger -P spkg-install ./spkg-install
 0.0  0.0                                               bash ./spkg-install
 0.2  0.5                                                 python3 -m pip install --verbose --no-deps --no-index --no-build-isolation --isolated --editable .
 1.7  3.2                                                   /sage/local/var/lib/sage/venv-python3.12/bin/python3 -c  exec(compile(''' # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py # # - It imports setuptools before invoking setup.py, to enable projects that directly #   import from `distutils.core` to work with newer packaging standards. # - It provides a clear error message when setuptools is not installed. # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so #   setuptools doesn't think the script is `-c`. This avoids the following warning: #     manifest_maker: standard file '-c' not found". # - It generates a shim setup.py, for handling setup.cfg-only projects. import os, sys, tokenize  try:     import setuptools except ImportError as error:     print(         "ERROR: Can not execute `setup.py` since setuptools is not available in "         "the build environment.",         file=sys.stderr,     )     sys.exit(1)  __file__ = %r sys.argv[0] = __file__  if os.path.exists(__file__):     filename = __file__     with tokenize.open(__file__) as f:         setup_py_code = f.read() else:     filename = "<auto-generated setuptools caller>"     setup_py_code = "from setuptools import setup; setup()"  exec(compile(setup_py_code, filename, "exec")) ''' % ('/sage/src/setup.py',), "<pip-setuptools-caller>", "exec")) --no-user-cfg develop --no-deps
 0.0  0.0                                                     /usr/bin/gcc -shared -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-z,pack-relative-relocs -flto=auto -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-z,pack-relative-relocs -flto=auto -Wl,-rpath-link,/sage/local/lib -L/sage/local/lib -Wl,-rpath,/sage/local/lib -Wl,-rpath-link,/sage/local/lib -L/sage/local/lib -Wl,-rpath,/sage/local/lib -g -O2 build/temp.linux-x86_64-cpython-312/sage/calculus/riemann.o -L/usr/lib -o build/lib.linux-x86_64-cpython-312/sage/calculus/riemann.cpython-312-x86_64-linux-gnu.so -lpari
 0.0  0.0                                                       /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/collect2 -plugin /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/lto-wrapper -plugin-opt=-fresolution=/tmp/cclhLWeQ.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -flto=auto -flto=auto --build-id --eh-frame-hdr --hash-style=gnu -m elf_x86_64 -shared -o build/lib.linux-x86_64-cpython-312/sage/calculus/riemann.cpython-312-x86_64-linux-gnu.so /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../lib/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/crtbeginS.o -L/sage/local/lib -L/sage/local/lib -L/usr/lib -L/sage/local/lib/../lib -L/sage/local/lib/../lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../lib -L/lib/../lib -L/usr/lib/../lib -L/sage/local/lib -L/sage/local/lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/../../.. -O1 --sort-common --as-needed -z relro -z now -z pack-relative-relocs -O1 --sort-common --as-needed -z relro -z now -z pack-relative-relocs -rpath-link /sage/local/lib -rpath /sage/local/lib -rpath-link /sage/local/lib -rpath /sage/local/lib build/temp.linux-x86_64-cpython-312/sage/calculus/riemann.o -lpari -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/crtendS.o /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../lib/crtn.o
 0.0  0.1                                                         /usr/bin/ld -plugin /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/lto-wrapper -plugin-opt=-fresolution=/tmp/cclhLWeQ.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --eh-frame-hdr --hash-style=gnu -m elf_x86_64 -shared -o build/lib.linux-x86_64-cpython-312/sage/calculus/riemann.cpython-312-x86_64-linux-gnu.so /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../lib/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/crtbeginS.o -L/sage/local/lib -L/sage/local/lib -L/usr/lib -L/sage/local/lib/../lib -L/sage/local/lib/../lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../lib -L/lib/../lib -L/usr/lib/../lib -L/sage/local/lib -L/sage/local/lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/../../.. -O1 --sort-common --as-needed -z relro -z now -z pack-relative-relocs -O1 --sort-common --as-needed -z relro -z now -z pack-relative-relocs -rpath-link /sage/local/lib -rpath /sage/local/lib -rpath-link /sage/local/lib -rpath /sage/local/lib build/temp.linux-x86_64-cpython-312/sage/calculus/riemann.o -lpari -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/crtendS.o /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../lib/crtn.o
 0.0  0.0                                                           /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/lto-wrapper @/tmp/ccdLEJEy.lto_wrapper_args
 0.0  0.0                                                             make -f /tmp/ccLdfkCA.mk all
 0.0  0.0                                                               /usr/bin/gcc -xlto -c -fno-openmp -fno-openacc -g -march=x86-64 -mtune=generic -mno-omit-leaf-frame-pointer -O3 -O3 -O3 -O3 -O2 -fcf-protection=full -fcf-protection=full -fcf-protection=full -fPIC -shared -L /sage/local/lib -L /sage/local/lib -g -O2 -L /usr/lib -mtune=generic -march=x86-64 -dumpdir build/lib.linux-x86_64-cpython-312/sage/calculus/riemann.cpython-312-x86_64-linux-gnu.so. -dumpbase build/lib.linux-x86_64-cpython-312/sage/calculus/riemann.cpython-312-x86_64-linux-gnu.so.ltrans0.ltrans -fltrans -o /tmp/ccKkFqyP.ltrans0.ltrans.o /tmp/ccKkFqyP.ltrans0.o
30.2  1.2                                                                 /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/lto1 -quiet -dumpbase build/lib.linux-x86_64-cpython-312/sage/calculus/riemann.cpython-312-x86_64-linux-gnu.so.ltrans0.ltrans -mno-omit-leaf-frame-pointer -mtune=generic -march=x86-64 -g -g -O3 -O3 -O3 -O3 -O2 -O2 -fno-openmp -fno-openacc -fcf-protection=full -fcf-protection=full -fcf-protection=full -fPIC -fltrans /tmp/ccKkFqyP.ltrans0.o -o /tmp/ccwyUcwS.s
 0.0  0.0                                                               /usr/bin/gcc -xlto -c -fno-openmp -fno-openacc -g -march=x86-64 -mtune=generic -mno-omit-leaf-frame-pointer -O3 -O3 -O3 -O3 -O2 -fcf-protection=full -fcf-protection=full -fcf-protection=full -fPIC -shared -L /sage/local/lib -L /sage/local/lib -g -O2 -L /usr/lib -mtune=generic -march=x86-64 -dumpdir build/lib.linux-x86_64-cpython-312/sage/calculus/riemann.cpython-312-x86_64-linux-gnu.so. -dumpbase build/lib.linux-x86_64-cpython-312/sage/calculus/riemann.cpython-312-x86_64-linux-gnu.so.ltrans1.ltrans -fltrans -o /tmp/ccKkFqyP.ltrans1.ltrans.o /tmp/ccKkFqyP.ltrans1.o
28.6  0.7                                                                 /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/lto1 -quiet -dumpbase build/lib.linux-x86_64-cpython-312/sage/calculus/riemann.cpython-312-x86_64-linux-gnu.so.ltrans1.ltrans -mno-omit-leaf-frame-pointer -mtune=generic -march=x86-64 -g -g -O3 -O3 -O3 -O3 -O2 -O2 -fno-openmp -fno-openacc -fcf-protection=full -fcf-protection=full -fcf-protection=full -fPIC -fltrans /tmp/ccKkFqyP.ltrans1.o -o /tmp/cc0S4B8W.s
 0.0  0.0                                                     /usr/bin/gcc -shared -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-z,pack-relative-relocs -flto=auto -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-z,pack-relative-relocs -flto=auto -Wl,-rpath-link,/sage/local/lib -L/sage/local/lib -Wl,-rpath,/sage/local/lib -Wl,-rpath-link,/sage/local/lib -L/sage/local/lib -Wl,-rpath,/sage/local/lib -g -O2 build/temp.linux-x86_64-cpython-312/sage/categories/category_singleton.o -L/usr/lib -o build/lib.linux-x86_64-cpython-312/sage/categories/category_singleton.cpython-312-x86_64-linux-gnu.so
 0.0  0.0                                                       /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/collect2 -plugin /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/lto-wrapper -plugin-opt=-fresolution=/tmp/cczCva65.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -flto=auto -flto=auto --build-id --eh-frame-hdr --hash-style=gnu -m elf_x86_64 -shared -o build/lib.linux-x86_64-cpython-312/sage/categories/category_singleton.cpython-312-x86_64-linux-gnu.so /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../lib/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/crtbeginS.o -L/sage/local/lib -L/sage/local/lib -L/usr/lib -L/sage/local/lib/../lib -L/sage/local/lib/../lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../lib -L/lib/../lib -L/usr/lib/../lib -L/sage/local/lib -L/sage/local/lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/../../.. -O1 --sort-common --as-needed -z relro -z now -z pack-relative-relocs -O1 --sort-common --as-needed -z relro -z now -z pack-relative-relocs -rpath-link /sage/local/lib -rpath /sage/local/lib -rpath-link /sage/local/lib -rpath /sage/local/lib build/temp.linux-x86_64-cpython-312/sage/categories/category_singleton.o -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/crtendS.o /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../lib/crtn.o
 0.0  0.0                                                         /usr/bin/ld -plugin /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/lto-wrapper -plugin-opt=-fresolution=/tmp/cczCva65.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --eh-frame-hdr --hash-style=gnu -m elf_x86_64 -shared -o build/lib.linux-x86_64-cpython-312/sage/categories/category_singleton.cpython-312-x86_64-linux-gnu.so /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../lib/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/crtbeginS.o -L/sage/local/lib -L/sage/local/lib -L/usr/lib -L/sage/local/lib/../lib -L/sage/local/lib/../lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../lib -L/lib/../lib -L/usr/lib/../lib -L/sage/local/lib -L/sage/local/lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/../../.. -O1 --sort-common --as-needed -z relro -z now -z pack-relative-relocs -O1 --sort-common --as-needed -z relro -z now -z pack-relative-relocs -rpath-link /sage/local/lib -rpath /sage/local/lib -rpath-link /sage/local/lib -rpath /sage/local/lib build/temp.linux-x86_64-cpython-312/sage/categories/category_singleton.o -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/crtendS.o /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../lib/crtn.o
 0.0  0.0                                                           /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/lto-wrapper @/tmp/cc9MaA9X.lto_wrapper_args
 0.0  0.0                                                             make -f /tmp/ccPbOvCC.mk all
 0.0  0.0                                                               /usr/bin/gcc -xlto -c -fno-openmp -fno-openacc -g -march=x86-64 -mtune=generic -mno-omit-leaf-frame-pointer -O3 -O3 -O3 -O3 -O2 -fcf-protection=full -fcf-protection=full -fcf-protection=full -fPIC -shared -L /sage/local/lib -L /sage/local/lib -g -O2 -L /usr/lib -mtune=generic -march=x86-64 -dumpdir build/lib.linux-x86_64-cpython-312/sage/categories/category_singleton.cpython-312-x86_64-linux-gnu.so. -dumpbase build/lib.linux-x86_64-cpython-312/sage/categories/category_singleton.cpython-312-x86_64-linux-gnu.so.ltrans0.ltrans -fltrans -o /tmp/ccDJ6rnN.ltrans0.ltrans.o /tmp/ccDJ6rnN.ltrans0.o29.1  0.8                                                                 /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/lto1 -quiet -dumpbase build/lib.linux-x86_64-cpython-312/sage/categories/category_singleton.cpython-312-x86_64-linux-gnu.so.ltrans0.ltrans -mno-omit-leaf-frame-pointer -mtune=generic -march=x86-64 -g -g -O3 -O3 -O3 -O3 -O2 -O2 -fno-openmp -fno-openacc -fcf-protection=full -fcf-protection=full -fcf-protection=full -fPIC -fltrans /tmp/ccDJ6rnN.ltrans0.o -o /tmp/cctXy3oq.s
 0.0  0.0                                                     /usr/bin/gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -ffat-lto-objects -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -g -O2 -fPIC -Isage/cpython -I/sage/src -I/sage/local/var/lib/sage/venv-python3.12/lib/python3.12/site-packages/numpy/core/include -I/usr/include/python3.12 -I/sage/local/var/lib/sage/venv-python3.12/include -I/usr/include/python3.12 -c sage/categories/examples/semigroups_cython.c -o build/temp.linux-x86_64-cpython-312/sage/categories/examples/semigroups_cython.o
50.5  0.5                                                       /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/cc1 -quiet -I sage/cpython -I /sage/src -I /sage/local/var/lib/sage/venv-python3.12/lib/python3.12/site-packages/numpy/core/include -I /usr/include/python3.12 -I /sage/local/var/lib/sage/venv-python3.12/include -I /usr/include/python3.12 -D NDEBUG -D_FORTIFY_SOURCE=3 -D_FORTIFY_SOURCE=3 -D_FORTIFY_SOURCE=3 sage/categories/examples/semigroups_cython.c -quiet -dumpdir build/temp.linux-x86_64-cpython-312/sage/categories/examples/ -dumpbase semigroups_cython.c -dumpbase-ext .c -march=x86-64 -mtune=generic -mno-omit-leaf-frame-pointer -g -g -g -g -g -O3 -O3 -O3 -O3 -O2 -Wsign-compare -Wall -Wformat=1 -Werror=format-security -Wformat=1 -Werror=format-security -Wformat=1 -Werror=format-security -fno-strict-overflow -fcf-protection=full -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -ffat-lto-objects -fcf-protection=full -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -fno-plt -fexceptions -fstack-clash-protection -fcf-protection=full -fno-omit-frame-pointer -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -fPIC -o -
 0.0  0.0                                                       as -I sage/cpython -I /sage/src -I /sage/local/var/lib/sage/venv-python3.12/lib/python3.12/site-packages/numpy/core/include -I /usr/include/python3.12 -I /sage/local/var/lib/sage/venv-python3.12/include -I /usr/include/python3.12 --gdwarf-5 --64 -o build/temp.linux-x86_64-cpython-312/sage/categories/examples/semigroups_cython.o
 0.0  0.0                                               bash /sage/build/bin/sage-logger -P spkg-install ./spkg-install
 0.0  0.0                                                 sed --unbuffered s/^/[spkg-install] /
 0.0  0.0                                     bash /sage/build/bin/sage-logger -p SAGE_CHECK=no PATH=/sage/src/bin:/sage/local/var/lib/sage/venv-python3.12/bin:$PATH sage-spkg -o  sagelib-10.5.beta9 /sage/local/var/lib/sage/venv-python3.12 /sage/logs/pkgs/sagelib-10.5.beta9.log
 0.0  0.0                                       tee -a /sage/logs/pkgs/sagelib-10.5.beta9.log
 0.0  0.0                                       bash /sage/build/bin/sage-logger -p SAGE_CHECK=no PATH=/sage/src/bin:/sage/local/var/lib/sage/venv-python3.12/bin:$PATH sage-spkg -o  sagelib-10.5.beta9 /sage/local/var/lib/sage/venv-python3.12 /sage/logs/pkgs/sagelib-10.5.beta9.log
 0.0  0.0                                         sed --unbuffered s/^/[sagelib-10.5.beta9] /

For reference, at the moment (I don't fully understand how it work) pyproject_hooks is used to invoke build_wheel, which in turn delegates to mesonpy.

def build_wheel(wheel_directory, config_settings, metadata_directory=None):
    """Invoke the mandatory build_wheel hook.

    If a wheel was already built in the
    prepare_metadata_for_build_wheel fallback, this
    will copy it rather than rebuilding the wheel.
    """
    prebuilt_whl = _find_already_built_wheel(metadata_directory)
    if prebuilt_whl:
        shutil.copy2(prebuilt_whl, wheel_directory)
        return os.path.basename(prebuilt_whl)

    return _build_backend().build_wheel(
        wheel_directory, config_settings, metadata_directory
    )

We need to modify config_settings, which are done by modifying hook_input in

    hook = globals()[hook_name]

    hook_input = read_json(pjoin(control_dir, "input.json"))

    json_out = {"unsupported": False, "return_val": None}
    try:
        json_out["return_val"] = hook(**hook_input["kwargs"])

If I understood the code correctly, it suffices to set config_settings={"compile-args": ["-jN"]} (for some integer N), which is done by setting hook_input to {"kwargs": {"config_settings": {"compile_args": ["-jN"]}}}.

I don't know how hook_input is controlled.

Looks like it comes from sdh_build_wheel which calls python3 -m build, and you can somehow pass --config-setting=--compile-args=-jN to it using $# build 1.2.2.post1

which is in turn managed by sage/build/pkgs/cython/spkg-install.in

In top level pyproject.toml there is build-backend = 'mesonpy' specification. So passing -jN to that backend should be safe?

In sage-env, it already sets SAGE_NUM_THREADS and SAGE_NUM_THREADS_PARALLEL environment variables.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
c: build
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@orlitzky @fchapoton @user202729