sagemathgh-37237: CI macOS: Fix failure with macos-13-homebrew, add t…

…ests on M1 runners, add timeouts

<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

- Fixing the failure with `macos-13-homebrew`, as seen in
https://github.com/sagemath/sage/actions/runs/7761817710/job/21171032513
and upstream uses of macos.yml
dimpase/primecountpy#11 (comment),
sagemath/cypari2#152
- Generalizing the `ld_classic` workaround from sagemath#36599 for the case of
full XCode.
  - This is to fix the failure seen on `homebrew-macos-13-usrlocal-
standard-xcode_15.0`, e.g. in https://github.com/scipopt/SCIP-
SDP/actions/runs/7793548242/job/21253452882?pr=9#step:10:2766
- Deactivating the `ld_classic` workaround when `LD` is set, as is in a
conda environment; as seen e.g. in https://groups.google.com/g/sage-
devel/c/1viBzw-ZaoQ/m/S_u9K20xAAAJ
- [Apple Silicon runners recently became
available](https://github.blog/changelog/2024-01-30-github-actions-
introducing-the-new-m1-macos-runner-available-to-open-source/); we add
some tests (example run:
https://github.com/mkoeppe/sage/actions/runs/7772086965/job/21194110510)
- On Apple Silicon, `tox -e local-conda-forge` now uses the correct
installer
- Intel runners were recently upgraded too, increase parallelism.
- Add self-destruct sequence to cancel workflows before the 6h limit
(catches linbox-related build hang)
- Actually handle the input `extra_sage_packages` (for
scipopt/PySCIPOpt#630)

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

Follow-ups, not addressed here:
- conda-forge-macos-latest-minimal tachyon build failure - needs sagemath#36969
- adding a config that tests arm64 conda-forge-minimal - would need
config.guess updates in numerous packages
- arm64 conda-forge-standard - matplotlib build failure (https://github.
com/mkoeppe/sage/actions/runs/7810111886/job/21313997438#step:10:2749)
- hangs or excessive build time that lead to timeouts, probably from
linbox

### :memo: Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### :hourglass: Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->

URL: sagemath#37237
Reported by: Matthias Köppe
Reviewer(s): Matthias Köppe, Tobias Diez

.github/workflows/ci-macos.yml

@@ -34,6 +34,7 @@ jobs:
     uses: ./.github/workflows/macos.yml
     with:
       stage: "1"
+      timeout: 14400
 
   stage-2:
     uses: ./.github/workflows/macos.yml
@@ -90,7 +91,7 @@ jobs:
         run: |
           git config --global user.email "nobody@example.com"
           git config --global user.name "Sage GitHub CI"
-          SAGE_ROOT=. SAGE_SRC=./src src/bin/sage-update-version $(git describe --tags) || echo "(ignoring error)"
+          SAGE_ROOT=. SAGE_SRC=./src src/bin/sage-update-version $(cat src/VERSION.txt).dev0 || echo "(ignoring error)"
       - name: make dist
         run: |
           ./configure --enable-download-from-upstream-url && make dist
@@ -108,7 +109,7 @@ jobs:
       fail-fast: false
       max-parallel: 4
       matrix:
-        os: [ macos-11, macos-12 ]
+        os: [ macos-11, macos-12, macos-14 ]
         tox_system_factor: [macos-nobootstrap]
         tox_packages_factor: [minimal]
         xcode_version_factor: [default]
@@ -129,7 +130,7 @@ jobs:
         if: contains(matrix.tox_system_factor, 'nobootstrap')
       - name: Move homebrew away
         run: |
-          (cd /usr/local && for a in bin etc include lib opt sbin share; do sudo mv $a $a-moved; done)
+          (cd $(brew --prefix) && for a in bin etc include lib opt sbin share; do sudo mv $a $a-moved; done)
       - name: Select Xcode version
         run: |
           if [ ${{ matrix.xcode_version_factor }} != default ]; then sudo xcode-select -s /Applications/Xcode_${{ matrix.xcode_version_factor }}.app; fi
@@ -140,7 +141,8 @@ jobs:
         # We use a high parallelization on purpose in order to catch possible parallelization bugs in the build scripts.
         # For doctesting, we use a lower parallelization to avoid timeouts.
         run: |
-          MAKE="make -j12" tox -e $TOX_ENV -- SAGE_NUM_THREADS=4 $TARGETS
+          (sleep 20000; pkill make) &
+          MAKE="make -j12" tox -e $TOX_ENV -- SAGE_NUM_THREADS=6 $TARGETS
       - name: Prepare logs artifact
         run: |
           mkdir -p "artifacts/$LOGS_ARTIFACT_NAME"; cp -r .tox/*/log "artifacts/$LOGS_ARTIFACT_NAME"

.github/workflows/macos.yml

@@ -19,17 +19,21 @@ on:
         type: string
       # System configuration
       osversion_xcodeversion_toxenv_tuples:
+        # As of 2024-02, "runs-on: macos-latest" is macos-12.
+        # and "runs-on: macos-14" selects the new M1 runners.
+        # https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories
         description: 'Stringified JSON object'
         default: >-
-          [["latest", "",             "homebrew-macos-usrlocal-minimal"],
-           ["latest", "",             "homebrew-macos-usrlocal-standard"],
-           ["11",     "xcode_13.2.1", "homebrew-macos-usrlocal-minimal"],
+          [["11",     "xcode_13.2.1", "homebrew-macos-usrlocal-minimal"],
+           ["12",     "",             "homebrew-macos-usrlocal-minimal"],
            ["12",     "",             "homebrew-macos-usrlocal-standard"],
+           ["12",     "",             "homebrew-macos-usrlocal-python3_xcode-standard"],
+           ["12",     "",             "homebrew-macos-usrlocal-maximal"],
            ["13",     "xcode_15.0",   "homebrew-macos-usrlocal-standard"],
-           ["latest", "",             "homebrew-macos-usrlocal-maximal"],
-           ["latest", "",             "homebrew-macos-usrlocal-python3_xcode-standard"],
+           ["14",     "",             "homebrew-macos-opthomebrew-standard"],
            ["latest", "",             "conda-forge-macos-minimal"],
-           ["latest", "",             "conda-forge-macos-standard"]]
+           ["latest", "",             "conda-forge-macos-standard"],
+           ["14",     "",             "conda-forge-macos-standard"]]
         type: string
       extra_sage_packages:
         description: 'Extra Sage packages to install as system packages'
@@ -41,6 +45,10 @@ on:
       free_disk_space:
         default: false
         type: boolean
+      timeout:
+        description: 'Elapsed time (seconds) at which to kill the build'
+        default: 20000
+        type: number
       #
       # For use in upstream CIs.
       #
@@ -74,10 +82,16 @@ jobs:
           repository: ${{ inputs.sage_repo }}
           ref: ${{ inputs.sage_ref }}
           fetch-depth: 10000
-
+      - uses: actions/setup-python@v5
+        # As of 2024-02-03, the macOS M1 runners do not have preinstalled python or pipx.
+        # Installing pipx follows the approach of https://github.com/pypa/cibuildwheel/pull/1743
+        id: python
+        with:
+          python-version: "3.8 - 3.12"
+          update-environment: false
       - name: Install test prerequisites
         run: |
-          brew install tox
+          "${{ steps.python.outputs.python-path }}" -m pip install pipx
       - name: Download upstream artifact
         uses: actions/download-artifact@v3
         with:
@@ -129,7 +143,8 @@ jobs:
             *)               export TARGETS_PRE="${{ inputs.targets_pre }}" TARGETS="${{ inputs.targets }} TARGETS_OPTIONAL="${{ inputs.targets_optional }}
                              ;;
           esac
-          MAKE="make -j12" tox -e $TOX_ENV -- SAGE_NUM_THREADS=4 $TARGETS
+          (sleep ${{ inputs.timeout }}; pkill make) &
+          MAKE="make -j12" EXTRA_SAGE_PACKAGES="${{ inputs.extra_sage_packages }}" "${{ steps.python.outputs.python-path }}" -m pipx run tox -e $TOX_ENV -- SAGE_NUM_THREADS=6 $TARGETS
       - name: Prepare logs artifact
         run: |
           mkdir -p "artifacts/$LOGS_ARTIFACT_NAME"; cp -r .tox/*/log "artifacts/$LOGS_ARTIFACT_NAME"

build/pkgs/configure/checksums.ini

@@ -1,4 +1,4 @@
 tarball=configure-VERSION.tar.gz
-sha1=a3fc8c3bf9198d28a8f2d5a08cffdcf519150b77
-md5=32f4f1763765232dc7e7431d5e48e129
-cksum=708121316
+sha1=c13ef5d495adc1f4c766bf81b3c3e58e1c6ccbf7
+md5=77d83bf62e2a2c94759b33dc17744299
+cksum=1695718869

build/pkgs/configure/package-version.txt

@@ -1 +1 @@
-0fb793fa91a0b46de452036b521cbbaeee878340
+2a4acca169c99f7b253fc28c41bfaad8165b2567

build/pkgs/dsdp/spkg-install.in

@@ -1,6 +1,7 @@
 cd src
 cp ../patches/CMakeLists.txt .
 sdh_cmake -DCMAKE_BUILD_TYPE=Release \
+          -DCMAKE_VERBOSE_MAKEFILE=ON \
           -DBUILD_SHARED_LIBS=ON \
           -DBLA_VENDOR=OpenBLAS \
           -DBLAS_LIBRARIES="$(pkg-config --libs blas)" \

src/bin/sage-env

@@ -127,7 +127,7 @@ fi
 # The compilers are set in order of priority by
 # 1) environment variables
 # 2) compiler installed by sage
-# 3) compiler set at configuration time 
+# 3) compiler set at configuration time
 if [ -z "$CC" ]; then
     if [ -n "$SAGE_LOCAL" -a -x "$SAGE_LOCAL/bin/gcc" ]; then
         CC=gcc
@@ -373,7 +373,14 @@ if [ -n "$SAGE_LOCAL" ]; then
     # On OS X, test whether "ld-classic" is present in the installed
     # version of the command-line tools. If so, we add "-ld_classic"
     # to LD_FLAGS. See #36599.
-    if [ "$UNAME" = "Darwin" ] && [ -x "$(xcode-select -p)/usr/bin/ld-classic" ] ; then
+    # When the full XCode is installed and in use, for example after
+    # "sudo xcode-select -s /Applications/Xcode.app", then "xcode-select -p"
+    # gives "/Applications/Xcode.app/Contents/Developer", but "ld-classic"
+    # is not in the subdirectory "usr/bin/" but rather in the subdirectory
+    # "Toolchains/XcodeDefault.xctoolchain/usr/bin/". See #37237.
+    # However, if LD is set explicitly, as it is within conda on macOS,
+    # do not not do this.
+    if [ "$UNAME" = "Darwin" ] && [ -z "$LD" ] && [ -x "$(xcode-select -p)/usr/bin/ld-classic" -o -x "$(xcode-select -p)/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld-classic" ] ; then
         LDFLAGS="-L$SAGE_LOCAL/lib -Wl,-ld_classic,-rpath,$SAGE_LOCAL/lib $LDFLAGS"
     else
         LDFLAGS="-L$SAGE_LOCAL/lib -Wl,-rpath,$SAGE_LOCAL/lib $LDFLAGS"

tox.ini

@@ -512,6 +512,9 @@ setenv =
     # brew caches downloaded files in ${HOME}/Library/Caches. We share it between different toxenvs.
     homebrew:         SYSTEM=homebrew
     local-homebrew:                        HOMEBREW={envdir}/homebrew
+    # /opt/homebrew is the default install location on arm64 macOS
+    local-homebrew-opthomebrew:            HOMEBREW=/opt/homebrew
+    # /usr/local is the default install location on x86_64 macOS
     local-{homebrew-usrlocal,nohomebrew}:  HOMEBREW=/usr/local
     # local-macos-nohomebrew: "best effort" isolation to avoid using a homebrew installation in /usr/local
     # We use liblzma from the macOS system - which is available but its headers are not (neither is the xz executable).
@@ -538,10 +541,11 @@ setenv =
     local-conda:      PATH={env:CONDA_PREFIX}/bin:/usr/bin:/bin:/usr/sbin:/sbin
     local-conda:      CONDA_PKGS_DIRS={env:SHARED_CACHE_DIR}/conda_pkgs
     local-conda:      CONDA_OS=$(uname | sed 's/^Darwin/MacOSX/;')
+    local-conda:      CONDA_ARCH=$(uname -m)
     local-conda-forge:     CONDA_INSTALLER_URL_BASE=https://github.com/conda-forge/miniforge/releases/latest/download/
-    local-conda-forge:         CONDA_INSTALLER_FILE=Miniforge3-{env:CONDA_OS}-x86_64.sh
+    local-conda-forge:         CONDA_INSTALLER_FILE=Miniforge3-{env:CONDA_OS}-{env:CONDA_ARCH}.sh
     local-conda-miniconda: CONDA_INSTALLER_URL_BASE=https://repo.anaconda.com/miniconda/
-    local-conda-miniconda:     CONDA_INSTALLER_FILE=Miniconda3-latest-{env:CONDA_OS}-x86_64.sh
+    local-conda-miniconda:     CONDA_INSTALLER_FILE=Miniconda3-latest-{env:CONDA_OS}-{env:CONDA_ARCH}.sh
     local-conda:                  SETENV=. {env:CONDA_PREFIX}/bin/activate base
     local-conda-environment:          CONDA_SAGE_ENVIRONMENT=sage-build
     local-conda-environment:            CONDA_SAGE_ENVIRONMENT_DIR=