Skip to content

Commit

Permalink
sagemathgh-38518: CI Linux: Consolidate jobs
Browse files Browse the repository at this point in the history
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Thanks to the more powerful runners in use since early 2024, we can do
away with the splits between
- `standard-pre` and `standard`
- `minimal-pre` and `minimal`
- `optional-0-o` and `optional-p-z`
- `experimental-0-o` and `experimental-p-z`

This makes the CI runs a bit easier to navigate for those who look at
them.

- Test run: https://github.com/mkoeppe/sage/actions/runs/10438362191

This should also fix "No space left on device" aborts as seen for
example in https://github.com/mkoeppe/sage/actions/runs/10426436105 for
standard-pre (where we did not use to maximize the available disk
space).

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [x] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation and checked the documentation
preview.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->

- Depends on sagemath#37434 (merged here)
    
URL: sagemath#38518
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee, Matthias Köppe
  • Loading branch information
Release Manager committed Aug 26, 2024
2 parents d7de426 + ee30cbf commit 043bc09
Showing 1 changed file with 20 additions and 84 deletions.
104 changes: 20 additions & 84 deletions .github/workflows/ci-linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ permissions:

jobs:

# standard-pre and standard (without ptest) for the default platform (used by build.yml etc.)
# standard (without ptest) for the default platform (used by build.yml etc.)
default:
uses: ./.github/workflows/docker.yml
with:
Expand All @@ -54,45 +54,27 @@ jobs:
# All platforms. This duplicates the default platform, but why not,
# it makes it more robust regarding random timeouts.

standard-pre:
if: ${{ success() || failure() }}
uses: ./.github/workflows/docker.yml
with:
# Build from scratch
docker_targets: "with-system-packages configured with-targets-pre"
# FIXME: duplicated from env.TARGETS
targets_pre: all-sage-local
tox_packages_factors: >-
["standard"]
docker_push_repository: ghcr.io/${{ github.repository }}/
# Make sure that all "standard-pre" jobs can start simultaneously,
# so that runners are available by the time that "default" starts.
max_parallel: 50

standard:
if: ${{ success() || failure() }}
needs: [standard-pre]
uses: ./.github/workflows/docker.yml
with:
# Build incrementally from previous stage (pre)
incremental: true
# Build from scratch
free_disk_space: true
from_docker_repository: ghcr.io/${{ github.repository }}/
from_docker_target: "with-targets-pre"
docker_targets: "with-targets with-targets-optional"
docker_targets: "with-system-packages configured with-targets-pre with-targets with-targets-optional"
# FIXME: duplicated from env.TARGETS
targets_pre: all-sage-local
targets: build doc-html
targets_optional: ptest
tox_packages_factors: >-
["standard"]
docker_push_repository: ghcr.io/${{ github.repository }}/
# Reduce from 30 to 20 because it runs in parallel with 'standard-sitepackages'
# and 'minimal-pre' below
max_parallel: 20
# Make sure that all "standard" jobs can start simultaneously,
# so that runners are available by the time that "default" starts.
max_parallel: 50

standard-constraints_pkgs-norequirements:
if: ${{ success() || failure() }}
needs: [standard-pre]
needs: [standard]
uses: ./.github/workflows/docker.yml
with:
# Build incrementally from previous stage (pre)
Expand All @@ -108,7 +90,7 @@ jobs:

standard-sitepackages:
if: ${{ success() || failure() }}
needs: [standard-pre]
needs: [standard]
uses: ./.github/workflows/docker.yml
with:
# Build incrementally from previous stage (pre)
Expand Down Expand Up @@ -144,38 +126,22 @@ jobs:
docker_push_repository: ghcr.io/${{ github.repository }}/
max_parallel: 8

minimal-pre:
if: ${{ success() || failure() }}
uses: ./.github/workflows/docker.yml
with:
# Build from scratch
docker_targets: "with-system-packages configured with-targets-pre"
# FIXME: duplicated from env.TARGETS
targets_pre: all-sage-local
tox_packages_factors: >-
["minimal"]
docker_push_repository: ghcr.io/${{ github.repository }}/
# Reduced from 30 because it may run in parallel with 'standard' and 'standard-sitepackages' above.
# Calibrated for clogging the job pipeline until the "default" job has finished.
max_parallel: 24

minimal:
if: ${{ success() || failure() }}
needs: [minimal-pre]
uses: ./.github/workflows/docker.yml
with:
# Build incrementally from previous stage (pre)
incremental: true
# Build from scratch
free_disk_space: true
from_docker_repository: ghcr.io/${{ github.repository }}/
from_docker_target: "with-targets-pre"
docker_targets: "with-targets with-targets-optional"
docker_targets: "with-system-packages configured with-targets-pre with-targets with-targets-optional"
# FIXME: duplicated from env.TARGETS
targets_pre: all-sage-local
targets: build doc-html
targets_optional: ptest
tox_packages_factors: >-
["minimal"]
docker_push_repository: ghcr.io/${{ github.repository }}/
# Reduced from 30 because it may run in parallel with 'standard' and 'standard-sitepackages' above.
# Calibrated for clogging the job pipeline until the "default" job has finished.
max_parallel: 24

maximal-pre:
Expand All @@ -192,7 +158,7 @@ jobs:
["maximal"]
docker_push_repository: ghcr.io/${{ github.repository }}/

optional-0-o:
optional:
if: ${{ success() || failure() }}
needs: [maximal-pre]
uses: ./.github/workflows/docker.yml
Expand All @@ -204,42 +170,12 @@ jobs:
tox_packages_factors: >-
["maximal"]
docker_targets: "with-targets-optional"
# [0-9a-o] excludes _, in particular package _develop
targets_optional: '$(echo $(export PATH=build/bin:$PATH && sage-package list :optional: --has-file "spkg-install.in|spkg-install|requirements.txt" --no-file "huge|has_nonfree_dependencies" | grep -v sagemath_doc | grep ^[0-9a-o]))'


optional-p-z:
if: ${{ success() || failure() }}
needs: [optional-0-o]
uses: ./.github/workflows/docker.yml
with:
incremental: true
free_disk_space: true
from_docker_repository: ghcr.io/${{ github.repository }}/
from_docker_target: "with-targets-pre"
tox_packages_factors: >-
["maximal"]
docker_targets: "with-targets-optional"
# [0-9a-o] excludes _, in particular package _develop
targets_optional: '$(echo $(export PATH=build/bin:$PATH && sage-package list :optional: --has-file "spkg-install.in|spkg-install|requirements.txt" --no-file "huge|has_nonfree_dependencies" | grep -v sagemath_doc | grep ^[p-z]))'

experimental-0-o:
if: ${{ success() || failure() }}
needs: [optional-p-z]
uses: ./.github/workflows/docker.yml
with:
incremental: true
free_disk_space: true
from_docker_repository: ghcr.io/${{ github.repository }}/
from_docker_target: "with-targets-pre"
tox_packages_factors: >-
["maximal"]
docker_targets: "with-targets-optional"
targets_optional: '$(echo $(export PATH=build/bin:$PATH && sage-package list :experimental: --has-file "spkg-install.in|spkg-install|requirements.txt" --no-file "huge|has_nonfree_dependencies" | grep -v sagemath_doc | grep ^[0-9a-o]))'
# We remove packages starting with _, in particular package _develop
targets_optional: '$(echo $(export PATH=build/bin:$PATH && sage-package list :optional: --has-file "spkg-install.in|spkg-install|requirements.txt" --no-file "huge|has_nonfree_dependencies" | grep -v sagemath_doc | grep -v ^_))'

experimental-p-z:
experimental:
if: ${{ success() || failure() }}
needs: [experimental-0-o]
needs: [optional]
uses: ./.github/workflows/docker.yml
with:
incremental: true
Expand All @@ -249,4 +185,4 @@ jobs:
tox_packages_factors: >-
["maximal"]
docker_targets: "with-targets-optional"
targets_optional: '$(echo $(export PATH=build/bin:$PATH && sage-package list :experimental: --has-file "spkg-install.in|spkg-install|requirements.txt" --no-file "huge|has_nonfree_dependencies" | grep -v sagemath_doc | grep ^[p-z]))'
targets_optional: '$(echo $(export PATH=build/bin:$PATH && sage-package list :experimental: --has-file "spkg-install.in|spkg-install|requirements.txt" --no-file "huge|has_nonfree_dependencies" | grep -v sagemath_doc))'

0 comments on commit 043bc09

Please sign in to comment.