Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up CI #2019

Merged
merged 15 commits into from
Oct 26, 2022
Merged

Speed up CI #2019

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
3fdb881
ninjaed test-everything
boeschf Oct 25, 2022
c46b407
ninjaed benchmark and sanitize
boeschf Oct 25, 2022
c977cd8
verbose flag
boeschf Oct 25, 2022
a921d0a
ccache trial
boeschf Oct 25, 2022
580a6b5
different ccache action
boeschf Oct 25, 2022
a20d57b
use vector extensions as part of ccache key
boeschf Oct 25, 2022
bfee8b1
github env
boeschf Oct 25, 2022
38418d6
lscpu on macos
boeschf Oct 25, 2022
e2187c1
os conditional
boeschf Oct 25, 2022
932fc4f
added comments
boeschf Oct 25, 2022
a5ba157
ccache for sanitize
boeschf Oct 25, 2022
c33f4a1
ccache for benchmarks
boeschf Oct 25, 2022
e318d08
added comments
boeschf Oct 25, 2022
bf3d69f
re-run
boeschf Oct 25, 2022
121bee4
Merge remote-tracking branch 'upstream/master' into ninja
boeschf Oct 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions .github/workflows/benchmarks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ jobs:
CC: gcc-11
CXX: g++-11
steps:
- name: Get build dependencies
run: |
sudo apt-get update
sudo apt-get install -y ninja-build ccache
- name: Set up cmake
uses: jwlawson/actions-setup-cmake@v1.13
with:
Expand All @@ -22,12 +26,25 @@ jobs:
uses: actions/checkout@v3
with:
submodules: recursive
# figure out vector extensions for ccache key
- name: Check vector extensions
run: |
HAS_AVX512F=$([[ $(lscpu | grep "avx512f" | wc -l) -eq 1 ]] && echo "_avx512f" || echo "")
HAS_AVX2=$([[ $(lscpu | grep "avx2" | wc -l) -eq 1 ]] && echo "_avx2" || echo "")
HAS_FMA=$([[ $(lscpu | grep "fma" | wc -l) -eq 1 ]] && echo "_fma" || echo "")
HAS_AVX=$([[ $(lscpu | grep "avx" | wc -l) -eq 1 ]] && echo "_avx" || echo "")
VECTOR_EXTENSIONS=${HAS_AVX512F}${HAS_AVX2}${HAS_FMA}${HAS_AVX}
echo "VECTOR_EXTENSIONS=$VECTOR_EXTENSIONS" >> $GITHUB_ENV
- name: Setup ccache
uses: hendrikmuhs/ccache-action@v1.2
with:
key: benchmarks-${{ env.CXX }}-${{ env.VECTOR_EXTENSIONS }}
- name: Build arbor
run: |
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_C_COMPILER=$CC -DARB_WITH_MPI=OFF -DARB_USE_BUNDLED_LIBS=ON
make -j4 ubenches
cmake .. -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_C_COMPILER=$CC -DARB_WITH_MPI=OFF -DARB_USE_BUNDLED_LIBS=ON
ninja -j4 ubenches
cd -
- name: Run benchmarks
run: |
Expand Down
22 changes: 20 additions & 2 deletions .github/workflows/sanitize.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,10 @@ jobs:
CXX: clang++-14
ASAN_OPTIONS: detect_leaks=1
steps:
- name: Get build dependencies
run: |
sudo apt-get update
sudo apt-get install -y ninja-build ccache
- name: Set up cmake
uses: jwlawson/actions-setup-cmake@v1.13
with:
Expand All @@ -29,13 +33,27 @@ jobs:
submodules: recursive
- name: Update pip
run: python -m pip install --upgrade pip
# figure out vector extensions for ccache key
- name: Check vector extensions
run: |
HAS_AVX512F=$([[ $(lscpu | grep "avx512f" | wc -l) -eq 1 ]] && echo "_avx512f" || echo "")
HAS_AVX2=$([[ $(lscpu | grep "avx2" | wc -l) -eq 1 ]] && echo "_avx2" || echo "")
HAS_FMA=$([[ $(lscpu | grep "fma" | wc -l) -eq 1 ]] && echo "_fma" || echo "")
HAS_AVX=$([[ $(lscpu | grep "avx" | wc -l) -eq 1 ]] && echo "_avx" || echo "")
VECTOR_EXTENSIONS=${HAS_AVX512F}${HAS_AVX2}${HAS_FMA}${HAS_AVX}
echo "VECTOR_EXTENSIONS=$VECTOR_EXTENSIONS" >> $GITHUB_ENV
- name: Setup ccache
uses: hendrikmuhs/ccache-action@v1.2
with:
key: sanitize-${{ env.CXX }}-${{ matrix.sanitizer }}-${{ env.VECTOR_EXTENSIONS }}
- name: Build arbor
run: |
export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH"
mkdir build
cd build
export SAN="-fsanitize=${{ matrix.sanitizer }} -fno-omit-frame-pointer"
cmake .. -DCMAKE_BUILD_TYPE=debug -DCMAKE_CXX_FLAGS="$SAN" -DCMAKE_C_FLAGS="$SAN" -DCMAKE_EXE_LINKER_FLAGS="$SAN" -DCMAKE_MODULE_LINKER_FLAGS="$SAN" -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_C_COMPILER=$CC -DARB_VECTORIZE=${{ matrix.simd }} -DARB_WITH_MPI=OFF -DARB_USE_BUNDLED_LIBS=ON -DARB_WITH_PYTHON=ON -DPython3_EXECUTABLE=`which python`
make -j4 VERBOSE=1 tests examples pyarb
cmake .. -GNinja -DCMAKE_BUILD_TYPE=debug -DCMAKE_CXX_FLAGS="$SAN" -DCMAKE_C_FLAGS="$SAN" -DCMAKE_EXE_LINKER_FLAGS="$SAN" -DCMAKE_MODULE_LINKER_FLAGS="$SAN" -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_C_COMPILER=$CC -DARB_VECTORIZE=${{ matrix.simd }} -DARB_WITH_MPI=OFF -DARB_USE_BUNDLED_LIBS=ON -DARB_WITH_PYTHON=ON -DPython3_EXECUTABLE=`which python`
ninja -j4 -v tests examples pyarb
cd -
- name: Run unit tests
run: |
Expand Down
35 changes: 28 additions & 7 deletions .github/workflows/test-everything.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,11 +97,11 @@ jobs:
if: ${{ startsWith(matrix.config.os, 'ubuntu') }}
run: |
sudo apt-get update
sudo apt-get install -y libxml2-dev libopenmpi-dev ${{ matrix.config.cc }}
sudo apt-get install -y libxml2-dev libopenmpi-dev ninja-build ccache ${{ matrix.config.cc }}
- name: "MacOS: get build dependencies"
if: ${{ startsWith(matrix.config.os, 'macos') }}
run: |
brew install libxml2 openmpi
brew install libxml2 openmpi ninja ccache
- name: Set up cmake
uses: jwlawson/actions-setup-cmake@v1.13
with:
Expand All @@ -118,6 +118,25 @@ jobs:
uses: actions/checkout@v3
with:
submodules: recursive
# figure out vector extensions for ccache key
- if: ${{ contains(matrix.config.os, 'macos') }}
name: Check vector extensions
# assume uniform hardware for macos
run: |
echo "VECTOR_EXTENSIONS=" >> $GITHUB_ENV
- if: ${{ contains(matrix.config.os, 'ubuntu') }}
name: Check vector extensions
run: |
HAS_AVX512F=$([[ $(lscpu | grep "avx512f" | wc -l) -eq 1 ]] && echo "_avx512f" || echo "")
HAS_AVX2=$([[ $(lscpu | grep "avx2" | wc -l) -eq 1 ]] && echo "_avx2" || echo "")
HAS_FMA=$([[ $(lscpu | grep "fma" | wc -l) -eq 1 ]] && echo "_fma" || echo "")
HAS_AVX=$([[ $(lscpu | grep "avx" | wc -l) -eq 1 ]] && echo "_avx" || echo "")
VECTOR_EXTENSIONS=${HAS_AVX512F}${HAS_AVX2}${HAS_FMA}${HAS_AVX}
echo "VECTOR_EXTENSIONS=$VECTOR_EXTENSIONS" >> $GITHUB_ENV
- name: Setup ccache
uses: hendrikmuhs/ccache-action@v1.2
with:
key: ${{ matrix.config.os }}-${{ matrix.config.cxx }}-${{ matrix.variant }}-${{ env.VECTOR_EXTENSIONS }}
- name: Check config
run: |
$CC --version
Expand All @@ -129,23 +148,25 @@ jobs:
- if: ${{ matrix.variant == 'static' }}
name: Build arbor
run: |
export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH"
mkdir build
cd build
cmake .. -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_C_COMPILER=$CC -DARB_WITH_PYTHON=ON -DARB_VECTORIZE=${{ matrix.config.simd }} -DPython3_EXECUTABLE=`which python` -DARB_WITH_MPI=${{ matrix.config.mpi }} -DARB_USE_BUNDLED_LIBS=ON -DARB_WITH_NEUROML=ON
make -j4 tests examples pyarb html
cmake .. -GNinja -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_C_COMPILER=$CC -DARB_WITH_PYTHON=ON -DARB_VECTORIZE=${{ matrix.config.simd }} -DPython3_EXECUTABLE=`which python` -DARB_WITH_MPI=${{ matrix.config.mpi }} -DARB_USE_BUNDLED_LIBS=ON -DARB_WITH_NEUROML=ON
ninja -j4 tests examples pyarb html
cd -
- if: ${{ matrix.variant == 'shared' }}
name: Build arbor
run: |
export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH"
mkdir build
cd build
cmake .. -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_C_COMPILER=$CC -DARB_WITH_PYTHON=ON -DARB_VECTORIZE=${{ matrix.config.simd }} -DPython3_EXECUTABLE=`which python` -DARB_WITH_MPI=${{ matrix.config.mpi }} -DARB_USE_BUNDLED_LIBS=ON -DARB_WITH_NEUROML=ON -DBUILD_SHARED_LIBS=ON
make -j4 tests examples pyarb html
cmake .. -GNinja -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_C_COMPILER=$CC -DARB_WITH_PYTHON=ON -DARB_VECTORIZE=${{ matrix.config.simd }} -DPython3_EXECUTABLE=`which python` -DARB_WITH_MPI=${{ matrix.config.mpi }} -DARB_USE_BUNDLED_LIBS=ON -DARB_WITH_NEUROML=ON -DBUILD_SHARED_LIBS=ON
ninja -j4 tests examples pyarb html
cd -
- name: Install arbor
run: |
cd build
sudo make install
sudo ninja install
cd -
- name: Run unit tests
run: |
Expand Down