Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specify use of self-hosted runners for github CI. #1387

Open
wants to merge 21 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
02b45c8
Specify use of self-hosted runners for github CI.
kquick Jul 21, 2021
d22b5f1
Use self hosted runner for ubuntu build job.
kquick Jul 21, 2021
0d89da5
Update label for self-hosted; no other labels currently assigned.
kquick Jul 21, 2021
5da9420
Add a better name for an interim step.
kquick Jul 21, 2021
f045956
Add system dependency initialization for Linux in case of self-hosted…
kquick Jul 21, 2021
6930777
Run CI s2n proofs on internal self-hosted runners.
kquick Jul 22, 2021
7b74cc1
Add names for various CI steps.
kquick Jul 23, 2021
74c2ac8
Modify CI upload to trigger on os type, not distribution name.
kquick Jul 23, 2021
39bff81
Add CI system dependencies for the "Test s2n proofs" jobs.
kquick Jul 23, 2021
5b24d3f
The s2n tests don't depend on GHC build version: run only for one build.
kquick Jul 23, 2021
e811539
Add libtinfo6 system dependency for s2n tests CI job.
kquick Jul 23, 2021
2a89736
Update CI workflow to reference uploads/downloads by runner OS for st…
kquick Jul 23, 2021
c008092
Add build-essential packages to s2n CI deployment.
kquick Jul 27, 2021
8642dbb
Add load-path debug for s2n testing under CI/docker-compose.
kquick Jul 27, 2021
676dab8
Add libtinfo6 to s2n docker configuration to allow running saw.
kquick Jul 27, 2021
0a85b79
Update s2n testing docker image to ubuntu:20.04 to match (self-hosted…
kquick Jul 27, 2021
ad6cdb4
Skip Windows builds in CI for collecting built cabal tests for later …
kquick Jul 28, 2021
72e12ea
CI consolidation on ubuntu 20.04 (from 18.04) to match internal runners.
kquick Jul 28, 2021
bc99ad1
[s2nTests/docker] Use clang/llvm v5; Ubuntu 20.04 removed 3.9
kquick Jul 28, 2021
fc70554
[s2nTests/docker] use clang/llvm version 10
kquick Jul 28, 2021
eddcdbf
Use static executable builds to resolve OS inconsistencies in shared …
kquick Jul 30, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/ci.sh
Original file line number Diff line number Diff line change
Expand Up @@ -117,11 +117,11 @@ build() {
pkgs=(saw saw-remote-api)
fi
tee -a cabal.project.local > /dev/null < cabal.project.ci
if ! retry cabal v2-build "$@" "${pkgs[@]}"; then
if ! retry cabal v2-build --disable-executable-dynamic --enable-executable-static "$@" "${pkgs[@]}"; then
if [[ "$RUNNER_OS" == "macOS" ]]; then
echo "Working around a dylib issue on macos by removing the cache and trying again"
cabal v2-clean
retry cabal v2-build "$@" "${pkgs[@]}"
retry cabal v2-build --disable-executable-dynamic --enable-executable-static "$@" "${pkgs[@]}"
else
return 1
fi
Expand Down
45 changes: 35 additions & 10 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ env:

jobs:
config:
runs-on: ubuntu-latest
runs-on: self-hosted
outputs:
name: ${{ steps.config.outputs.name }}
version: ${{ steps.config.outputs.version }}
Expand Down Expand Up @@ -60,7 +60,7 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [ubuntu-18.04]
os: [self-hosted]
ghc: ["8.8.4", "8.10.3"]
include:
- os: macos-latest
Expand All @@ -71,6 +71,15 @@ jobs:
cabal-test-suites-json: ${{ steps.cabal-test-suites.outputs.targets-json }}
steps:

- name: System Dependencies
if: runner.os == 'Linux'
run: |
sudo apt update
sudo apt install -y software-properties-common apt-transport-https ca-certificates wget curl
sudo apt install -y zlibc zlib1g zlib1g-dev git build-essential language-pack-en-base language-pack-en
sudo locale-gen en_US.UTF-8
sudo update-locale LANG=$LANG LANGUAGE=$LANGUAGE

- uses: actions/checkout@v2
- run: |
git submodule update --init
Expand Down Expand Up @@ -99,6 +108,7 @@ jobs:
${{ env.CACHE_VERSION }}-cabal-${{ runner.os }}-${{ matrix.ghc }}-${{ hashFiles(format('cabal.GHC-{0}.config', matrix.ghc)) }}-

- if: needs.config.outputs.release == 'true'
name: Set release version
shell: bash
run: |
sed -i.bak \
Expand All @@ -116,6 +126,7 @@ jobs:
run: .github/ci.sh build

- uses: GaloisInc/.github/actions/cabal-collect-bins@v1
if: runner.os != 'Windows'
id: cabal-test-suites
with:
targets: |
Expand All @@ -127,15 +138,17 @@ jobs:
matrix.ghc == '8.8.4' &&
runner.os != 'Windows'
uses: actions/upload-artifact@v2
name: Upload abc
with:
path: ${{ steps.abc.outputs.path }}
name: abc-${{ runner.os }}

- uses: actions/upload-artifact@v2
name: Upload dist-tests
if: "matrix.ghc == '8.8.4'"
with:
path: dist-tests
name: dist-tests-${{ matrix.os }}
name: dist-tests-${{ runner.os }}

- shell: bash
run: .github/ci.sh setup_dist_bins
Expand All @@ -154,6 +167,7 @@ jobs:
run: .github/ci.sh sign $NAME.tar.gz

- if: matrix.ghc == '8.8.4'
name: Upload ${{ steps.config.outputs.name }}.tar.gz
uses: actions/upload-artifact@v2
with:
name: ${{ steps.config.outputs.name }} (GHC ${{ matrix.ghc }})
Expand All @@ -163,12 +177,14 @@ jobs:

- if: "matrix.ghc == '8.8.4'"
uses: actions/upload-artifact@v2
name: Upload ${{ runner.os }} dist-bin
with:
path: dist/bin
name: ${{ runner.os }}-bins

- if: "matrix.os == 'ubuntu-18.04'"
- if: "runner.os == 'Linux'"
uses: actions/upload-artifact@v2
name: Upload saw-${{ runner.os }}-${{ matrix.ghc }}
with:
name: "saw-${{ runner.os }}-${{ matrix.ghc }}"
path: "dist/bin/saw"
Expand Down Expand Up @@ -215,11 +231,11 @@ jobs:
matrix:
include:
- test: saw-remote-api/scripts/run_rpc_tests.sh
os: ubuntu-18.04
os: ubuntu-20.04
- test: saw-remote-api/scripts/run_rpc_tests.sh
os: macos-latest
- test: saw-remote-api/scripts/check_docs.sh
os: ubuntu-18.04
os: ubuntu-20.04
steps:
- uses: actions/checkout@v2
with:
Expand Down Expand Up @@ -267,7 +283,7 @@ jobs:
fail-fast: false
matrix:
suite: ${{ fromJson(needs.build.outputs.cabal-test-suites-json) }}
os: [ubuntu-18.04]
os: [ubuntu-20.04]
continue-on-error: [false]
include:
- suite: integration_tests
Expand Down Expand Up @@ -312,7 +328,7 @@ jobs:

- uses: actions/download-artifact@v2
with:
name: dist-tests-${{ matrix.os }}
name: dist-tests-${{ runner.os }}
path: dist-tests

- shell: bash
Expand Down Expand Up @@ -435,7 +451,7 @@ jobs:
name: "Test s2n proofs"
timeout-minutes: 60
needs: build
runs-on: ubuntu-18.04
runs-on: self-hosted
strategy:
fail-fast: false
matrix:
Expand All @@ -448,8 +464,17 @@ jobs:
- hmac-failure
- awslc
- blst
ghc: ["8.8.4", "8.10.3"]
ghc: ["8.10.3"]
steps:
- name: System Dependencies
if: runner.os == 'Linux'
run: |
sudo apt update
sudo apt install -y software-properties-common apt-transport-https ca-certificates wget curl
sudo apt install -y docker-compose build-essential libtinfo6 language-pack-en-base language-pack-en
sudo locale-gen en_US.UTF-8
sudo update-locale LANG=$LANG LANGUAGE=$LANGUAGE

- uses: actions/checkout@v2
- run: |
mkdir -p s2nTests/bin
Expand Down
16 changes: 13 additions & 3 deletions s2nTests/docker/s2n.dockerfile
Original file line number Diff line number Diff line change
@@ -1,14 +1,24 @@
FROM ubuntu:18.04
# Note: this Dockerfile will be used to run the saw binary built by
# the CI process, which means that the loadable libraries should be
# the same between the two systems. This stipulates that the "FROM"
# instance specified here should match the OS image used to perform
# the build stage (whether that's a Github runner or a self-hosted
# runner).
#
# A common symptom of these not matching is that the bin/saw image
# cannot be run because libtinfoX.so is not found.

FROM ubuntu:20.04

RUN apt-get update -y -q && \
apt-get install -y software-properties-common && \
apt-get update -q -y && \
apt install -y \
clang-3.9 \
clang-10 \
curl \
gcc \
git \
llvm-3.9 \
llvm-10 \
make \
sudo \
&& \
Expand Down
2 changes: 2 additions & 0 deletions s2nTests/scripts/s2n-entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,7 @@ source codebuild/bin/s2n_setup_env.sh
SAW=true SAW_INSTALL_DIR=tmp-saw codebuild/bin/s2n_install_test_dependencies.sh
cp /saw-bin/saw "$SAW_INSTALL_DIR"/bin/saw
cp /saw-bin/abc "$SAW_INSTALL_DIR"/bin/abc
ldd "$SAW_INSTALL_DIR"/bin/saw
"$SAW_INSTALL_DIR"/bin/saw --version
ldd "$SAW_INSTALL_DIR"/bin/abc
exec codebuild/bin/s2n_codebuild.sh