Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Devops ci/move fuzz to reusable workflow #133

Merged
merged 9 commits into from
Sep 5, 2022
Merged

Devops ci/move fuzz to reusable workflow #133

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
f48eceb
Update ci_pipeline.yml
nathan-bo Aug 23, 2022
c04027b
Update test_fuzz.yml
nathan-bo Aug 23, 2022
b1a03f1
removed sudo from tee, fixed failure and swithed network hotfix position
nathan-bo Sep 4, 2022
bbac2d7
added group in case fuzz fails
nathan-bo Sep 4, 2022
894ab3a
added github hash to log name
nathan-bo Sep 4, 2022
2a42fb1
fixed naming for log files to upload to s3
nathan-bo Sep 4, 2022
9c263d8
changed upload to s3 only on failure
nathan-bo Sep 5, 2022
f3ea638
changed fuzz reusable workflow to local syntax
nathan-bo Sep 5, 2022
6790e4a
changed syntax for run on failure of fuzz
nathan-bo Sep 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 4 additions & 103 deletions .github/workflows/ci_pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,106 +104,7 @@ jobs:
Fuzz:
if: ${{ github.event.review.state == 'approved' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/heads/release') }}
needs: [Build]
runs-on: [self-hosted, ubuntu, asrunner]
container:
image: ubuntu:18.04
strategy:
matrix:
include:
- name: db_fuzzer
- name: db_map_fuzzer

steps:
- name: Pre-build
run: |
apt update && apt install -y sudo python3 git clang-tools cmake make automake ucommon-utils libtool gettext pkg-config build-essential clang-10 zlib1g-dev libbz2-dev ninja-build liblzma-dev autoconf libsnappy-dev libzstd-dev liblz4-dev binutils m4 g++-10 unzip
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
echo "nameserver 8.8.8.8" > /etc/resolv.conf

- uses: actions/checkout@v3

- name: Configure AWS credentials from Test account
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: eu-west-2

- name: ${{ matrix.name }}
run: |
echo 'git clone https://github.com/google/libprotobuf-mutator.git \n
cd libprotobuf-mutator \n
git checkout ffd86a32874e5c08a143019aad1aaf0907294c9f \n
cd .. \n
export CC=clang && export CXX=clang++ && mkdir LPM && cd LPM \n
ln -s /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /usr/lib/x86_64-linux-gnu/libstdc++.so \n
ln -s /usr/bin/clang-10 /usr/bin/clang \n
ln -s /usr/bin/clang++-10 /usr/bin/clang++ \n
cmake ../libprotobuf-mutator -GNinja -DLIB_PROTO_MUTATOR_DOWNLOAD_PROTOBUF=ON -DLIB_PROTO_MUTATOR_TESTING=OFF -DCMAKE_BUILD_TYPE=Release \n
ninja \n
ninja install \n
export PKG_CONFIG_PATH=$PWD:$PWD/external.protobuf/lib/pkgconfig/ \n
export PATH=$PWD/external.protobuf/bin:$PATH \n
cd $GITHUB_WORKSPACE \n
COMPILE_WITH_ASAN=1 PORTABLE=1 make -j2 static_lib \n
cd $GITHUB_WORKSPACE/fuzz \n
make ${{ matrix.name }} \n
ls -alFh $GITHUB_WORKSPACE/fuzz/ \n
echo ASAN_OPTIONS=detect_leaks=0 ./db_fuzzer \n' > prepfuz.sh
chmod +x prepfuz.sh
bash -xv prepfuz.sh
mkdir -p $GITHUB_WORKSPACE/out/
ASAN_OPTIONS=detect_odr_violation=0 $GITHUB_WORKSPACE/fuzz/${{ matrix.name }} 2>&1 | sudo tee $GITHUB_WORKSPACE/out/${{ matrix.name }}.log
tail -20 $GITHUB_WORKSPACE/out/${{ matrix.name }}.log | grep "==AddressSanitizer. Thread limit (4194304 threads) exceeded\. Dying\." || echo "${{ matrix.name }} failed!"

- name: Copy ${{ matrix.name }} logs to S3
run: |
aws s3 cp $GITHUB_WORKSPACE/out/${{ matrix.name }}.log s3://spdb-github-ci/

fuzzer_sum:
runs-on: [self-hosted, ubuntu, asrunner]
container:
image: ubuntu:18.04
needs: [ Fuzz ]

steps:
- uses: actions/checkout@v3

- name: pre
run: |
apt update && apt install -y sudo unzip curl
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

- name: Configure AWS credentials from Test account
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: eu-west-2

- name: download db_fuzzer.log from s3
run: |
mkdir -p $GITHUB_WORKSPACE/out
aws s3 cp s3://spdb-github-ci/db_fuzzer.log $GITHUB_WORKSPACE/out/db_fuzzer.log || echo "db_fuzzer.log not found"
ls -alFh $GITHUB_WORKSPACE/out

- name: download db_map_fuzzer.log from s3
run: |
mkdir -p $GITHUB_WORKSPACE/out
aws s3 cp s3://spdb-github-ci/db_map_fuzzer.log $GITHUB_WORKSPACE/out/db_map_fuzzer.log || echo "db_map_fuzzer.log not found"
ls -alFh $GITHUB_WORKSPACE/out

- name: fuzzersum
run: |
echo "~~~~ db_map_fuzzer LOGS ~~~~~"
tail -20 $GITHUB_WORKSPACE/out/db_map_fuzzer.log | grep "==AddressSanitizer. Thread limit (4194304 threads) exceeded\. Dying\." || echo "db_map_fuzzer failed!"
echo "~~~~ db_fuzzer LOGS ~~~~~"
tail -20 $GITHUB_WORKSPACE/out/db_fuzzer.log | grep "==AddressSanitizer. Thread limit (4194304 threads) exceeded\. Dying\." || echo "db_fuzzer failed!"

- name: clear s3 bucket
run: |
aws s3 rm s3://spdb-github-ci/ --recursive --include "*"
uses: ./.github/workflows/test_fuzz.yml
secrets:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
71 changes: 15 additions & 56 deletions .github/workflows/test_fuzz.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,16 @@
name: new_fuzz
name: Fuzz Test

on:
workflow_dispatch:
workflow_call:
secrets:
AWS_ACCESS_KEY_ID:
required: true
AWS_SECRET_ACCESS_KEY:
required: true

jobs:
fuzz:
Fuzz:
runs-on: [self-hosted, ubuntu, asrunner]
container:
image: ubuntu:18.04
Expand All @@ -16,15 +21,16 @@ jobs:
- name: db_map_fuzzer

steps:
- uses: actions/checkout@v3

- name: Pre-build
run: |
echo "nameserver 8.8.8.8" > /etc/resolv.conf
apt update && apt install -y sudo python3 git clang-tools cmake make automake ucommon-utils libtool gettext pkg-config build-essential clang-10 zlib1g-dev libbz2-dev ninja-build liblzma-dev autoconf libsnappy-dev libzstd-dev liblz4-dev binutils m4 g++-10 unzip
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

- uses: actions/checkout@v3

- name: Configure AWS credentials from Test account
uses: aws-actions/configure-aws-credentials@v1
with:
Expand Down Expand Up @@ -56,57 +62,10 @@ jobs:
chmod +x prepfuz.sh
bash -xv prepfuz.sh
Comment on lines 62 to 63
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not simply run the commands instead of outputting them to a file and executing it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the way it was ran in gitlab ci, I think @maxb-io explained it to me but I can't really remember

mkdir -p $GITHUB_WORKSPACE/out/
ASAN_OPTIONS=detect_odr_violation=0 $GITHUB_WORKSPACE/fuzz/${{ matrix.name }} 2>&1 | sudo tee $GITHUB_WORKSPACE/out/${{ matrix.name }}.log-$GITHUB_SHA
tail -20 $GITHUB_WORKSPACE/out/${{ matrix.name }}.log-$GITHUB_SHA | grep "==AddressSanitizer. Thread limit (4194304 threads) exceeded\. Dying\." || echo "${{ matrix.name }} failed!"
ASAN_OPTIONS=detect_odr_violation=0 $GITHUB_WORKSPACE/fuzz/${{ matrix.name }} 2>&1 | tee $GITHUB_WORKSPACE/out/${{ matrix.name }}.log
tail -20 $GITHUB_WORKSPACE/out/${{ matrix.name }}.log | grep "==AddressSanitizer. Thread limit (4194304 threads) exceeded\. Dying\." || { echo "${{ matrix.name }} failed!" && false; }

- name: Copy ${{ matrix.name }} logs to S3
run: |
aws s3 cp $GITHUB_WORKSPACE/out/${{ matrix.name }}.log-$GITHUB_SHA s3://spdb-github-ci/

fuzzer_sum:
runs-on: ubuntu-latest
container:
image: ubuntu:18.04
needs: [ fuzz ]

steps:
- uses: actions/checkout@v3

- name: pre
run: |
apt update && apt install -y sudo unzip curl
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

- name: Configure AWS credentials from Test account
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: eu-west-2

- name: download db_fuzzer.log from s3
run: |
mkdir -p $GITHUB_WORKSPACE/out
aws s3 cp s3://spdb-github-ci/db_fuzzer.log $GITHUB_WORKSPACE/out/db_fuzzer.log || echo "db_fuzzer.log not found"
ls -alFh $GITHUB_WORKSPACE/out

- name: download db_map_fuzzer.log from s3
run: |
mkdir -p $GITHUB_WORKSPACE/out
aws s3 cp s3://spdb-github-ci/db_map_fuzzer.log $GITHUB_WORKSPACE/out/db_map_fuzzer.log || echo "db_map_fuzzer.log not found"
ls -alFh $GITHUB_WORKSPACE/out

- name: fuzzersum
- name: Copy ${{ matrix.name }} logs to S3 due to failure
if: steps.${{ matrix.name }}.outcome == 'failure'
run: |
echo "~~~~ db_map_fuzzer LOGS ~~~~~"
tail -20 $GITHUB_WORKSPACE/out/db_map_fuzzer.log | grep "==AddressSanitizer. Thread limit (4194304 threads) exceeded\. Dying\." || echo "db_map_fuzzer failed!"
echo "~~~~ db_fuzzer LOGS ~~~~~"
tail -20 $GITHUB_WORKSPACE/out/db_fuzzer.log | grep "==AddressSanitizer. Thread limit (4194304 threads) exceeded\. Dying\." || echo "db_fuzzer failed!"

- name: clear s3 bucket
run: |
aws s3 rm s3://spdb-github-ci/ --recursive --include "*"


aws s3 cp $GITHUB_WORKSPACE/out/${{ matrix.name }}.log s3://spdb-github-ci/$GITHUB_SHA-${{ matrix.name }}.log