Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce the number of configurations used for the sims tests. #1138

Merged
merged 14 commits into from
Oct 18, 2022
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 7 additions & 99 deletions .github/workflows/sims.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,17 +56,12 @@ jobs:
# * Go binary directory
# * Go module directory
# * Go build cache (Linux)
# * Go build cache (Mac)
go-cache-path: |
~/go/bin
~/go/pkg/mod
~/.cache/go-build
~/Library/Caches/go-build

build-linux:
# Note: Can't use a matrix to combine build-linux and build-mac because they'd both have the same job id and that would cause
# problems below with the "needs" directive on the tests. Basically, as soon as the first one finished, the tests
# would kick off for both OSes even though one might not be ready yet.
needs: setup
if: needs.setup.outputs.should-run
runs-on: ubuntu-latest
Expand Down Expand Up @@ -128,98 +123,19 @@ jobs:
- name: Provenanced version
run: build/provenanced version --long

build-mac:
# Note: Can't use a matrix to combine build-linux and build-mac because they'd both have the same job id and that would cause
# problems below with the "needs" directive on the tests. Basically, as soon as the first one finished, the tests
# would kick off for both OSes even though one might not be ready yet.
needs: setup
if: needs.setup.outputs.should-run
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
- name: Output setup
run: |
echo " go-version: [${{ needs.setup.outputs.go-version }}]"
echo " should-run: [${{ needs.setup.outputs.should-run }}]"
echo " file-prefix: [${{ needs.setup.outputs.file-prefix }}]"
echo "db-cache-key-suffix: [${{ needs.setup.outputs.db-cache-key-suffix }}]"
echo " db-cache-path: [${{ needs.setup.outputs.db-cache-path }}]"
echo "go-cache-key-suffix: [${{ needs.setup.outputs.go-cache-key-suffix }}]"
echo " go-cache-path: [${{ needs.setup.outputs.go-cache-path }}]"
- uses: actions/cache@v3
name: Load db cache
id: db-cache-setup
with:
key: ${{ runner.os }}-${{ needs.setup.outputs.db-cache-key-suffix }}
path: ${{ needs.setup.outputs.db-cache-path }}
- uses: actions/cache@v3
name: Load go cache
id: go-cache-setup
with:
key: ${{ runner.os }}-${{ needs.setup.outputs.go-cache-key-suffix }}
path: ${{ needs.setup.outputs.go-cache-path }}
- name: Setup build environment
run: |
brew install cmake sqlite3
- name: Build cleveldb
if: steps.db-cache-setup.outputs.cache-hit != 'true'
# As of 2022-06-13 (it was a Monday), the macos-latest runners do not have the nproc command.
# But have 3 cores according to https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources
# The linux runners have nproc, so we don't need to set the _JOBS in that job.
run: |
export CLEVELDB_JOBS=3
export CLEVELDB_DO_BUILD='true'
export CLEVELDB_DO_INSTALL='false'
export CLEVELDB_DO_CLEANUP='false'
make cleveldb
- name: Install cleveldb
run: |
export CLEVELDB_DO_BUILD='false'
export CLEVELDB_DO_INSTALL='true'
export CLEVELDB_SUDO='true'
export CLEVELDB_DO_CLEANUP='false'
make cleveldb
- uses: actions/setup-go@v3
with:
go-version: ${{ needs.setup.outputs.go-version }}
- name: Display go version
run: go version
- name: Install runsim
if: steps.go-cache-setup.outputs.cache-hit != 'true'
run: export GO111MODULE="on" && go get github.com/cosmos/tools/cmd/runsim@v1.0.0
- name: Update provwasm contract
run: make download-smart-contracts
- name: Build provenanced
run: make build
- name: Provenanced version
run: build/provenanced version --long

runsim:
# These tests are the ones that use the runsim program (see sims.mk).
needs: [setup, build-linux]
if: needs.setup.outputs.should-run
strategy:
fail-fast: false
matrix:
# The test-sim-import-export test can take up to two hours.
# The test-sim-after-import test can take even longer.
# The test-sim-multi-seed-short test should take 30-40 minutes.
# TODO: Add the "after-import" test once it's fixed: https://github.com/provenance-io/provenance/issues/754
test: ["import-export", "multi-seed-short"]
db-backend: ["goleveldb", "cleveldb", "badgerdb"]
# Not putting "macos-latest" in this because:
# a) They take longer than on ubuntu to begin with.
# b) We're limited to 5 Mac runners at once so they get queued up, taking even longer to finish all of them.
# c) Most of the devs use a mac, so it's less likely that a Mac-specific bug goes unnoticed without these tests.
test: ["import-export", "multi-seed-short", "after-import"]
db-backend: ["goleveldb"]
os: ["ubuntu-latest"]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
- uses: actions/cache@v3
name: Load db cache
with:
key: ${{ runner.os }}-${{ needs.setup.outputs.db-cache-key-suffix }}
path: ${{ needs.setup.outputs.db-cache-path }}
- uses: actions/cache@v3
name: Load go cache
with:
Expand All @@ -228,14 +144,6 @@ jobs:
- name: Define test-logs
id: test-logs
run: echo "::set-output name=test-logs::${{ needs.setup.outputs.file-prefix }}-${{ matrix.test }}-${{ matrix.db-backend }}-${{ matrix.os }}"
- name: Install cleveldb
if: matrix.db-backend == 'cleveldb'
run: |
export CLEVELDB_DO_BUILD='false'
export CLEVELDB_DO_INSTALL='true'
export CLEVELDB_SUDO='true'
export CLEVELDB_DO_CLEANUP='false'
make cleveldb
- uses: actions/setup-go@v3
with:
go-version: ${{ needs.setup.outputs.go-version }}
Expand Down Expand Up @@ -271,16 +179,16 @@ jobs:

go-test-multi-db:
# These are tests that use go test to run (see sims.mk), and that we want to test using different database backends.
needs: [setup, build-linux, build-mac]
needs: [setup, build-linux]
if: needs.setup.outputs.should-run
strategy:
fail-fast: false
matrix:
# The test-sim-simple test is pretty quick and should be able to identify glaring problems.
# The test-sim-benchmark is handy to have for each db type.
test: ["simple", "benchmark"]
db-backend: ["goleveldb", "cleveldb", "badgerdb"]
os: ["ubuntu-latest", "macos-latest"]
db-backend: ["goleveldb", "cleveldb"]
os: ["ubuntu-latest"]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
Expand Down Expand Up @@ -326,15 +234,15 @@ jobs:

go-test-single-db:
# These are tests that use go test to run (see sims.mk), and that we don't care about testing using different database backends.
needs: [setup, build-linux, build-mac]
needs: [setup, build-linux]
if: needs.setup.outputs.should-run
strategy:
fail-fast: false
matrix:
# The test-sim-nondeterminism test hard-codes the db backend to use memdb.
# The test-sim-benchmark-invariants test can use different db backends, but to save resources, is down here.
test: ["nondeterminism", "benchmark-invariants"]
os: ["ubuntu-latest", "macos-latest"]
os: ["ubuntu-latest"]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
Expand Down