Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A/B test reports #309

Merged
merged 11 commits into from
Sep 9, 2022
Merged

A/B test reports #309

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
80a68f1
WIP barchart for comparing different commits rather than a timeseries.
Sep 1, 2022
b03c3f1
Add PyCharm files to .gitignore
crusaderky Sep 2, 2022
87fd490
Merge branch 'main' into barchart
crusaderky Sep 2, 2022
80c6461
Merge branch 'main' into barchart
crusaderky Sep 2, 2022
ca9c525
A/B bar charts
crusaderky Sep 7, 2022
43460fa
Gracefully fail imports
crusaderky Sep 7, 2022
bae7564
Cosmetic
crusaderky Sep 7, 2022
35148aa
Build test report in PRs
crusaderky Sep 7, 2022
905b716
bugfix
crusaderky Sep 7, 2022
13a21d9
test-upstream
crusaderky Sep 7, 2022
907fe17
upstream is mutually exclusive with latest
crusaderky Sep 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 19 additions & 3 deletions .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,16 +57,19 @@ jobs:
if [[ "$COMMIT" == *"test-upstream"* || ${{ github.event_name }} == "schedule" ]]
then
export TEST_UPSTREAM="true"
export AB_BASELINE="coiled-upstream-py3.9 coiled-latest-py3.9 coiled-0.1.0-py3.9"
else
export TEST_UPSTREAM="false"
export AB_BASELINE="coiled-latest-py3.9 coiled-0.1.0-py3.9"
fi

# Put TEST_UPSTREAM into $GITHUB_ENV so it can be used in subsequent workflow steps
echo $TEST_UPSTREAM
echo TEST_UPSTREAM=$TEST_UPSTREAM >> $GITHUB_ENV

# Put TEST_UPSTREAM into a file so it can be downloaded in subsequent workflow jobs
# Put env variables into files so it can be downloaded in subsequent workflow jobs
echo $TEST_UPSTREAM > test_upstream.txt
echo $AB_BASELINE > ab_baseline.txt

- name: Build Coiled Software Environment
env:
Expand Down Expand Up @@ -104,6 +107,7 @@ jobs:
latest.yaml
software_name.txt
test_upstream.txt
ab_baseline.txt

runtime:
name: Runtime - ${{ matrix.os }}, Python ${{ matrix.python-version }}, Runtime ${{ matrix.runtime-version }}
Expand Down Expand Up @@ -437,7 +441,7 @@ jobs:
static-site:
needs: process-results
# Always generate the site, as this can be skipped even if an indirect dependency fails (like a test run)
if: always() && github.ref == 'refs/heads/main' && github.repository == 'coiled/coiled-runtime'
if: always()
name: Build static dashboards
runs-on: ubuntu-latest
steps:
Expand All @@ -457,12 +461,24 @@ jobs:
python-version: "3.9"
environment-file: ci/environment-dashboard.yml

- name: Download software environment assets
uses: actions/download-artifact@v3
with:
name: software-environment-py3.9
Comment on lines +464 to +467
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where are you using this? Hopefully these assets will be going away soon

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the following change (line 471)


- name: Generate dashboards
run: |
python dashboard.py
python dashboard.py -d benchmark.db -o static -b $(cat ab_baseline.txt)

- name: Upload artifact
uses: actions/upload-artifact@v3
with:
name: static-dashboard
path: static

- name: Deploy 🚀
uses: JamesIves/github-pages-deploy-action@4.1.7
if: github.ref == 'refs/heads/main' && github.repository == 'coiled/coiled-runtime'
with:
branch: gh-pages
folder: static
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,9 @@ venv.bak/
# Rope project settings
.ropeproject

# PyCharm project settings
.idea

# mkdocs documentation
/site

Expand Down
1 change: 1 addition & 0 deletions ci/environment-dashboard.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ dependencies:
- dask
- dask-ml
- distributed
- xarray
- xgboost
- pandas
- tabulate
Loading