Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance Regression Testing Revamp Stage 1 #4602

Merged
merged 166 commits into from
Apr 12, 2022
Merged

Performance Regression Testing Revamp Stage 1 #4602

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
166 commits
Select commit Hold shift + click to select a range
26b0efb
update readme with draft of v2
Jan 20, 2022
848f381
cache runner binary instead of storing it as an artifact
Jan 20, 2022
f765c41
rename job
Jan 20, 2022
a59baf9
add debug lines
Jan 20, 2022
45d9233
checkout source before accessing cache
Jan 20, 2022
ec83913
add temporary pull request trigger for dev
Jan 20, 2022
56dc946
fix path accessing runner
Jan 20, 2022
672e9c9
add more debug steps
Jan 20, 2022
bd281ba
add even more debug steps
Jan 20, 2022
d774fd9
move debug lines to the right place
Jan 20, 2022
d3dd754
change cache path
Jan 20, 2022
38936cf
update cache action
Jan 21, 2022
03e3419
update baseline to 1.0.latest
Jan 21, 2022
92c267d
remove debug steps
Jan 21, 2022
ea0b92b
update formatting
Jan 21, 2022
f75bc01
fix cache hashing
Jan 21, 2022
f51138a
update readme with threshold information
Jan 21, 2022
e5a7fad
runner uses 3sigma to detect regressions
Jan 21, 2022
f066d9d
fmt
Jan 21, 2022
dc565e2
move runner first
Jan 21, 2022
63d34de
minor wording fix
Feb 9, 2022
fc6e512
model output file for baseline measurements
Feb 9, 2022
4c2b4a1
Merge branch 'main' into nate/regression-testing-2
Feb 15, 2022
d86bc29
fleshed out statistics section
Feb 15, 2022
0d280dc
readme fixes
Feb 15, 2022
669903b
newline with breaks instead of spaces
Feb 15, 2022
1b21253
remove lines from svg
Feb 15, 2022
6330650
redefine ranges
Feb 15, 2022
a79c07d
test frame for compare_from
Feb 15, 2022
8a08ba9
more readme work
Feb 16, 2022
ed07f51
version tree with boxes
Feb 16, 2022
383fd1d
version tree with refs
Feb 16, 2022
8ac060a
implemented get for version tree.
Feb 16, 2022
52f3155
finished compare_from
Feb 16, 2022
68ae9e6
fmt
Feb 16, 2022
0a6cf96
update readme
Feb 18, 2022
9bfa6c3
add paragraph on reruns
Feb 21, 2022
5558e68
scaffold some of the big changes. doesn't compile
Feb 21, 2022
717bdab
followed compiler errors to fix a bunch of stuff
Feb 21, 2022
09a282e
added lots of cloning but now it compiles
Feb 21, 2022
12fdaa5
fixed some warnings
Feb 21, 2022
f91f8da
write take_samples and fix some compile errors
Feb 21, 2022
8a347b5
minor refactor
Feb 22, 2022
76a482d
fix tests, found bug
Feb 22, 2022
c4bc58b
struct opt takes Version struct as input now
Feb 22, 2022
5549f5e
renamed to more sensible names
Feb 22, 2022
0a065fd
moved type definitions to separate module
Feb 22, 2022
d2355df
fixed tests
Feb 22, 2022
e71776a
temp name metricc to perm name metric
Feb 22, 2022
309e16b
remove file writing from main
Feb 22, 2022
398cf72
make sample run work
Feb 22, 2022
cd229c0
renamed workflow
Feb 22, 2022
5481a2e
change version input
Feb 22, 2022
0cb7534
temporary new workflow for modeling before it gets put into the relea…
Feb 22, 2022
1fd9168
change name
Feb 23, 2022
a0deb8e
remove unused bits
Feb 23, 2022
1455bc4
develop the modeling action here
Feb 23, 2022
89585b1
stub version input
Feb 23, 2022
04e1503
fmt
Feb 23, 2022
e631594
expose number of runs to model command
Feb 23, 2022
6785761
add debug ls
Feb 23, 2022
c3c41c7
fmt
Feb 23, 2022
28dc0f6
merge two two model jobs into one
Feb 23, 2022
9cefe67
resolve relative paths from cli
Feb 23, 2022
b2e7776
fmt
Feb 23, 2022
d1fc2ee
rename to app
Feb 23, 2022
1838300
merge into one job
Feb 23, 2022
ff0abf5
rename job
Feb 23, 2022
b5a4d9d
remove cache retrieval step
Feb 23, 2022
892aba6
remove cache retrieval step here too
Feb 23, 2022
923bed3
require relative paths to have some prefix
Feb 23, 2022
0cfb691
fix error message
Feb 23, 2022
cefb76c
add debug line
Feb 23, 2022
5dbe170
try trailing slashes
Feb 23, 2022
9c1cb0f
try absolute paths
Feb 23, 2022
dc525c6
stop canonicalizing
Feb 23, 2022
ae18587
fmt
Feb 23, 2022
e9e9e1d
fixed warnings
Feb 23, 2022
21fedee
push baselines dir
Feb 23, 2022
d731edf
print modeling results to stdout
Feb 23, 2022
a772161
catch if a baseline is being built with no models
Feb 23, 2022
1d28405
fix bug
Feb 23, 2022
89a4822
make tmp baseline dir and do run of 20
Feb 23, 2022
80735e7
add version directory
Feb 24, 2022
c695838
reduce modeling work temporarily
Feb 24, 2022
be1d825
update readme with gha data
Feb 24, 2022
b61c35b
fmt
Feb 24, 2022
6328821
seprate jobs makes more sense
Feb 24, 2022
7bb8721
rename
Feb 24, 2022
b563f50
rename calculate error to runner error
Feb 24, 2022
2b9e6af
fixed up comment
Mar 2, 2022
dcea10f
fix caching
Mar 2, 2022
6291070
fixed tests for generated project
Mar 3, 2022
b929ec7
update project config
Mar 3, 2022
38bc1b8
add first manually executed baseline model
Mar 4, 2022
e58e155
run two samples just for now
Mar 4, 2022
0157827
does not compile. saving work.
Mar 4, 2022
8a9deee
fmt
Mar 7, 2022
dcf666f
split out runner cache/build job to pull from main
Mar 7, 2022
902b683
minor cleanup
Mar 7, 2022
05e1bd9
update test to new types
Mar 7, 2022
090fd3c
compiles now
Mar 7, 2022
d812ccb
moved sample-baseline matching up a level
Mar 7, 2022
3eccee7
add BadFilestemError to remove some unwraps
Mar 7, 2022
041a214
better comment wording
Mar 7, 2022
55bece9
forgot to get runner from cache
Mar 7, 2022
40998a5
refactored working with filestems
Mar 7, 2022
fa5a886
tmp pull runner from this branch instead
Mar 7, 2022
08e84cc
what why when comments
Mar 8, 2022
b43eda1
fmt
Mar 8, 2022
b78842d
fix warnings
Mar 8, 2022
47c7351
add debug print=
Mar 8, 2022
7a8a6d0
use absolute paths in workflow
Mar 8, 2022
47b33e2
deal with double measurements
Mar 8, 2022
4ab7a76
rename measure
Mar 8, 2022
e472b54
minor refactor
Mar 8, 2022
38373fb
add latest version function
Mar 8, 2022
83bc505
nicer refactor of latest_version_from
Mar 8, 2022
c8112e1
hook in latest version logic
Mar 8, 2022
0f64ce7
add test for version order
Mar 8, 2022
01667ec
use filename of dir not full path
Mar 8, 2022
81aee63
add nosamplescomputed error
Mar 8, 2022
dad0691
add debug lines to runner
Mar 8, 2022
c84e882
Revert "add debug lines to runner"
Mar 8, 2022
731ef1b
add todo
Mar 8, 2022
1161883
add debug lines
Mar 8, 2022
d35e591
add new exception when samples are missing from the baseline
Mar 8, 2022
fb6f9a5
debug lines
Mar 8, 2022
2ad8cfb
check if baselines are empty
Mar 8, 2022
ec54dc5
fix missing baseline bug
Mar 8, 2022
5b555ac
add sample value to final calculation
Mar 9, 2022
54d7bc2
split out file reading
Mar 9, 2022
5ae4f30
compose pure and impure operations
Mar 9, 2022
fdd23c1
remove formatting tests because they're not that helpful anyway
Mar 9, 2022
b9627e2
remove unused improt
Mar 9, 2022
dc5e946
use asref path interface instead of concrete pathbufs for params
Mar 9, 2022
d4b59dc
remove todo
Mar 9, 2022
ba01919
remove another todo
Mar 9, 2022
bb791c6
fmt
Mar 9, 2022
a414e4d
Merge branch 'main' into nate/regression-testing-2
Mar 9, 2022
2d18c15
changelog entry
Mar 9, 2022
25323f3
complete todo referencing modeling action
Mar 9, 2022
fd31552
first pass at modeling the correct version
Mar 9, 2022
9e6f54f
add version input for manual trigger
Mar 9, 2022
3c1f7b1
get version string from both trigger events properly
Mar 10, 2022
eb8e1c0
add stale info warning to README
Mar 10, 2022
40a53d4
update model comment
Mar 10, 2022
5c656e3
save cache key from runner build
Mar 10, 2022
37a0ca8
update readme
Mar 10, 2022
49388c3
add future work bullet to readme
Mar 10, 2022
f1282b8
commit logic
Mar 10, 2022
2ef524f
remove bad working directory
Mar 10, 2022
c22bc4d
fixed all bugs I could find by manually testing in a private fork
Mar 11, 2022
2475a1f
lots of work in personal fork again.
Mar 15, 2022
d65bf76
update changelog entry
Mar 16, 2022
1568528
remove actions. will be developed in another pr
Mar 16, 2022
8918836
remove baseline and add placeholder for directory
Mar 16, 2022
9b41637
revert project changes. fixed in another pr
Mar 16, 2022
142ec21
clear_dir creates any missing parent dirs as well.
Apr 12, 2022
43e9e32
hyperfine cleans up with dbt clean
Apr 12, 2022
9d0bf9a
add clearer call out for the risk of gha variation to readme
Apr 12, 2022
f0efe54
add concrete example of performance modeling work to readme
Apr 12, 2022
3984024
add paragraph about branching to investigating regressions section of…
Apr 12, 2022
60cde92
add comment about parsing versions
Apr 12, 2022
e2ff7c4
add comment to empty baseline check for clarity
Apr 12, 2022
6fef3ef
improve wording of confusing sentence
Apr 12, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changes/unreleased/Under the Hood-20220309-142133.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
kind: Under the Hood
body: add performance regression testing runner without orchestration
time: 2022-03-09T14:21:33.884043-05:00
custom:
Author: nathaniel-may
Issue: "4021"
PR: "4602"
176 changes: 0 additions & 176 deletions .github/workflows/performance.yml
View file
Open in desktop

This file was deleted.

Loading