Performance Regression Testing Revamp Stage 1

[nathaniel-may]

resolves #4021

Description

This is the first part of a multi-PR feature. Merging this will have no effect on the code base, but should be reviewed in full because this code will be activated by the actions that are under development in #4872. The goal of the overarching feature is to run performance regression tests on PRs with minimal failures due to noise. This work shifts the entire strategy for detecting performance regressions away from fully modeling performance characteristics on each test, to sampling and comparing to a baseline model. The strategy and the math to support it is outlined in detail in the README.

To start, this PR only includes a single test project with 2000 model files for the performance of dbt parse, but once the actions are merged it will be able to run efficiently on each PR. The next step is to add more commands and projects. This stage is outlined in #4768.

Reviewers

Please read the README in detail. It is intended to provide all the information necessary to develop this without me. If there's something that's unclear, please bring it up.

To navigate the runner, start in main::run_app which handles all the IO, then hit the two top-level functions: fs::model and calculate::regressions. The split between fs and calculate modules isn't perfect and I could probably have some better naming of these functions. Suggestions are welcome.

The runner is not perfect, and I have left TODOs for all the places I think we should improve. It will be worked on in subsequent stages of this feature so there will be time to fix more of these. If anything looks below the code quality we want to merge, I am happy to make those changes in this PR.

Please also review the 1 change PR #4875 if it is not already merged. It updates the test project by regenerating it with a ref. I split it out because it touches 2000 files.

Tasks

• regenerate test project to be less wrong (use ref dbt-timing-project#14)
• manually model 1.0.latest and (NO MERGE manual perf modeling only. #4764) commit baseline json file to main (this branch)
• wrap up runner sampling code path
• merge baseline file to 1.0.latest (manually add performance baseline for 1.0.3 #4847)
• modify model action to accurately obtain version information from both manual triggers and release triggers. (tested in a personal sandbox repo here)
• polish README (remove adding projects and commands and replace with phase 2 ticket link)
• backport new README to 1.0.latest branch as well so it has the stale info warning. (to be done after pr is approved to include any readme changes) backport perf readme  #5042

Checklist

• I have signed the CLA
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have updated the CHANGELOG.md and added information about my change

[leahwicz]

Overall this LGTM!

Two thing that I would feel like would be good to add to the README is:

• What happens/example of when a release is completed and now we want to set a baseline for that release- what runs, on what branch, what gets committed, etc.
• What happens when we test a commit- you mention this in the Concrete Example but it doesn't mention scenario details such as: I commit to main vs I'm backporting to 1.0.latest, if I'm on 1.0.latest then I'm being compared to the latest final release or RC? Basically the decision tree of what is the baseline that I'm getting compared to