Write automated tests to test and ensure expected performance characteristics

[greazer]

No description provided.

[amunger]

Breakdown of the work involved:

• Select an initial test that represents a basic perf scenario.
  • Modify the test to measure specific checkpoints. Some already exist, like extension activation time.
• Update the existing perf-test.yml to run the initial test. (https://github.com/microsoft/vscode-jupyter/blob/main/.github/workflows/perf-test.yml)
• Publish the timing data to Kusto and GH actions summary(ensure the telemetry is reaching Kusto from CI pipeline runs.)
  • Kusto for historical trends and GH actions summary for immediate results validation.
  • Add dimensions to the telemetry so we can tie data back to specific commits.
• Run the workflow several times to measure the variance that we experience without any changes.
  • Adjust the workflow and/or perf test to address unacceptable variance if needed.
  • Create a dedicated runner VM and modify the perf workflow to use it if needed
    • Security concern https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security
• Set the workflow trigger to run on PRs to main.