Improve reliability and assurances provided by end-to-end tests

[ranweiler]

Our end-to-end integration tests create jobs, run, them, and block on the presence of some number of files in blob storage containers. They also assert the absence of error telemetry in certain time windows. They do not validate any file-level properties, and the error telemetry assertions are susceptible to false positive failures due to transient and/or handled errors. Additionally, the current end-to-end tests depend on a complex and hard-to-change and debug custom script. This script does not report test results in an aggregate form.

Let's revisit our end-to-end testing strategy. Ideally, we'd be able to use some kind of standard test runner, increase our resilience to transient errors, continue to log warnings about suspicious telemetry, and make meaningful assertions about file contents.

Related issues:

• Support integration tests with fixed binary artifacts #1143
• Enable assertions about integration test output blobs/containers #697

AB#35939