Coverage API behaves oddly with coverage-only runs

[connor4312]

          I think I ran into a bug, trying to use the coverage API to integrate coverage data loaded from a file. I am trying to follow the instructions which were previously [provided up-thread](https://github.com/microsoft/vscode/issues/123713#issuecomment-840908164)

Scenario

I am creating two test runs. Those test runs have no test cases attached, they only contain a name, some output (appendOutput) and the coverage data.

Minimal repro

import * as vscode from "vscode";

class BazelCoverageProvider implements vscode.TestCoverageProvider {
  provideFileCoverage(
    token: vscode.CancellationToken,
  ): vscode.ProviderResult<vscode.FileCoverage[]> {
    const detailedCoverage = [
      new vscode.DeclarationCoverage(
        "test_func",
        12,
        new vscode.Position(1, 12),
      ),
      new vscode.DeclarationCoverage(
        "test_func2",
        0,
        new vscode.Position(10, 0),
      ),
      new vscode.StatementCoverage(true, new vscode.Position(15, 3)),
      new vscode.StatementCoverage(false, new vscode.Position(16, 0)),
    ];
    return [
      vscode.FileCoverage.fromDetails(
        vscode.Uri.file("/Users/avogelsgesang/hyper/hyper-db/hyper/ir/IR.cpp"),
        detailedCoverage,
      ),
    ];
  }

  // Already fully resolved. Noting to resolve
  resolveFileCoverage?(
    coverage: vscode.FileCoverage,
    token: vscode.CancellationToken,
  ): vscode.ProviderResult<vscode.FileCoverage> {
    return coverage;
  }
}

export async function activate(context: vscode.ExtensionContext) {
  // Create the test controller
  const testController = vscode.tests.createTestController(
    "bazel-coverage",
    "Bazel Coverage",
  );
  context.subscriptions.push(testController);

  // Provide coverage info 1
  const run = testController.createTestRun(
    new vscode.TestRunRequest(),
    "/my/coverage.lcov",
    false,
  );
  run.appendOutput("Information about where we loaded the coverage from...");
  run.coverageProvider = new BazelCoverageProvider();
  run.end();

  // Provide coverage info 2
  const run2 = testController.createTestRun(
    new vscode.TestRunRequest(),
    "/my/second_coverage.lcov",
    false,
  );
  run2.appendOutput("Some other information...");
  run2.coverageProvider = new BazelCoverageProvider();
  run2.end();
}

Observed behavior

I see two coverage entries in the test result list, and I can switch between them. Also, I see the output from the 2nd test run.

However, I do not see the test run names "/my/coverage.lcov" and "/my/second_coverage.lcov" anywhere. Also, there seems to be no way to see inspect the appendOutput output from the first run.

Expected behavior

I guess the "Close test coverage"/"View test coverage" should be grouped under a test run? Even if there are no tests and only coverage data inside this test run? Also, it would be good to be able to set the overall test run into a "success"/"failed" state, even without listing explicit sub-tests

Originally posted by @vogelsgesang in #123713 (comment)

[DonJayamanne]

@connor4312 unable to verify this, as I cannot find any type defs for TestCoverageProvider
perhaps the API changed

[connor4312]

Just ignore any compilation errors, this will still run since we have back-compat shims still in place for the API as it was when the comment was written.

[meganrogge]

@connor4312 using your minimal repro, I do not see the test names anywhere

[meganrogge]

Actually, I see this.... is that what is expected?

[connor4312]

You actually verified it correctly. With the shims that's what you expect to see in the tree view. The issue was the coverage node was compressing into the test run itself (as in the original screenshot) and you wouldn't see the "test run at..." header at all.