Failing integration test: SendReceiveAfterProjectReset

[myieye]

For the test SendReceiveAfterProjectReset:

All 6 executions (2 protocols x 3 jobs) failed after/during the last deployment to staging.

For the deployment before that the results were 5 failures / 8 test executions.

I think we should try to see why it's failing and stabilize it.

It might be worth noting, that when querying changesets we explicitly return an empty list if the repo is less than 40 seconds old. We found that we had to wait longer than 20 seconds even though the hgweb refreshinterval was set to 20 seconds at the time. See Project.GetChangesets().

[rmunn]

Currently working on #712 (PR for #709) so I have a chance at figuring this out; without Playwright traces, it's a little hard. (BTW, #712 is ready for review; at this point any tweaks I make to it are going to be minor).

[myieye]

@rmunn Are you sure Playwright traces are relevant for this issue? SendReceiveAfterProjectReset is a .NET Send/Receive integration test.

[rmunn]

#709 was marked high priority, so I looked into it first and didn't actually read this one carefully. Yes, the Playwright traces are irrelevant to .NET tests. I simply wrote that comment in haste, without realizing that this issue was about a .NET integration test.

Anyway, #716 doesn't actually modify the SendReceiveAfterProjectReset yet, but it does make it possible to run it correctly with Mercurial 6. Which may or may not actually fix this issue; I'll find out once I can finally get a S/R run. (The self-hosted runner has been flaky; sometimes it stops responding during an apt install step, or the setup-pnpm step... it's been a long process of pushing the retry button and hoping that this time I get a run that makes it as far as the test I need to see. Oh, and the develop hgweb server returning 503 errors didn't help much either...)

[rmunn]

More 5xx errors, this time 502 Bad Gateway. Still haven't gotten a clean run of the SendReceiveAfterProjectReset test, and I've been trying all morning. I'll keep trying, but at some point I'm going to give up on this and try to go do something actually productive.

[rmunn]

And the latest test run had "The active test run was aborted. Reason: Test host process crashed." Funny, though, it reported 0 failures and 48 successes when that happened. Weird.

[rmunn]

STILL getting "Test host process crashed". We might be running into https://www.github.com/microsoft/vstest/issues/2952 — if so, there's no way I'm going to be able to fix this. It's frustrating, though, because I can't quite trust that the 48 test successes are real.

[rmunn]

I thought commit 06c027a would fix the "Test host process crashed" errors. I just noticed the compilation warning about not using async void methods, which I had used in the "project codes may not start with a hyphen" test I wrote a couple weeks ago. Changed that test's return type to async Task and ran the integration test workflow again, but it's still crashing. Still 48 tests succeeding, though, so I'm hopeful that I've at least made SendReceiveAfterProjectReset start passing. But I can't prove it.

[hahn-kev]

@rmunn are you running this test from your machine or the self hosted runner? I'd run it from your machine as we want to stabilize the test itself, anything specific to the self hosted runner we also want to fix, but it would be better to isolate the issues to solve them.

[rmunn]

Running it on my machine passes every time, so I don't know what else to do to stabilize it. So I'm trying to run it on the self-hosted runner, because that's where it's unstable. But I've had so many issues with the runner that I'm starting to feel like the past couple days have been a complete waste of time.

[myieye]

It's been refactored a bit and hg runner timeouts have been increased. Recent runs have been smooth.

[rmunn]

Probably fixed by a combination of #715 and #716, I'm guessing.