Fix Http2 MultiConnection test race conditions

[MihaZupan]

Closes #91075

These tests have logic along the lines of

var connection1 = await SetupConnectionAsync(StreamLimit);
await StartStreamLimitRequestsAsync();

var connection2 = await SetupConnectionAsync(StreamLimit);
await StartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

---

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Closes #91075

These tests have logic along the lines of

var connection1 = await SetupConnectionAsync(StreamLimit);
await StartStreamLimitRequestsAsync();

var connection2 = await SetupConnectionAsync(StreamLimit);
await StartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

---

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

Author:	MihaZupan
Assignees:	MihaZupan
Labels:	area-System.Net.Http
Milestone:	9.0.0

[MihaZupan]

/azp run runtime-libraries-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[wfurt]

LGTM

[MihaZupan]

/backport to release/8.0

[github-actions]

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6027293964