HTTP/2 frame writer CPU usage

[JamesNK]

The HTTP/2 frame writer uses lock to ensure only one stream can write the connection at a time. For example, Http2FrameWriter.WriteDataAsync.

When a connection has many streams with frequent writes, there is a lot of contention on the lock. Profiling shows high CPU usage from threads fighting over the lock.

A potential improvement would be to change the writer to use a producer/consumer queue using Channel<T>. The streams add write operations to the queue and a single consumer loop is responsible for writing frames to the connection.

Today:

public ValueTask<FlushResult> WriteDataAsync(byte[] data)
{
    lock (_writeLock)
    {
        _writer.Write(data);
        return _writer.FlushAsync()
    }
}

Future:

public ValueTask WriteDataAsync(byte[] data)
{
    var operation = SetupWriteData(data);
    _channelWriter.TryWrite(operation);
    return operation.WaitForCompletedAsync();
}

// Consumer loop that reads operations from channel
private async ValueTask WriterConsumer()
{
    while (true)
    {
        if (!await _channelReader.WaitToReadAsync())
        {
            return;
        }

        if (_channelReader.TryRead(out T item))
        {
            switch (item.OperationType)
            {
                case OperationType.WriteData:
                    _writer.Write(item.Data);
                    await _writer.FlushAsync();
                    item.Complete();
                    break;
                case OperationType.WriteHeaders:
                    // etc..
                    break;
            }
        }
    }
}

[JamesNK]

Duplicate of #30235