Add a mode to the receive buffer to handle external buffers #4758

guhetier · 2025-01-22T00:01:02Z

Description

As part of #4747, the receive buffer needs a mode to operate on buffers provided by the application.
This PR defines a new QUIC_RECV_BUF_MODE_EXTERNAL mode.

QUIC_RECV_CHUNKs now have a pointer to the actual buffer instead of an end-of-struct array
- Except for the EXTERNAL mode, the buffer is still part of the same allocation as the chunk
- For EXTERNAL mode, buffers are provided by the application and chunks point to them
The EXTERNAL mode writes only once to the provided buffers and discard them when they are drained. Buffers are never re-used.
Reads and writes can span through multiple chunks. They are never treated as circular buffers.
The EXTERNAL mode never allocates / re-allocate buffers
- The virtual buffer size is set to the sum of the provided buffer sizes to avoid receiving more data than we can buffer
- The virtual buffer size can now decrease when data is drained - but the maximum byte offset that can be received still only increases.

Testing

Unit tests have been added

System test to be done.

Documentation

To be done

src/core/recv_buffer.h

src/core/recv_buffer.c

src/core/recv_buffer.h

src/core/recv_buffer.c

nibanks · 2025-01-27T12:36:22Z

src/core/recv_buffer.h

@@ -12,19 +12,31 @@ extern "C" {
 typedef enum QUIC_RECV_BUF_MODE {
    QUIC_RECV_BUF_MODE_SINGLE,      // Only one receive with a single contiguous buffer at a time.
    QUIC_RECV_BUF_MODE_CIRCULAR,    // Only one receive that may indicate two contiguous buffers at a time.
-    QUIC_RECV_BUF_MODE_MULTIPLE     // Multiple independent receives that may indicate up to two contiguous buffers at a time.
+    QUIC_RECV_BUF_MODE_MULTIPLE,    // Multiple independent receives that may indicate up to two contiguous buffers at a time.
+    QUIC_RECV_BUF_MODE_EXTERNAL     // Uses memory buffer provided by the app. Only one receive at a time,


Do you think the fact that we use the word 'external' to mean the app currently has a reference to a chunk via a receive call might confuse things? One option is to all this _APP.

Good point. I wasn't very happy with the name already.
I might go with _APP_OWNED (only _APP seems a bit unclear to me).

src/core/recv_buffer.h

src/core/unittest/RecvBufferTest.cpp

codecov · 2025-02-01T02:33:09Z

Codecov Report

Attention: Patch coverage is 88.80309% with 29 lines in your changes missing coverage. Please review.

Project coverage is 78.18%. Comparing base (451c1a8) to head (d70b754).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
src/core/api.c	74.19%	16 Missing ⚠️
src/core/stream_recv.c	65.00%	7 Missing ⚠️
src/core/recv_buffer.c	97.18%	4 Missing ⚠️
src/core/operation.c	60.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4758      +/-   ##
==========================================
- Coverage   87.00%   78.18%   -8.82%     
==========================================
  Files          56       56              
  Lines       17400    17620     +220     
==========================================
- Hits        15139    13777    -1362     
- Misses       2261     3843    +1582

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

nibanks · 2025-02-05T19:07:02Z

src/core/api.c

+    if (!!(Flags & QUIC_STREAM_OPEN_FLAG_EXTERNAL_BUFFERS) &&
+        Connection->Settings.StreamMultiReceiveEnabled) {


What if they want to use multi-receive for streams that don't use external buffers? I think we can't block that.

You would allow an "app buffer" stream when multi-receive is enabled?
The problem is that this stream would not support the multi-receive behavior (at least for now, we could make it happen), which could be confusing for the app.

And if app takes dependency on the fact that "app buffer" stream ignore multi-receive, then we can't add support without an API break or new flags.

nibanks · 2025-02-05T19:08:57Z

src/core/api.c

+    // The allocation is done here to make the worker thread task failure free.
+    //
+    for (uint32_t i = 0; i < BufferCount; ++i) {
+        QUIC_RECV_CHUNK* Chunk = CXPLAT_ALLOC_NONPAGED(sizeof(QUIC_RECV_CHUNK), QUIC_POOL_RECVBUF);


Should use a pool allocator (to be stored in the worker, like the others) since this is (a) fixed size and (b) on the datapath.

This means each chunk will need a flag to track whether they are allocated from the pool or not then. The size of the chunk seems to be rather optimized (AllocLength restricted to 31bits to avoid a few extra bytes), is it ok to simply add a "flag" byte?

Which makes me realize I need to check the size of the app provided buffers won't override that bit.

src/core/api.c

src/core/operation.c

src/core/stream.h

src/inc/msquic.h

nibanks · 2025-02-05T19:15:02Z

src/inc/quic_trace.h

@@ -103,6 +103,7 @@ typedef enum QUIC_TRACE_API_TYPE {
    QUIC_TRACE_API_DATAGRAM_SEND,
    QUIC_TRACE_API_CONNECTION_COMPLETE_RESUMPTION_TICKET_VALIDATION,
    QUIC_TRACE_API_CONNECTION_COMPLETE_CERTIFICATE_VALIDATION,
+    QUIC_TRACE_API_STREAM_PROVIDE_RECEIVE_BUFFERS,


Our ETW plugin will need to be updated too.

As well as the windbg plugin

nibanks · 2025-02-05T19:16:30Z

src/test/lib/DataTest.cpp

+        TEST_EQUAL(ReceiveContext.ReceivedBytes, BufferSize);
+        TEST_EQUAL(0, memcmp(SendDataBuffer, ReceiveDataBuffer, BufferSize));
+    }
+}
 #endif // QUIC_API_ENABLE_PREVIEW_FEATURES


Why does this need to be in the preview features section?

I haven't done it yet, but I assumed the entire new API should go initially in the preview feature, until we treat it stable.
Let me know if I misunderstand what preview feature section is for and the new API doesn't need to be hidden behind it.

Co-authored-by: Nick Banks <nibanks@microsoft.com>

guhetier force-pushed the guhetier_app_buffers branch 2 times, most recently from 07dc4d9 to 06d5bcf Compare January 22, 2025 00:44

nibanks added Area: API Area: Core Related to the shared, core protocol logic labels Jan 22, 2025

nibanks reviewed Jan 22, 2025

View reviewed changes

src/core/recv_buffer.h Show resolved Hide resolved

nibanks reviewed Jan 22, 2025

View reviewed changes