More WriteGather fixes #109826

adamsitnik · 2024-11-14T15:08:21Z

don't run these tests in parallel, as each test cases uses more than 4 GB ram and disk
fix the test: handle incomplete reads that should happen when we hit the max buffer limit
incomplete write fix:
- pin the buffers only once
- when re-trying, do that only for the actual reminder
- handle Int32 overflow on macOS

…4 GB ram and disk!

…the max buffer limit

- pin the buffers only once - when re-trying, do that only for the actual reminder

dotnet-policy-service · 2024-11-14T15:08:50Z

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

adamsitnik · 2024-11-14T18:14:54Z

src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Unix.cs

-                        ReadOnlyMemory<byte> buffer = buffers[i];
-                        totalBytesToWrite += buffer.Length;
-
-                        MemoryHandle memoryHandle = buffer.Pin();


So far, for incomplete writes we were pinning the memory for every retry attempt. I am not sure if this can create some kind of edge case bugs, but I think we can do it just once, before we enter the main loop

adamsitnik · 2024-11-14T18:17:27Z

src/libraries/System.Runtime/tests/System.IO.FileSystem.Tests/RandomAccess/WriteGatherAsync.cs

            {
                if (asyncMethod)
                {
-                    await RandomAccess.WriteAsync(sfh, writeBuffers, fileOffset);
-                    bytesRead = await RandomAccess.ReadAsync(sfh, readBuffers, fileOffset);


this was a test bug, the test assumed that the read won't ever be a partial read

adamsitnik · 2024-11-14T18:18:24Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2024-11-14T18:18:39Z

Azure Pipelines successfully started running 1 pipeline(s).

adamsitnik · 2024-11-15T10:37:15Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2024-11-15T10:37:25Z

Azure Pipelines successfully started running 1 pipeline(s).

adamsitnik · 2024-11-18T13:03:29Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2024-11-18T13:03:39Z

Azure Pipelines successfully started running 1 pipeline(s).

…going to work?

adamsitnik · 2024-11-18T14:55:55Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2024-11-18T14:56:06Z

Azure Pipelines successfully started running 1 pipeline(s).

adamsitnik · 2024-11-18T17:49:58Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2024-11-18T17:50:09Z

Azure Pipelines successfully started running 1 pipeline(s).

src/native/libs/System.Native/pal_io.c

adamsitnik · 2024-11-19T14:33:44Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2024-11-19T14:33:53Z

Azure Pipelines successfully started running 1 pipeline(s).

…han Int32.MaxValue

adamsitnik · 2024-11-20T10:22:24Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2024-11-20T10:22:34Z

Azure Pipelines successfully started running 1 pipeline(s).

stephentoub · 2024-12-03T17:38:49Z

src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Unix.cs

                    long bytesWritten;
-                    Span<Interop.Sys.IOVector> left = vectors.Slice(buffersOffset);
+                    Span<Interop.Sys.IOVector> left = vectors.Slice(buffersOffset, buffersCount - buffersOffset);


Would it make sense to change L171 to stackalloc Interop.Sys.IOVector[IovStackThreshold].Slice(0, buffersCount), and then vectors will have Length == buffersCount and you won't need the second argument here, since vectors will always be correctly sized?

It looks better, thanks!

stephentoub · 2024-12-03T17:40:26Z

src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Unix.cs

                    {
-                        int n = buffers[i].Length;
+                        int n = buffers[buffersOffset].Length;


Is there anything that could go horribly wrong if the caller-provided IReadOnlyList changed between accesses? It looks ok, just confirming. We'd want to avoid a concurrency bug becoming a memory safety bug.

this was great question! since we perform a "copy" for the purpose of pinning and IOV, I was able to read that value from the copy.

stephentoub · 2024-12-03T20:15:09Z

src/native/libs/System.Native/pal_io.c

+
+        totalLength += vectors[i].Count;
+
+        if (totalLength > INT_MAX)


Can size_t be 32-bit, in which case totalLength might overflow above?

It can be 32-bit (on 32 bit systems), but it's unsigned so for:

inputs that are > int.MaxValue && < uint.MaxValue there will be no overflow

valid inputs that are > uint.MaxValue should not be possible, as it's impossible to allocate more than uint.MaxValue memory on 32 bit systems.

adamsitnik · 2024-12-11T19:08:33Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2024-12-11T19:08:43Z

Azure Pipelines successfully started running 1 pipeline(s).

adamsitnik · 2024-12-11T20:16:48Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2024-12-11T20:16:58Z

Azure Pipelines successfully started running 1 pipeline(s).

adamsitnik · 2024-12-12T09:38:33Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2024-12-12T09:38:46Z

Azure Pipelines successfully started running 1 pipeline(s).

adamsitnik · 2024-12-12T19:21:29Z

/ba-g unrelated job timeout (Started 8h 28m 8s ago and still runinng)

* don't run these tests in parallel, as each test cases uses more than 4 GB ram and disk! * fix the test: handle incomplete reads that should happen when we hit the max buffer limit * incomplete write fix: - pin the buffers only once - when re-trying, do that only for the actual reminder * Use native memory to get OOM a soon as we run out of memory (hoping to avoid the process getting killed on Linux when OOM happens) * For macOS preadv and pwritev can fail with EINVAL when the total length of all vectors overflows a 32-bit integer. * add an assert that is going to warn us if vector.Count is ever more than Int32.MaxValue --------- Co-authored-by: Michał Petryka <35800402+MichalPetryka@users.noreply.github.com>

* don't run these tests in parallel, as each test cases uses more than 4 GB ram and disk! * fix the test: handle incomplete reads that should happen when we hit the max buffer limit * incomplete write fix: - pin the buffers only once - when re-trying, do that only for the actual reminder * Use native memory to get OOM a soon as we run out of memory (hoping to avoid the process getting killed on Linux when OOM happens) * For macOS preadv and pwritev can fail with EINVAL when the total length of all vectors overflows a 32-bit integer. * add an assert that is going to warn us if vector.Count is ever more than Int32.MaxValue --------- Co-authored-by: Michał Petryka <35800402+MichalPetryka@users.noreply.github.com> # Conflicts: # src/libraries/System.IO.FileSystem/tests/RandomAccess/WriteGatherAsync.cs

* don't run these tests in parallel, as each test cases uses more than 4 GB ram and disk! * fix the test: handle incomplete reads that should happen when we hit the max buffer limit * incomplete write fix: - pin the buffers only once - when re-trying, do that only for the actual reminder * Use native memory to get OOM a soon as we run out of memory (hoping to avoid the process getting killed on Linux when OOM happens) * For macOS preadv and pwritev can fail with EINVAL when the total length of all vectors overflows a 32-bit integer. * add an assert that is going to warn us if vector.Count is ever more than Int32.MaxValue --------- Co-authored-by: Michał Petryka <35800402+MichalPetryka@users.noreply.github.com>

adamsitnik added 4 commits November 14, 2024 13:51

macOS does not define IOV_MAX, it uses UIO_MAXIOV instead

02f7f4f

don't run these tests in parallel, as each test cases uses more than …

93c6b8c

…4 GB ram and disk!

fix the test: handle incomplete reads that should happen when we hit …

e7a288c

…the max buffer limit

incomplete write fix:

cb37a35

- pin the buffers only once - when re-trying, do that only for the actual reminder

adamsitnik added the area-System.IO label Nov 14, 2024

adamsitnik added this to the 10.0.0 milestone Nov 14, 2024

adamsitnik self-assigned this Nov 14, 2024

This was referenced Nov 14, 2024

chrome-DebuggerTests timing out #108078

Open

[debugger] Inspector exceptions in debugger tests #109025

Open

chrome-DebuggerTests failing on mono while waiting for pause #109795

Open

adamsitnik commented Nov 14, 2024

View reviewed changes

try one more thing: IOV_MAX should be available everywhere

00d209a

adamsitnik mentioned this pull request Nov 15, 2024

macOS does not define IOV_MAX, it uses UIO_MAXIOV instead #109819

Closed

build-analysis bot mentioned this pull request Nov 15, 2024

iOS tests failing with WORKLOAD TIMED OUT - Killing user command. #108103

Open

I can't reproduce it locally, so here we go...

31a3a00

IOV_MAX is reported as 1024, but it fails with EINVAL. What value is …

2de6b8c

…going to work?

new approach: prefer sysconf(_SC_IOV_MAX) over other options

6a53ac4

MichalPetryka reviewed Nov 18, 2024

View reviewed changes

src/native/libs/System.Native/pal_io.c Outdated Show resolved Hide resolved

build-analysis bot mentioned this pull request Nov 19, 2024

JIT: libraries jitstress Assertion failed 'doesVNMatch' during 'Assertion prop' #109745

Closed

add an assert that is going to warn us if vector.Count is ever more t…

2c685d4

…han Int32.MaxValue

adamsitnik requested a review from stephentoub November 20, 2024 10:23

adamsitnik mentioned this pull request Nov 20, 2024

[release/9.0-staging] Fix few RandomAccess.Write edge case bugs #109646

Open

4 tasks

build-analysis bot mentioned this pull request Nov 20, 2024

SIGKILL (OOM?) while running LibraryImportGenerator.Tests w/o actionable log messages or artifacts dotnet/dnceng#2496

Open

3 tasks

stephentoub reviewed Dec 3, 2024

View reviewed changes

stephentoub approved these changes Dec 3, 2024

View reviewed changes

address code review feedback

6380599

adamsitnik force-pushed the evenMore branch from e4170c4 to 6380599 Compare December 11, 2024 19:48

build-analysis bot mentioned this pull request Dec 11, 2024

System.Formats.Nrbf.Tests timeouts #110285

Closed

Merge branch 'main' into evenMore

819f938

adamsitnik merged commit 4951e38 into dotnet:main Dec 12, 2024
152 of 164 checks passed

adamsitnik mentioned this pull request Dec 12, 2024

[release/8.0-staging] Fix few RandomAccess.Write edge case bugs #109648

Open

4 tasks

github-actions bot locked and limited conversation to collaborators Jan 12, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More WriteGather fixes #109826

More WriteGather fixes #109826

adamsitnik commented Nov 14, 2024 •

edited

Loading

dotnet-policy-service bot commented Nov 14, 2024

adamsitnik Nov 14, 2024 •

edited

Loading

adamsitnik Nov 14, 2024

adamsitnik commented Nov 14, 2024

azure-pipelines bot commented Nov 14, 2024

adamsitnik commented Nov 15, 2024

azure-pipelines bot commented Nov 15, 2024

adamsitnik commented Nov 18, 2024

azure-pipelines bot commented Nov 18, 2024

adamsitnik commented Nov 18, 2024

azure-pipelines bot commented Nov 18, 2024

adamsitnik commented Nov 18, 2024

azure-pipelines bot commented Nov 18, 2024

adamsitnik commented Nov 19, 2024

azure-pipelines bot commented Nov 19, 2024

adamsitnik commented Nov 20, 2024

azure-pipelines bot commented Nov 20, 2024

stephentoub Dec 3, 2024

adamsitnik Dec 12, 2024

stephentoub Dec 3, 2024 •

edited

Loading

adamsitnik Dec 12, 2024

stephentoub Dec 3, 2024

adamsitnik Dec 11, 2024

adamsitnik commented Dec 11, 2024

azure-pipelines bot commented Dec 11, 2024

adamsitnik commented Dec 11, 2024

azure-pipelines bot commented Dec 11, 2024

adamsitnik commented Dec 12, 2024

azure-pipelines bot commented Dec 12, 2024

adamsitnik commented Dec 12, 2024

More WriteGather fixes #109826

More WriteGather fixes #109826

Conversation

adamsitnik commented Nov 14, 2024 • edited Loading

dotnet-policy-service bot commented Nov 14, 2024

adamsitnik Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

adamsitnik Nov 14, 2024

Choose a reason for hiding this comment

adamsitnik commented Nov 14, 2024

azure-pipelines bot commented Nov 14, 2024

adamsitnik commented Nov 15, 2024

azure-pipelines bot commented Nov 15, 2024

adamsitnik commented Nov 18, 2024

azure-pipelines bot commented Nov 18, 2024

adamsitnik commented Nov 18, 2024

azure-pipelines bot commented Nov 18, 2024

adamsitnik commented Nov 18, 2024

azure-pipelines bot commented Nov 18, 2024

adamsitnik commented Nov 19, 2024

azure-pipelines bot commented Nov 19, 2024

adamsitnik commented Nov 20, 2024

azure-pipelines bot commented Nov 20, 2024

stephentoub Dec 3, 2024

Choose a reason for hiding this comment

adamsitnik Dec 12, 2024

Choose a reason for hiding this comment

stephentoub Dec 3, 2024 • edited Loading

Choose a reason for hiding this comment

adamsitnik Dec 12, 2024

Choose a reason for hiding this comment

stephentoub Dec 3, 2024

Choose a reason for hiding this comment

adamsitnik Dec 11, 2024

Choose a reason for hiding this comment

adamsitnik commented Dec 11, 2024

azure-pipelines bot commented Dec 11, 2024

adamsitnik commented Dec 11, 2024

azure-pipelines bot commented Dec 11, 2024

adamsitnik commented Dec 12, 2024

azure-pipelines bot commented Dec 12, 2024

adamsitnik commented Dec 12, 2024

adamsitnik commented Nov 14, 2024 •

edited

Loading

adamsitnik Nov 14, 2024 •

edited

Loading

stephentoub Dec 3, 2024 •

edited

Loading