FileStream.FlushAsync ends up doing synchronous writes

[stephentoub]

https://github.com/dotnet/coreclr/blob/85ed652fef5f1dec7c532bd4963dd3cde0199211/src/System.Private.CoreLib/shared/System/IO/FileStream.Windows.cs#L1547-L1561

            // The always synchronous data transfer between the OS and the internal buffer is intentional 
            // because this is needed to allow concurrent async IO requests. Concurrent data transfer
            // between the OS and the internal buffer will result in race conditions. Since FlushWrite and
            // FlushRead modify internal state of the stream and transfer data between the OS and the 
            // internal buffer, they cannot be truly async. We will, however, flush the OS file buffers
            // asynchronously because it doesn't modify any internal state of the stream and is potentially 
            // a long running process.
            try
            {
                FlushInternalBuffer();
            }
            catch (Exception e)
            {
                return Task.FromException(e);
            }

and

        private void FlushInternalBuffer()
        {
            AssertBufferInvariants();
            if (_writePos > 0)
            {
                FlushWriteBuffer();
            }
            else if (_readPos < _readLength && CanSeek)
            {
                FlushReadBuffer();
            }
        }

This is particularly problematic when the FileStream is configured to use async I/O on Windows, as then FlushWriteBuffer needs to use sync-over-async to do the write, queueing the overlapped I/O and then blocking waiting for the operation to complete... so we have the asynchronous FlushAsync synchronously blocking waiting for the overlapped I/O to complete. Ugh.

Related to https://github.com/dotnet/corefx/issues/31914#issuecomment-427120612
Related to https://github.com/dotnet/corefx/issues/6007
Related to https://github.com/dotnet/corefx/issues/6039
Related to https://github.com/dotnet/corefx/issues/29129
Related to https://github.com/dotnet/corefx/issues/27226
cc: @JeremyKuhne, @danmosemsft

[benaadams]

This is a troubling fix NuGet/NuGet.Client#2488 😢

[stephentoub]

This is a troubling fix

That is actually the fix/workaround I've recommended: setting useAsync:false. As far as I can tell, there's little benefit in that scenario for using overlapped I/O, and even if FileStream were perfect, there's overhead to overlapped I/O.

[benaadams]

Yeah, don't mean its not the right fix for usage; just mean FileStream needs some TLC for async.

Though when I tried it before, I did find it was quite complicated to improve...

[stephentoub]

just mean FileStream needs some TLC for async

It definitely does. :)

[danmoseley]

This cannot fit in 3.0. It's on the radar for v next.