-
Notifications
You must be signed in to change notification settings - Fork 696
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
restore: trying out mmap-based file extraction #3524
Conversation
2e8f363
to
e1c1e9c
Compare
src/NuGet.Core/NuGet.Packaging/PackageExtraction/StreamExtensions.cs
Outdated
Show resolved
Hide resolved
src/NuGet.Core/NuGet.Packaging/PackageExtraction/StreamExtensions.cs
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've never used mmap before in .NET, and don't remember reading much about it in the past. How confident are we that this will work for everyone in all scenarios? Should we consider checking an environment variable to allow people to turn it off, considering otherwise the only workaround will be to downgrade to older tooling, which might not be an option for other reasons?
Should we bother flushing? Basically none of our code flushes streams to disk
Dispose implicitly flushes. And therefore it's not a perf hit. In fact, for async code this is a problem because dispose is synchornous, and therefore flushing on dispose can block the thread when the code is otherwise async. Therefore for perf it's important to call FlushAsync
in async code before disposing. But I think this is a bigger problem with network streams, rather than disk IO. In any case, the method modified in this PR is not async, so it doesn't really matter.
I don't know if it is true that mmaped files also flush on dispose for mmaped files in .NET. I would guess so, as it would be very confusing for developers like us knowing that one type of stream flushes on dispose but another does not.
a process crash won't prevent mmapped files from being persisted
making mmap more relisient than "normal" streams :)
src/NuGet.Core/NuGet.Packaging/PackageExtraction/StreamExtensions.cs
Outdated
Show resolved
Hide resolved
src/NuGet.Core/NuGet.Packaging/Core/ExtractPackageFileDelegate.cs
Outdated
Show resolved
Hide resolved
src/NuGet.Core/NuGet.Packaging/PackageExtraction/StreamExtensions.cs
Outdated
Show resolved
Hide resolved
src/NuGet.Core/NuGet.Packaging/PackageExtraction/StreamExtensions.cs
Outdated
Show resolved
Hide resolved
src/NuGet.Core/NuGet.Packaging/PackageExtraction/StreamExtensions.cs
Outdated
Show resolved
Hide resolved
src/NuGet.Core/NuGet.Packaging/Core/ExtractPackageFileDelegate.cs
Outdated
Show resolved
Hide resolved
src/NuGet.Core/NuGet.Packaging/PackageExtraction/StreamExtensions.cs
Outdated
Show resolved
Hide resolved
src/NuGet.Core/NuGet.Packaging/PackageExtraction/StreamExtensions.cs
Outdated
Show resolved
Hide resolved
src/NuGet.Core/NuGet.Packaging/PackageExtraction/StreamExtensions.cs
Outdated
Show resolved
Hide resolved
ef632d6
to
e31d8b2
Compare
e31d8b2
to
4d5ba72
Compare
This is ready for another review! I simplified the whole thing. The only stream type that didn't have |
src/NuGet.Core/NuGet.Packaging/PackageExtraction/StreamExtensions.cs
Outdated
Show resolved
Hide resolved
src/NuGet.Core/NuGet.Packaging/PackageExtraction/StreamExtensions.cs
Outdated
Show resolved
Hide resolved
@dtivel fixed. I had to change how I'm doing the mmap because of issues I had on Linux, but this version is actually passing tests AND performing well :) |
Fixes: NuGet/Home#9807
Results are promising:
Benchmark results from one of our test benchmark projects (the Orchard example)
Note: I couldn't get the NuGet.Client-based benchmarks to work, but the other two worked fine.
TODO
try{}
around the mmap code that falls through to a regular write. These are rare enough that it shouldn't affect performance, and if we catch the right exception, won't affect correctness (because the incoming stream won't have been consumed at all).