Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TarWriter_WriteEntry_LongFile_Tests require 104GB free #77012

Closed
danmoseley opened this issue Oct 13, 2022 · 11 comments · Fixed by #79899
Closed

TarWriter_WriteEntry_LongFile_Tests require 104GB free #77012

danmoseley opened this issue Oct 13, 2022 · 11 comments · Fixed by #79899
Labels
area-System.Formats.Tar needs-further-triage Issue has been initially triaged, but needs deeper consideration or reconsideration
Milestone
8.0.0

Comments

@danmoseley
Copy link
Member

danmoseley commented Oct 13, 2022 •
edited
Loading 

C:\git\runtime\artifacts\bin\System.Formats.Tar.Tests\Debug\net7.0-windows\testResults.xml:
 6061:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests.WriteEntry_LongFileSize(entryFormat: Gnu, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests" method="WriteEntry_LongFileSize" time="9.3895568" result="Fail">
 6084:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests.WriteEntry_LongFileSize(entryFormat: Pax, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests" method="WriteEntry_LongFileSize" time="0.0203511" result="Fail">
 6107:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests.WriteEntry_LongFileSize(entryFormat: Pax, size: 8589934592, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests" method="WriteEntry_LongFileSize" time="0.0119002" result="Fail">
 6130:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: V7, size: 8589934591, unseekableStream: False)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0089555" result="Fail">
 6152:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Ustar, size: 8589934591, unseekableStream: False)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0068875" result="Fail">
 6174:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Gnu, size: 8589934591, unseekableStream: False)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0039391" result="Fail">
 6189:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Pax, size: 8589934591, unseekableStream: False)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0017366" result="Fail">
 6204:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Pax, size: 8589934592, unseekableStream: False)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.001866" result="Fail">
 6219:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: V7, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0020215" result="Fail">
 6234:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Ustar, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0022796" result="Fail">
 6249:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Gnu, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0016468" result="Fail">
 6264:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Pax, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0016205" result="Fail">
 6279:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Pax, size: 8589934592, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0019141" result="Fail">

example stack

        <failure exception-type="System.IO.IOException">
          <message><![CDATA[System.IO.IOException : There is not enough space on the disk. : 'C:\\Users\\test\\AppData\\Local\\Temp\\#TarWriter_WriteEntry_LongFile_Tests_ufupzyvy.aau\\WriteEntry_LongFileSize_33_hr1z2gp6']]></message>
          <stack-trace><![CDATA[   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\RandomAccess.Windows.cs:line 171
   at System.IO.Strategies.OSFileStreamStrategy.Write(ReadOnlySpan`1 buffer) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\OSFileStreamStrategy.cs:line 246
   at System.IO.Strategies.OSFileStreamStrategy.Write(Byte[] buffer, Int32 offset, Int32 count) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\OSFileStreamStrategy.cs:line 236
   at System.IO.Strategies.BufferedFileStreamStrategy.WriteSpan(ReadOnlySpan`1 source, ArraySegment`1 arraySegment) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\BufferedFileStreamStrategy.cs:line 486
   at System.IO.Strategies.BufferedFileStreamStrategy.Write(Byte[] buffer, Int32 offset, Int32 count) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\BufferedFileStreamStrategy.cs:line 474
   at System.Formats.Tar.WrappedStream.Write(Byte[] buffer, Int32 offset, Int32 count) in C:\git\runtime\src\libraries\System.Formats.Tar\tests\WrappedStream.cs:line 65
   at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Stream.cs:line 79
   at System.Formats.Tar.TarHeader.WriteData(Stream archiveStream, Stream dataStream, Int64 actualLength) in C:\git\runtime\src\libraries\System.Formats.Tar\src\System\Formats\Tar\TarHeader.Write.cs:line 611
   at System.Formats.Tar.TarHeader.WriteAsGnuInternal(Stream archiveStream, Span`1 buffer) in C:\git\runtime\src\libraries\System.Formats.Tar\src\System\Formats\Tar\TarHeader.Write.cs:line 249
   at System.Formats.Tar.TarHeader.WriteAsGnu(Stream archiveStream, Span`1 buffer) in C:\git\runtime\src\libraries\System.Formats.Tar\src\System\Formats\Tar\TarHeader.Write.cs:line 193
   at System.Formats.Tar.TarWriter.WriteEntryInternal(TarEntry entry) in C:\git\runtime\src\libraries\System.Formats.Tar\src\System\Formats\Tar\TarWriter.cs:line 304
   at System.Formats.Tar.TarWriter.WriteEntry(TarEntry entry) in C:\git\runtime\src\libraries\System.Formats.Tar\src\System\Formats\Tar\TarWriter.cs:line 225
   at System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests.WriteEntry_LongFileSize(TarEntryFormat entryFormat, Int64 size, Boolean unseekableStream) in C:\git\runtime\src\libraries\System.Formats.Tar\tests\TarWriter\TarWriter.WriteEntry.LongFile.Tests.cs:line 40
   at InvokeStub_TarWriter_WriteEntry_LongFile_Tests.WriteEntry_LongFileSize(Object, Object, IntPtr*)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)]]></stack-trace>

I have 68GB free on this disk (which includes %temp%). The failure is because these are all async tests, so they are running concurrently, requiring 13 * 8 GB = 104 GB which I don't have.

edit: actually there are 10 async tests, so 80GB, but the non async tests run concurrently, so at least 88GB plus some margin is required.
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Oct 13, 2022
@ghost
Copy link

ghost commented Oct 13, 2022

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details 
C:\git\runtime\artifacts\bin\System.Formats.Tar.Tests\Debug\net7.0-windows\testResults.xml:
 6061:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests.WriteEntry_LongFileSize(entryFormat: Gnu, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests" method="WriteEntry_LongFileSize" time="9.3895568" result="Fail">
 6084:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests.WriteEntry_LongFileSize(entryFormat: Pax, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests" method="WriteEntry_LongFileSize" time="0.0203511" result="Fail">
 6107:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests.WriteEntry_LongFileSize(entryFormat: Pax, size: 8589934592, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests" method="WriteEntry_LongFileSize" time="0.0119002" result="Fail">
 6130:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: V7, size: 8589934591, unseekableStream: False)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0089555" result="Fail">
 6152:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Ustar, size: 8589934591, unseekableStream: False)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0068875" result="Fail">
 6174:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Gnu, size: 8589934591, unseekableStream: False)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0039391" result="Fail">
 6189:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Pax, size: 8589934591, unseekableStream: False)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0017366" result="Fail">
 6204:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Pax, size: 8589934592, unseekableStream: False)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.001866" result="Fail">
 6219:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: V7, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0020215" result="Fail">
 6234:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Ustar, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0022796" result="Fail">
 6249:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Gnu, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0016468" result="Fail">
 6264:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Pax, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0016205" result="Fail">
 6279:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Pax, size: 8589934592, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0019141" result="Fail">

example stack

        <failure exception-type="System.IO.IOException">
          <message><![CDATA[System.IO.IOException : There is not enough space on the disk. : 'C:\\Users\\test\\AppData\\Local\\Temp\\#TarWriter_WriteEntry_LongFile_Tests_ufupzyvy.aau\\WriteEntry_LongFileSize_33_hr1z2gp6']]></message>
          <stack-trace><![CDATA[   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\RandomAccess.Windows.cs:line 171
   at System.IO.Strategies.OSFileStreamStrategy.Write(ReadOnlySpan`1 buffer) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\OSFileStreamStrategy.cs:line 246
   at System.IO.Strategies.OSFileStreamStrategy.Write(Byte[] buffer, Int32 offset, Int32 count) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\OSFileStreamStrategy.cs:line 236
   at System.IO.Strategies.BufferedFileStreamStrategy.WriteSpan(ReadOnlySpan`1 source, ArraySegment`1 arraySegment) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\BufferedFileStreamStrategy.cs:line 486
   at System.IO.Strategies.BufferedFileStreamStrategy.Write(Byte[] buffer, Int32 offset, Int32 count) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\BufferedFileStreamStrategy.cs:line 474
   at System.Formats.Tar.WrappedStream.Write(Byte[] buffer, Int32 offset, Int32 count) in C:\git\runtime\src\libraries\System.Formats.Tar\tests\WrappedStream.cs:line 65
   at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Stream.cs:line 79
   at System.Formats.Tar.TarHeader.WriteData(Stream archiveStream, Stream dataStream, Int64 actualLength) in C:\git\runtime\src\libraries\System.Formats.Tar\src\System\Formats\Tar\TarHeader.Write.cs:line 611
   at System.Formats.Tar.TarHeader.WriteAsGnuInternal(Stream archiveStream, Span`1 buffer) in C:\git\runtime\src\libraries\System.Formats.Tar\src\System\Formats\Tar\TarHeader.Write.cs:line 249
   at System.Formats.Tar.TarHeader.WriteAsGnu(Stream archiveStream, Span`1 buffer) in C:\git\runtime\src\libraries\System.Formats.Tar\src\System\Formats\Tar\TarHeader.Write.cs:line 193
   at System.Formats.Tar.TarWriter.WriteEntryInternal(TarEntry entry) in C:\git\runtime\src\libraries\System.Formats.Tar\src\System\Formats\Tar\TarWriter.cs:line 304
   at System.Formats.Tar.TarWriter.WriteEntry(TarEntry entry) in C:\git\runtime\src\libraries\System.Formats.Tar\src\System\Formats\Tar\TarWriter.cs:line 225
   at System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests.WriteEntry_LongFileSize(TarEntryFormat entryFormat, Int64 size, Boolean unseekableStream) in C:\git\runtime\src\libraries\System.Formats.Tar\tests\TarWriter\TarWriter.WriteEntry.LongFile.Tests.cs:line 40
   at InvokeStub_TarWriter_WriteEntry_LongFile_Tests.WriteEntry_LongFileSize(Object, Object, IntPtr*)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)]]></stack-trace>

I have 68GB free on this disk (which includes %temp%). The failure is because these are all async tests, so they are running concurrently, requiring 13 * 8 GB = 104 GB which I don't have.

Author: danmoseley
Assignees: -
Labels:

area-System.IO
Milestone: -

@jozkee jozkee added this to the 8.0.0 milestone Oct 13, 2022
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Oct 13, 2022
@jozkee
Copy link
Member

jozkee commented Oct 13, 2022

I don't understand, why does [Collection(nameof(DisableParallelization))] doesn't have an effect on async Task tests?

runtime/src/libraries/System.Formats.Tar/tests/TarWriter/TarWriter.WriteEntryAsync.LongFile.Tests.cs

Lines 11 to 21 in a41ac2c

[OuterLoop]
[Collection(nameof(DisableParallelization))] // don't create multiple large files at the same time
public class TarWriter_WriteEntryAsync_LongFile_Tests : TarTestsBase
{
public static IEnumerable<object[]> WriteEntry_LongFileSize_TheoryDataAsync()
=> TarWriter_WriteEntry_LongFile_Tests.WriteEntry_LongFileSize_TheoryData();
[Theory]
[MemberData(nameof(WriteEntry_LongFileSize_TheoryDataAsync))]
public async Task WriteEntry_LongFileSizeAsync(TarEntryFormat entryFormat, long size, bool unseekableStream)
{

@jozkee
Copy link
Member

jozkee commented Oct 13, 2022

cc @carlossanlop

@danmoseley
Copy link
Member Author

danmoseley commented Oct 13, 2022 •
edited
Loading

I don't understand, why does [Collection(nameof(DisableParallelization))] doesn't have an effect on async Task tests?

Seems a lot of people are surprised by this -- xunit/xunit#2003
You could change them to non-async tests, and make the tests wait on the async API. Or maybe create some kind of mutex they share.

BTW I was curious how paralleism in xunit works generally. Each class is an implicit collection of its own, and within a collection everything is serial (except async case above) but collections can run in parallel with each other. So [Collection(nameof(DisableParallelization))] does not mean "don't run anything else in parallel" it means "for the purposes of parallelism, treat all classes with this marker as a single class" or put another way "don't run any tests in this custom collection concurrently with each other" (except async case!)
https://xunit.net/docs/running-tests-in-parallel#parallelism-in-test-frameworks

@danmoseley
Copy link
Member Author

this should probably backport to 7 (when that opens again - after GA)

@danmoseley danmoseley mentioned this issue Oct 20, 2022
Merged
@wfurt wfurt mentioned this issue Dec 16, 2022
Closed
@wfurt
Copy link
Member

wfurt commented Dec 16, 2022

I'm not sure if the link is relevant @danmoseley. They use MaxParallelThreads as method to disable parallelism and that is different from what we do IMHO. People seems to comment that disabling parallelization works ... with large performance hit. We use it for networking and it seems to work as expected.
A single test takes more than hour to execute on my MacBook. (#79773)

This was referenced Dec 19, 2022
Merged
Merged
@jozkee
Copy link
Member

jozkee commented Dec 21, 2022

I ran the test locally with a Console.WriteLine at the time the file is created and other when the FileStream is disposed. It seems that for me locally the tests do run serial. @danmoseley could it be possible that the file deletion is no occurring at the moment that the FS is disposed but rather it is being scheduled and that's why you get "System.IO.IOException : There is not enough space on the disk"?

We could just catch this one exception, ignore it and continue execution.

C:\git\runtime\src\libraries\System.Formats.Tar\tests [main]> dotnet build /t:Test /p:xunitmethodname=*WriteEntry_LongFileSizeAsync /p:outerloop=true
MSBuild version 17.5.0-preview-22580-01+ee6f71d15 for .NET
  Determining projects to restore...
  All projects are up-to-date for restore.
  Microsoft.Interop.SourceGeneration -> C:\git\runtime\artifacts\bin\Microsoft.Interop.SourceGeneration\Debug\netstandard2.0\Microsoft.Interop.SourceGeneration.dll
  LibraryImportGenerator -> C:\git\runtime\artifacts\bin\LibraryImportGenerator\Debug\netstandard2.0\Microsoft.Interop.LibraryImportGenerator.dll
  TestUtilities -> C:\git\runtime\artifacts\bin\TestUtilities\Debug\net6.0\TestUtilities.dll
  StreamConformanceTests -> C:\git\runtime\artifacts\bin\StreamConformanceTests\Debug\net8.0\StreamConformanceTests.dll
  System.Formats.Tar.Tests -> C:\git\runtime\artifacts\bin\System.Formats.Tar.Tests\Debug\net8.0-windows\System.Formats.Tar.Tests.dll
  ----- start Tue 12/20/2022 15:00:53.44 ===============  To repro directly: =====================================================
  pushd C:\git\runtime\artifacts\bin\System.Formats.Tar.Tests\Debug\net8.0-windows\
  "C:\git\runtime\artifacts\bin\testhost\net8.0-windows-Debug-x64\dotnet.exe" exec --runtimeconfig System.Formats.Tar.Tests.runtimeconfig.json --depsfile System.Formats.Tar.Tests.deps.js
  on xunit.console.dll System.Formats.Tar.Tests.dll -xml testResults.xml -nologo -method *WriteEntry_LongFileSizeAsync -notrait category=failing
  popd
  ===========================================================================================================
    Discovering: System.Formats.Tar.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Formats.Tar.Tests (found 1 of 625 test case)
    Starting:    System.Formats.Tar.Tests (parallel test collections = on, max threads = 12)
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_qq3iub23.epw\WriteEntry_LongFileSizeAsync_23_l4oaiclx
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_qq3iub23.epw\WriteEntry_LongFileSizeAsync_23_l4oaiclx
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_ud3zfpxs.eik\WriteEntry_LongFileSizeAsync_23_lhfctyer
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_ud3zfpxs.eik\WriteEntry_LongFileSizeAsync_23_lhfctyer
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_xt2yltgk.m24\WriteEntry_LongFileSizeAsync_23_xitgx3mm
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_xt2yltgk.m24\WriteEntry_LongFileSizeAsync_23_xitgx3mm
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_2ptcj32w.h0e\WriteEntry_LongFileSizeAsync_23_sclzpnul
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_2ptcj32w.h0e\WriteEntry_LongFileSizeAsync_23_sclzpnul
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_pirj3yeg.hba\WriteEntry_LongFileSizeAsync_23_i0r8hy3k
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_pirj3yeg.hba\WriteEntry_LongFileSizeAsync_23_i0r8hy3k
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_g4si45zo.xqo\WriteEntry_LongFileSizeAsync_23_b25vzlgh
     System.Formats.Tar.Tests: [Long Running Test] 'System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync', Elapsed: 00:02:00
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_g4si45zo.xqo\WriteEntry_LongFileSizeAsync_23_b25vzlgh
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_x4x0dcco.uzc\WriteEntry_LongFileSizeAsync_23_rt042dl6
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_x4x0dcco.uzc\WriteEntry_LongFileSizeAsync_23_rt042dl6
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_wkga4ufa.hci\WriteEntry_LongFileSizeAsync_23_6aokaa32
     System.Formats.Tar.Tests: [Long Running Test] 'System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync', Elapsed: 00:04:00
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_wkga4ufa.hci\WriteEntry_LongFileSizeAsync_23_6aokaa32
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_sg5xkwgi.cvt\WriteEntry_LongFileSizeAsync_23_ih5btuol
     System.Formats.Tar.Tests: [Long Running Test] 'System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync', Elapsed: 00:06:00
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_sg5xkwgi.cvt\WriteEntry_LongFileSizeAsync_23_ih5btuol
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_ti2a5q2b.04n\WriteEntry_LongFileSizeAsync_23_3q4obmgc
     System.Formats.Tar.Tests: [Long Running Test] 'System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync', Elapsed: 00:08:00
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_ti2a5q2b.04n\WriteEntry_LongFileSizeAsync_23_3q4obmgc
    Finished:    System.Formats.Tar.Tests
  === TEST EXECUTION SUMMARY ===
     System.Formats.Tar.Tests  Total: 10, Errors: 0, Failed: 0, Skipped: 0, Time: 496.031s
  ----- end Tue 12/20/2022 15:09:11.46 ----- exit code 0 ----------------------------------------------------------

Build succeeded.
    0 Warning(s)
    0 Error(s)

@jozkee jozkee added the needs-further-triage Issue has been initially triaged, but needs deeper consideration or reconsideration label Dec 21, 2022
@adamsitnik
Copy link
Member

could it be possible that the file deletion is no occurring at the moment that the FS is disposed but rather it is being scheduled

It should get deleted as soon as it gets disposed (https://stackoverflow.com/a/1895066/5852046). The question is when does it get disposed?

FileStream gets created with FileOptions.DeleteOnClose without being wrapped in an using block, then it's wrapped by WrappedStream with also no using block.

runtime/src/libraries/System.Formats.Tar/tests/TarWriter/TarWriter.WriteEntryAsync.LongFile.Tests.cs

Lines 23 to 24 in a41ac2c

FileStream tarFile = File.Open(GetTestFilePath(), new FileStreamOptions { Access = FileAccess.ReadWrite, Mode = FileMode.Create, Options = FileOptions.DeleteOnClose });
Stream s = unseekableStream ? new WrappedStream(tarFile, tarFile.CanRead, tarFile.CanWrite, canSeek: false) : tarFile;

Later the WrappedStream is passed to TarReader which should get disposed at the end of the test:

runtime/src/libraries/System.Formats.Tar/tests/TarWriter/TarWriter.WriteEntryAsync.LongFile.Tests.cs

Line 36 in a41ac2c

await using TarReader reader = new TarReader(s);

So the quesiton is whether the disposal of TarReader closes FileStream?

@jozkee
Copy link
Member

jozkee commented Dec 21, 2022 •
edited
Loading

So the quesiton is whether the disposal of TarReader closes FileStream?

It is not disposing it, but it should. So leaveOpen currently has no effect on the stream argument of TarReader. Fixing that may fix this issue.

Alternatively, we can manually dispose the FileStream after we are done with TarReader if we don't want to block on fixing the TarReader.Dispose bug.

Great question, Adam.

@jozkee jozkee mentioned this issue Dec 22, 2022
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Dec 22, 2022
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Dec 22, 2022
akoeplinger added a commit to akoeplinger/runtime that referenced this issue Dec 22, 2022
@akoeplinger
Clean up tar file at the end of WriteEntry_LongFileSizeAsync test
8773a30 
Otherwise multiple tar files could stick around for some time, contributing to issues like dotnet#77012
@akoeplinger akoeplinger mentioned this issue Dec 22, 2022
Closed
@adamsitnik adamsitnik mentioned this issue Dec 23, 2022
Merged
@adamsitnik
Copy link
Member

Even with the correct disposal it takes around 10 minutes to execute single test case (#79920 (comment)). We either need to optimize these tests, reduce the number of test cases or just turn them into manual tests that are not included in Outerloop

@adamsitnik adamsitnik reopened this Dec 23, 2022
@akoeplinger
Copy link
Member

The disk space issue was fixed, we already have one tracking the timeout issue here so closing this: #79773

@jozkee jozkee mentioned this issue Jan 13, 2023
Merged
@ghost ghost locked as resolved and limited conversation to collaborators Jan 22, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-System.Formats.Tar needs-further-triage Issue has been initially triaged, but needs deeper consideration or reconsideration
Projects
None yet
Milestone
8.0.0
Development

Successfully merging a pull request may close this issue.

TarReader should dispose underlying stream if leaveOpen is false jozkee/runtime
6 participants
@jeffhandley @akoeplinger @adamsitnik @danmoseley @wfurt @jozkee