Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Tar async APIs #70574

Merged
merged 75 commits into from
Jul 2, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
bd7cd72
ref: TarEntry.ExtractToFileAsync
carlossanlop May 28, 2022
dcc070a
src: Implement TarEntry.ExtractToFileAsync
carlossanlop May 28, 2022
a6024cf
ref: Add TarReader.GetNextEntryAsync
carlossanlop Jun 10, 2022
a36e1dc
src: Implement TarReader.GetNextEntryAsync
carlossanlop Jun 23, 2022
cb2926e
ref: Add TarWriter.WriteEntryAsync
carlossanlop Jun 23, 2022
5f2249c
src: Implement TarWriter.WriteEntryAsync
carlossanlop Jun 23, 2022
24a26dd
ref: Add TarFile.CreateFromDirectoryAsync
carlossanlop Jun 23, 2022
2fdfeeb
src: Implement TarFile.CreateFromDirectoryAsync
carlossanlop Jun 23, 2022
ea2fc3e
ref: Add TarFile.ExtractToDirectoryAsync
carlossanlop Jun 23, 2022
a36a455
src: Implement TarFile.ExtractToDirectoryAsync
carlossanlop Jun 23, 2022
bfa8831
ref: Add TarWriter.DisposeAsync
carlossanlop Jun 10, 2022
259d5d7
src: Implement TarWriter.DisposeAsync
carlossanlop Jun 10, 2022
2999ff7
ref: Add TarReader.DisposeAsync
carlossanlop Jun 10, 2022
cb6996a
src: Implement TarReader.DisposeAsync
carlossanlop Jun 10, 2022
beb29b6
Don't use partial methods in TarEntry.
carlossanlop Jun 17, 2022
e8e40cb
Don't use partial methods in TarWriter.
carlossanlop Jun 17, 2022
9aeffa4
TarFile exceptions async handling.
carlossanlop Jun 17, 2022
ea3fab0
Move TarWriter awaited code to separate methods.
carlossanlop Jun 17, 2022
99234b2
Update Global Extended Attributes async methods to the latest rebase.
carlossanlop Jun 24, 2022
6cbd587
Remove unused parameter in SetModeOnFile after latest rebase that use…
carlossanlop Jun 24, 2022
1ba88bf
Adjust TarEntry async methods to handle exceptions properly.
carlossanlop Jun 24, 2022
e5a9c9b
ref: Restore System.IAsyncDisposable inheritance in TarReader, it was…
carlossanlop Jun 24, 2022
c9dbfe6
Add async tests for TarReader.TarEntry.ExtractToFile.
carlossanlop Jun 24, 2022
e0029e9
Adjust rented buffer handling based on feedback.
carlossanlop Jun 24, 2022
c657666
This needs to be awaited.
carlossanlop Jun 24, 2022
97f31d9
Reserve ArrayPool in TarWriter WriteEntryAsyncInternal, not MemoryPoo…
carlossanlop Jun 24, 2022
4248de9
Split writing methods in TarHeader into span based and memory based.
carlossanlop Jun 24, 2022
dcd9dfc
tests: TarReader.GetNextEntryAsync.
carlossanlop Jun 24, 2022
244f844
Uncomment InlineData from sync file reading method.
carlossanlop Jun 24, 2022
b3fa156
Simplify exception handling. Only return *Task.FromException in publi…
carlossanlop Jun 24, 2022
ffa5115
Make the TarHeader async methods static if they depend on the struct …
carlossanlop Jun 25, 2022
d45b9ad
Add missing GEA changes from the last PR.
carlossanlop Jun 25, 2022
3c5b92f
TarReader GetNextEntryAsync tests for file assets.
carlossanlop Jun 25, 2022
23894be
Add async compressed tar tests.
carlossanlop Jun 25, 2022
2ff86ec
Adjust checksum calculation and saving after writing in async methods.
carlossanlop Jun 25, 2022
9f674b0
Add async V7 checksum verification test.
carlossanlop Jun 25, 2022
c5019b8
Fix await using for FileStreams in TarFile. Move awaited code to inte…
carlossanlop Jun 25, 2022
c742415
Add async tests for TarFile.CreateFromDirectoryAsync for file assets.
carlossanlop Jun 25, 2022
bf86f37
Add async tests for TarFile.CreateFromDirectoryAsync for streams.
carlossanlop Jun 25, 2022
b750e1b
Add async tests for TarFile.ExtractToDirectoryAsync for file assets.
carlossanlop Jun 25, 2022
aa3eb3e
Add async tests for TarFile.ExtractToDirectoryAsync for streams.
carlossanlop Jun 25, 2022
1074f11
Add tests for TarFile.ExtractToDirectoryAsync platform specific.
carlossanlop Jun 25, 2022
d69965b
Add TarEntry.ExtractToFileAsync tests
carlossanlop Jun 25, 2022
ed71ab1
Join "regular entry vs writer format" tests into single theory test.
carlossanlop Jun 25, 2022
02b2a08
Missing using directive in Unix test file.
carlossanlop Jun 25, 2022
915d5f9
Add TarWriter.WriteEntryAsync tests for entries.
carlossanlop Jun 25, 2022
5eec198
Add TarWriter.WriteEntryAsync tests for file assets.
carlossanlop Jun 25, 2022
39a4560
Fix test faiest failure - Windows test accidentally being run on Linux
carlossanlop Jun 28, 2022
fb60a8c
Set Options to Asynchornous in TarWriter internal method in Windows. …
carlossanlop Jun 28, 2022
331a763
Simplify AdvanceStream* helper methods with Math.Min and ReadExactly*.
carlossanlop Jun 28, 2022
188312d
Simplify CopyBytes* helper methods with ReadExactly*.
carlossanlop Jun 29, 2022
216937f
Add CancellationToken handling.
carlossanlop Jun 29, 2022
4b588bf
Weird spacing.
carlossanlop Jun 29, 2022
82310f2
Move awaited code from GetNextEntryAsync to a private method. This pr…
carlossanlop Jun 29, 2022
2f1428f
Add tests for throwing with a cancelled CancellationToken
carlossanlop Jun 29, 2022
990fc1a
Simplify async calls in tests.
carlossanlop Jun 29, 2022
1bf7f96
Use await using in FileStream used when extracting as regular file.
carlossanlop Jun 29, 2022
f6a0a18
Reuse code in TarEntry.ExtractRelativeToDirectory* methods
carlossanlop Jun 29, 2022
de1fa06
Reuse code in TarFile methods.
carlossanlop Jun 29, 2022
71d6901
Reuse TarWriter.WriteEntry*(string,string) code.
carlossanlop Jun 29, 2022
f629d9b
Add tests for null check in TarWriter.WriteEntry(TarEntry)
carlossanlop Jun 29, 2022
2496e12
Revert renaming of TarHeader.Read methods. Not needed.
carlossanlop Jun 29, 2022
66f851d
Remove unnecessary try-finally in TarHeader.TryGetNextHeader
carlossanlop Jun 29, 2022
bc9c1fd
GetDataStreamAsync should return a null if size is 0, not a MemoryStr…
carlossanlop Jun 29, 2022
4733094
Reuse code in TarHeader.ReadExtendedAttributesBlock*
carlossanlop Jun 29, 2022
5f8f30c
Make GetBasePathForcreateFromDirectory a one-liner.
carlossanlop Jun 29, 2022
631c2ce
Make internal argument verification methods conditioned to debug.
carlossanlop Jun 29, 2022
0f0a5ba
Use stackalloc in TryGetNextHeader instead of ArrayPool rent.
carlossanlop Jun 29, 2022
6cd55bc
Use Array.MaxLength instead of int.MaxValue for array size pre-checks.
carlossanlop Jun 29, 2022
eaa1e85
Use Math.Min for minimumLength in ArrayPool rent in CopyBytes* methods.
carlossanlop Jun 29, 2022
78b8e2a
spacing
carlossanlop Jun 29, 2022
8552796
Only one WriteLeftAlignedBytesAndGetChecksum method is needed. Revert…
carlossanlop Jun 29, 2022
3d444c2
Remove unnecessary TarHeader.Write memory-based methods, use only spa…
carlossanlop Jun 29, 2022
9e173af
Unused directive.
carlossanlop Jun 29, 2022
433c804
Prevent test failure in async tests due to race condition.
carlossanlop Jun 30, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions src/libraries/System.Formats.Tar/ref/System.Formats.Tar.cs
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ internal TarEntry() { }
public string Name { get { throw null; } set { } }
public int Uid { get { throw null; } set { } }
public void ExtractToFile(string destinationFileName, bool overwrite) { }
public System.Threading.Tasks.Task ExtractToFileAsync(string destinationFileName, bool overwrite, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; }
public override string ToString() { throw null; }
}
public enum TarEntryFormat
Expand Down Expand Up @@ -83,24 +84,33 @@ public static partial class TarFile
{
public static void CreateFromDirectory(string sourceDirectoryName, System.IO.Stream destination, bool includeBaseDirectory) { }
public static void CreateFromDirectory(string sourceDirectoryName, string destinationFileName, bool includeBaseDirectory) { }
public static System.Threading.Tasks.Task CreateFromDirectoryAsync(string sourceDirectoryName, System.IO.Stream destination, bool includeBaseDirectory, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; }
public static System.Threading.Tasks.Task CreateFromDirectoryAsync(string sourceDirectoryName, string destinationFileName, bool includeBaseDirectory, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; }
public static void ExtractToDirectory(System.IO.Stream source, string destinationDirectoryName, bool overwriteFiles) { }
public static void ExtractToDirectory(string sourceFileName, string destinationDirectoryName, bool overwriteFiles) { }
public static System.Threading.Tasks.Task ExtractToDirectoryAsync(System.IO.Stream source, string destinationDirectoryName, bool overwriteFiles, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; }
public static System.Threading.Tasks.Task ExtractToDirectoryAsync(string sourceFileName, string destinationDirectoryName, bool overwriteFiles, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; }
}
public sealed partial class TarReader : System.IDisposable
public sealed partial class TarReader : System.IAsyncDisposable, System.IDisposable
{
public TarReader(System.IO.Stream archiveStream, bool leaveOpen = false) { }
public void Dispose() { }
public System.Threading.Tasks.ValueTask DisposeAsync() { throw null; }
public System.Formats.Tar.TarEntry? GetNextEntry(bool copyData = false) { throw null; }
public System.Threading.Tasks.ValueTask<System.Formats.Tar.TarEntry?> GetNextEntryAsync(bool copyData = false, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; }
}
public sealed partial class TarWriter : System.IDisposable
public sealed partial class TarWriter : System.IAsyncDisposable, System.IDisposable
{
public TarWriter(System.IO.Stream archiveStream) { }
public TarWriter(System.IO.Stream archiveStream, bool leaveOpen = false) { }
public TarWriter(System.IO.Stream archiveStream, System.Formats.Tar.TarEntryFormat format = System.Formats.Tar.TarEntryFormat.Pax, bool leaveOpen = false) { }
public System.Formats.Tar.TarEntryFormat Format { get { throw null; } }
public void Dispose() { }
public System.Threading.Tasks.ValueTask DisposeAsync() { throw null; }
public void WriteEntry(System.Formats.Tar.TarEntry entry) { }
public void WriteEntry(string fileName, string? entryName) { }
public System.Threading.Tasks.Task WriteEntryAsync(System.Formats.Tar.TarEntry entry, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; }
public System.Threading.Tasks.Task WriteEntryAsync(string fileName, string? entryName, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; }
}
public sealed partial class UstarTarEntry : System.Formats.Tar.PosixTarEntry
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@
<data name="TarSizeFieldNegative" xml:space="preserve">
<value>The size field is negative in the tar entry '{0}'.</value>
</data>
<data name="TarSizeFieldTooLargeForExtendedAttribute" xml:space="preserve">
<data name="TarSizeFieldTooLargeForEntryType" xml:space="preserve">
<value>The value of the size field for the current entry of type '{0}' is beyond the expected length.</value>
</data>
<data name="TarSymbolicLinkTargetNotExists" xml:space="preserve">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,10 @@ public override int Read(Span<byte> destination)

public override ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken = default)
{
if (cancellationToken.IsCancellationRequested)
{
return ValueTask.FromCanceled<int>(cancellationToken);
}
ThrowIfDisposed();
VerifyPositionInSuperStream();
return ReadAsyncCore(buffer, cancellationToken);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -132,12 +132,20 @@ public override int ReadByte()

public override Task<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
{
if (cancellationToken.IsCancellationRequested)
{
return Task.FromCanceled<int>(cancellationToken);
}
ValidateBufferArguments(buffer, offset, count);
return ReadAsync(new Memory<byte>(buffer, offset, count), cancellationToken).AsTask();
}

public override ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken = default)
{
if (cancellationToken.IsCancellationRequested)
{
return ValueTask.FromCanceled<int>(cancellationToken);
}
ThrowIfDisposed();
ThrowIfBeyondEndOfStream();
return ReadAsyncCore(buffer, cancellationToken);
Expand All @@ -147,6 +155,8 @@ protected async ValueTask<int> ReadAsyncCore(Memory<byte> buffer, CancellationTo
{
Debug.Assert(!_hasReachedEnd);

cancellationToken.ThrowIfCancellationRequested();

Comment on lines +158 to +159
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it seems like there is no need to check for the cancellation again here, as ReadAsync (the caller) always does that

Suggested change
cancellationToken.ThrowIfCancellationRequested();

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was under the impression that the token cancellation status should be checked on every method.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was under the impression that the token cancellation status should be checked on every method

As long as some work was executed between them (so it could have actually changed in the meantime). At least this is how I see it.

if (_positionInSuperStream > _endInSuperStream - buffer.Length)
{
buffer = buffer.Slice(0, (int)(_endInSuperStream - _positionInSuperStream));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,28 +11,28 @@ namespace System.Formats.Tar
public abstract partial class TarEntry
{
// Unix specific implementation of the method that extracts the current entry as a block device.
partial void ExtractAsBlockDevice(string destinationFileName)
private void ExtractAsBlockDevice(string destinationFileName)
{
Debug.Assert(EntryType is TarEntryType.BlockDevice);
Interop.CheckIo(Interop.Sys.CreateBlockDevice(destinationFileName, (uint)Mode, (uint)_header._devMajor, (uint)_header._devMinor), destinationFileName);
}

// Unix specific implementation of the method that extracts the current entry as a character device.
partial void ExtractAsCharacterDevice(string destinationFileName)
private void ExtractAsCharacterDevice(string destinationFileName)
{
Debug.Assert(EntryType is TarEntryType.CharacterDevice);
Interop.CheckIo(Interop.Sys.CreateCharacterDevice(destinationFileName, (uint)Mode, (uint)_header._devMajor, (uint)_header._devMinor), destinationFileName);
}

// Unix specific implementation of the method that extracts the current entry as a fifo file.
partial void ExtractAsFifo(string destinationFileName)
private void ExtractAsFifo(string destinationFileName)
{
Debug.Assert(EntryType is TarEntryType.Fifo);
Interop.CheckIo(Interop.Sys.MkFifo(destinationFileName, (uint)Mode), destinationFileName);
}

// Unix specific implementation of the method that extracts the current entry as a hard link.
partial void ExtractAsHardLink(string targetFilePath, string hardLinkFilePath)
private void ExtractAsHardLink(string targetFilePath, string hardLinkFilePath)
{
Debug.Assert(EntryType is TarEntryType.HardLink);
Debug.Assert(!string.IsNullOrEmpty(targetFilePath));
Expand All @@ -41,7 +41,7 @@ partial void ExtractAsHardLink(string targetFilePath, string hardLinkFilePath)
}

// Unix specific implementation of the method that specifies the file permissions of the extracted file.
partial void SetModeOnFile(SafeFileHandle handle, string destinationFileName)
private void SetModeOnFile(SafeFileHandle handle)
{
// Only extract USR, GRP, and OTH file permissions, and ignore
// S_ISUID, S_ISGID, and S_ISVTX bits.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,28 +10,28 @@ namespace System.Formats.Tar
public abstract partial class TarEntry
{
// Throws on Windows. Block devices are not supported on this platform.
partial void ExtractAsBlockDevice(string destinationFileName)
private void ExtractAsBlockDevice(string destinationFileName)
{
Debug.Assert(EntryType is TarEntryType.BlockDevice or TarEntryType.CharacterDevice);
throw new InvalidOperationException(SR.IO_DeviceFiles_NotSupported);
}

// Throws on Windows. Character devices are not supported on this platform.
partial void ExtractAsCharacterDevice(string destinationFileName)
private void ExtractAsCharacterDevice(string destinationFileName)
{
Debug.Assert(EntryType is TarEntryType.BlockDevice or TarEntryType.CharacterDevice);
throw new InvalidOperationException(SR.IO_DeviceFiles_NotSupported);
}

// Throws on Windows. Fifo files are not supported on this platform.
partial void ExtractAsFifo(string destinationFileName)
private void ExtractAsFifo(string destinationFileName)
{
Debug.Assert(EntryType is TarEntryType.Fifo);
throw new InvalidOperationException(SR.IO_FifoFiles_NotSupported);
}

// Windows specific implementation of the method that extracts the current entry as a hard link.
partial void ExtractAsHardLink(string targetFilePath, string hardLinkFilePath)
private void ExtractAsHardLink(string targetFilePath, string hardLinkFilePath)
{
Debug.Assert(EntryType is TarEntryType.HardLink);
Debug.Assert(!string.IsNullOrEmpty(targetFilePath));
Expand All @@ -41,7 +41,7 @@ partial void ExtractAsHardLink(string targetFilePath, string hardLinkFilePath)

// Mode is not used on Windows.
#pragma warning disable CA1822 // Member 'SetModeOnFile' does not access instance data and can be marked as static
partial void SetModeOnFile(SafeFileHandle handle, string destinationFileName)
private void SetModeOnFile(SafeFileHandle handle)
#pragma warning restore CA1822
{
// TODO: Verify that executables get their 'executable' permission applied on Windows when extracted, if applicable. https://github.com/dotnet/runtime/issues/68230
Expand Down
Loading