Add async ZipFile APIs #1541

ericstj · 2016-01-25T22:33:08Z

All ZipFile APIs are currently synchronous. This means manipulations to zip files will always block a thread. We should investigate using an async file and calling async IO APIs (ReadAsync/WriteAsync) to free up the thread while IO is happening. dotnet/corefx#5680

Edit by @carlossanlop:

System.IO.Compression

- public partial class ZipArchive : IDisposable
+ public partial class ZipArchive : IAsyncDisposable, IDisposable
  {
      public void Dispose();
      protected virtual void Dispose(bool disposing);
+     public ValueTask DisposeAsync(); // From the new implemented interface

      public ReadOnlyCollection<ZipArchiveEntry> Entries { get; }
+     public Task<ReadOnlyCollection<ZipArchiveEntry>> GetEntriesAsync(CancellationToken cancellationToken = default);
+ // Should this use IAsyncEnumerable?

      public ZipArchiveEntry? GetEntry(string entryName);
+     public Task<ZipArchiveEntry?> GetEntryAsync(string entryName, CancellationToken cancellationToken = default);
...

Note there is no CreateEntryAsync above. That is because I didn't find any code inside CreateEntry (or inside DoCreateEntry) that can be made async, Aside for the cancellationToken.ThrowIfCancellationRequested() call that would have to be added at the top.

But here's the method in case we want to add it regardless. We can get it approved here, and if in the end it is not needed, we don't have to ship it:

...
      public ZipArchiveEntry CreateEntry(string entryName);
+     public Task<ZipArchiveEntry> CreateEntryAsync(string entryName, CancellationToken cancellationToken = default);  
  }

Also:

  public partial class ZipArchiveEntry
  {
      void Delete();
+     public Task DeleteAsync(CancellationToken cancellationToken = default);

      public Stream Open();
+     Task<Stream> OpenAsync(CancellationToken cancellationToken = default);
  }

Usage examples:

// Open read, get entries, get entry stream
FileStream fs = File.OpenRead("/path/to/archive.zip");
await using (ZipArchive archive =  new ZipArchive(stream: fs, ZipArchiveMode.Read, ct))
{
    foreach (ZipArchiveEntry entry in await archive.GetEntriesAsync(ct))
    {
        Stream s = await entry.OpenAsync(ct);
        // Do something
    }
}

// Open update, get entries, delete entry
FileStream fs = File.OpenRead("/path/to/archive.zip");
await using (ZipArchive archive = new ZipArchive(stream: fs, ZipArchiveMode.Update, ct))
{
    foreach (ZipArchiveEntry entry in await archive.GetEntriesAsync(ct))
    {
        if (/* some condition */)
        {
            await entry.DeleteAsync(ct);
        }
    }
}

System.IO.Compression.ZipFile

  public static partial class ZipFile
  {
      public static void CreateFromDirectory(string sourceDirectoryName, Stream destination);
+     public static Task CreateFromDirectoryAsync(string sourceDirectoryName, Stream destination, CancellationToken cancellationToken = default);

      public static void CreateFromDirectory(string sourceDirectoryName, Stream destination, CompressionLevel compressionLevel, bool includeBaseDirectory);
+     public static Task CreateFromDirectoryAsync(string sourceDirectoryName, Stream destination, CompressionLevel compressionLevel, bool includeBaseDirectory, CancellationToken cancellationToken = default);

      public static void CreateFromDirectory(string sourceDirectoryName, Stream destination, CompressionLevel compressionLevel, bool includeBaseDirectory, Encoding? entryNameEncoding);
+     public static Task CreateFromDirectoryAsync(string sourceDirectoryName, Stream destination, CompressionLevel compressionLevel, bool includeBaseDirectory, Encoding? entryNameEncoding, CancellationToken cancellationToken = default);

      public static void CreateFromDirectory(string sourceDirectoryName, string destinationArchiveFileName);
+     public static Task CreateFromDirectoryAsync(string sourceDirectoryName, string destinationArchiveFileName, CancellationToken cancellationToken = default);

      public static void CreateFromDirectory(string sourceDirectoryName, string destinationArchiveFileName, CompressionLevel compressionLevel, bool includeBaseDirectory);
+     public static Task CreateFromDirectoryAsync(string sourceDirectoryName, string destinationArchiveFileName, CompressionLevel compressionLevel, bool includeBaseDirectory, CancellationToken cancellationToken = default);

      public static void CreateFromDirectory(string sourceDirectoryName, string destinationArchiveFileName, CompressionLevel compressionLevel, bool includeBaseDirectory, Encoding? entryNameEncoding);
+     public static Task CreateFromDirectoryAsync(string sourceDirectoryName, string destinationArchiveFileName, CompressionLevel compressionLevel, bool includeBaseDirectory, Encoding? entryNameEncoding, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(Stream source, string destinationDirectoryName);
+     public static Task ExtractToDirectoryAsync(Stream source, string destinationDirectoryName, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(Stream source, string destinationDirectoryName, bool overwriteFiles);
+     public static Task ExtractToDirectoryAsync(Stream source, string destinationDirectoryName, bool overwriteFiles, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(Stream source, string destinationDirectoryName, Encoding? entryNameEncoding);
+     public static Task ExtractToDirectoryAsync(Stream source, string destinationDirectoryName, Encoding? entryNameEncoding, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(Stream source, string destinationDirectoryName, Encoding? entryNameEncoding, bool overwriteFiles);
+     public static Task ExtractToDirectoryAsync(Stream source, string destinationDirectoryName, Encoding? entryNameEncoding, bool overwriteFiles, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(string sourceArchiveFileName, string destinationDirectoryName);
+     public static Task ExtractToDirectoryAsync(string sourceArchiveFileName, string destinationDirectoryName, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(string sourceArchiveFileName, string destinationDirectoryName, bool overwriteFiles);
+     public static Task ExtractToDirectoryAsync(string sourceArchiveFileName, string destinationDirectoryName, bool overwriteFiles, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(string sourceArchiveFileName, string destinationDirectoryName, Encoding? entryNameEncoding);
+     public static Task ExtractToDirectoryAsync(string sourceArchiveFileName, string destinationDirectoryName, Encoding? entryNameEncoding, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(string sourceArchiveFileName, string destinationDirectoryName, Encoding? entryNameEncoding, bool overwriteFiles);
+     public static Task ExtractToDirectoryAsync(string sourceArchiveFileName, string destinationDirectoryName, Encoding? entryNameEncoding, bool overwriteFiles, CancellationToken cancellationToken = default);

      public static ZipArchive Open(string archiveFileName, ZipArchiveMode mode);
+     public static Task<ZipArchive> OpenAsync(string archiveFileName, ZipArchiveMode mode, CancellationToken cancellationToken = default);

      public static ZipArchive Open(string archiveFileName, ZipArchiveMode mode, Encoding? entryNameEncoding);
+     public static Task<ZipArchive> OpenAsync(string archiveFileName, ZipArchiveMode mode, Encoding? entryNameEncoding, CancellationToken cancellationToken = default);

      public static ZipArchive OpenRead(string archiveFileName);
+     public static Task<ZipArchive> OpenReadAsync(string archiveFileName, CancellationToken cancellationToken = default);
  }
  
  public static partial class ZipFileExtensions
  {
      public static ZipArchiveEntry CreateEntryFromFile(this ZipArchive destination, string sourceFileName, string entryName);
+     public static Task<ZipArchiveEntry> CreateEntryFromFileAsync(this ZipArchive destination, string sourceFileName, string entryName, CancellationToken cancellationToken = default);

      public static ZipArchiveEntry CreateEntryFromFile(this ZipArchive destination, string sourceFileName, string entryName, CompressionLevel compressionLevel);
+     public static Task<ZipArchiveEntry> CreateEntryFromFileAsync(this ZipArchive destination, string sourceFileName, string entryName, CompressionLevel compressionLevel, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(this ZipArchive source, string destinationDirectoryName);
+     public static Task ExtractToDirectoryAsync(this ZipArchive source, string destinationDirectoryName, CancellationToken cancellationToken = default);

      public static void ExtractToDirectory(this ZipArchive source, string destinationDirectoryName, bool overwriteFiles);
+     public static Task ExtractToDirectoryAsync(this ZipArchive source, string destinationDirectoryName, bool overwriteFiles, CancellationToken cancellationToken = default);

      public static void ExtractToFile(this ZipArchiveEntry source, string destinationFileName);
+     public static Task ExtractToFileAsync(this ZipArchiveEntry source, string destinationFileName, CancellationToken cancellationToken = default);

      public static void ExtractToFile(this ZipArchiveEntry source, string destinationFileName, bool overwrite);
+     public static Task ExtractToFileAsync(this ZipArchiveEntry source, string destinationFileName, bool overwrite, CancellationToken cancellationToken = default);
  }

Usage examples

/// ZipFile creation and extraction

MemoryStream ms = new();
await CreateFromDirectoryAsync(sourceDirectoryName: "C:/path/to/source/folder", destination: ms, ct);
ms.Position = 0;
await ExtractToDirectoryAsync(source: ms, destinationDirectoryName: "C:/path/to/destination/folder", ct);

/// ZipFile open, ZipFileExtensions create entry from file

await using (ZipArchive archive = await ZipFile.OpenAsync(archiveFileName: "/path/to/archive.zip", ZipArchiveMode.Update, ct))
{
    await archive.CreateEntryFromFileAsync(sourceFileName: "/path/to/file.txt", entryName: "file.txt", ct);
}

// ZipFile open read, ZipFileExtensions extract entry to file

await using (ZipArchive archive = await ZipFile.OpenReadAsync(archiveFileName: "/path/to/archive.zip", ct))
{
    foreach (ZipArchiveEntry entry in await archive.GetEntriesAsync(ct))
    {
        string fileName = Path.Combine(rootFolder, entry.Name);
        await archive.ExtractToFileAsync(destinationFileName: fileName, overwrite: true, ct);
    }
    // or instead
    await archive.ExtractToDirectoryAsync(rootFolder, ct);
}

Plan

The text was updated successfully, but these errors were encountered:

danmoseley · 2016-12-27T18:25:27Z

This needs someone to write up a formal API proposal then we can review and if approved, offer up for implementation. Anyone?

Docs on the API review process and see Jon's API request for a great example of a strong proposal. The more concrete the proposal is (e.g. has examples of usage, real-world scenarios, fleshed out API), the more discussion it will start and the better the chances of us being able to push the addition through the review process.

abdulbeard · 2018-06-19T15:45:10Z

Forked the repo, and working on an api proposal.
Seems like most of the work outlined in this item is already done. e.g
https://github.com/dotnet/corefx/blob/master/src/System.IO.FileSystem/src/System/IO/File.cs#L682
https://github.com/dotnet/corefx/blob/master/src/System.IO.FileSystem/src/System/IO/File.cs#L732

That probably means this item is done, right?
@danmosemsft

ianhays · 2018-06-19T16:51:00Z

Not quite, @abdulbeard. This particular issue is around adding ZipFile async APIs.

For example, currently we have ZipFile.ExtractToDirectory(string, string). This issue is around adding async versions of those API e.g. ZipFile.ExtractToDirectoryAsync(string, string).

abdulbeard · 2018-06-19T16:59:25Z

Gotcha @ianhays . I did a terrible job reading the title 😛

abdulbeard · 2018-06-19T17:42:50Z

Summary

Curently, System.IO.Compression.Zipfile doesn't have async methods for CreateFromDirectory, ExtractToDirectory, Open and OpenRead. This means that these zipfile operations are thread-blocking, and not asynchronous, and hence, comparatively less performant.

Proposed API

namespace System.IO.Compression {
    public partial class Zipfile {
        public static Task CreateFromDirectoryAsync(string sourceDirectoryName, string destinationArchiveFileName, CancellationToken cancellationToken = default(CancellationToken));

        public static Task CreateFromDirectoryAsync(string sourceDirectoryName, string destinationArchiveFileName,	CompressionLevel compressionLevel, bool includeBaseDirectory, CancellationToken cancellationToken = default(CancellationToken));

        public static Task CreateFromDirectoryAsync(string sourceDirectoryName, string destinationArchiveFileName,	CompressionLevel compressionLevel, bool includeBaseDirectory, Encoding entryNameEncoding, CancellationToken cancellationToken = default(CancellationToken));

        public static Task ExtractToDirectoryAsync(string sourceArchiveFileName, string destinationDirectoryName, CancellationToken cancellationToken = default(CancellationToken));

        public static Task ExtractToDirectoryAsync(string sourceArchiveFileName, string destinationDirectoryName, Encoding entryNameEncoding, CancellationToken cancellationToken = default(CancellationToken));

        public static Task<ZipArchive> OpenAsync(string archiveFileName, ZipArchiveMode mode, CancellationToken cancellationToken = default(CancellationToken));

        public static Task<ZipArchive> OpenAsync(string archiveFileName, ZipArchiveMode mode, Encoding entryNameEncoding, CancellationToken cancellationToken = default(CancellationToken));

        public static Task<ZipArchive> OpenReadAsync(string archiveFileName, CancellationToken cancellationToken = default(CancellationToken));
    }
}

Expected Use and Benefits

These new functions are asynchronous versions of their synchronous counterparts. This allows the .net runtime to allow the thread to do other work while the asynchronous operation completes (as mandated by the TPL and async/await scheduling).

Zipfile operations will no longer be thread-blocking, improving performance.
Long-running Zipfile operations can now be cancelled via the CancellationToken, by the user, allowing them more control.
Asynchronous FileStream or other OS/runtime optimizations can be utilized to improve performance here.

abdulbeard · 2018-06-19T17:43:14Z

Here's a proposal ^. Looking forward to constructive feedback and insight.

svick · 2018-06-19T19:56:41Z

Should async APIs be added to ZipArchive at the same time? For example, consider the following code:

using (var archive = await ZipFile.OpenAsync("archive.zip", ZipArchiveMode.Update))
{
    archive.CreateEntry("empty.txt");
}

using (var archive = await ZipFile.OpenReadAsync("archive.zip"))
{
    foreach (var entry in archive.Entries)
    {
        Console.WriteLine(entry.FullName);
    }
}

As far as I can tell from skimming the source code, even though the above code uses the proposed async APIs as much as it can, it still blocks when accessing Entries and, more insidiously, when disposing the first archive.

abdulbeard · 2018-06-27T13:53:55Z

Here's a proposal for ZipArchive class:

namespace System.IO.Compression {
    public class ZipArchive {
        public Task<ReadOnlyCollection<ZipArchiveEntry>> GetEntriesAsync();
        public Task<ZipArchiveEntry> GetEntryAsync(string entryName, CancellationToken cancellationToken = default(cancellationToken));
    }
}

svick · 2018-06-27T14:06:13Z

@abdulbeard How does that help? As far as I can tell, CreateEntry does not block in the current implementation. On the other hand, accessing Entries and calling Dispose() can block, depending on the mode.

abdulbeard · 2018-06-27T16:00:11Z

@svick you're right. Async versions of CreateEntry don't do much, since their underlying functionality is not blocking.
On the matter of Dispose() being async, not sure there's a good way to do that.
In the matter of Entries, I can imagine replacing that with something like:

public Task<ReadOnlyCollection<ZipArchiveEntry>> GetEntriesAsync();
public Task<ZipArchiveEntry> GetEntryAsync(string entryName, CancellationToken cancellationToken = default(cancellationToken));

Amended the proposal above.

danmoseley · 2018-06-27T16:04:36Z

@JeremyKuhne could you give feedback on this proposal? Perhaps we can move it forward

svick · 2018-06-28T14:39:43Z

@abdulbeard

On the matter of Dispose() being async, not sure there's a good way to do that.

Yeah, there isn't a great approach until dotnet/roslyn#114 is added. But in the meantime, I think there are ways to make it work. The approaches I considered are:

Add FlushAsync(). This would write the current state of the ZipArchive to disk, which means that calling Dispose() right after FlushAsync() would not block.

Example usage:

using (var archive = await ZipFile.OpenAsync("archive.zip", ZipArchiveMode.Update))
{
    archive.CreateEntry("empty.txt");
    await archive.FlushAsync();
}

Add DisposeAsync(). Once IAsyncDisposable, using statements, and async/await roslyn#114 is added, you would be able to use it with using.

Example usage:

var archive = await ZipFile.OpenAsync("archive.zip", ZipArchiveMode.Update)
try
{
    archive.CreateEntry("empty.txt");
}
finally
{
    await archive.DisposeAsync();
}

Looking at the two examples above, I think adding FlushAsync() is the better option.

abdulbeard · 2018-07-02T20:45:20Z

@svick
To me it makes sense to wait on resolution of dotnet/roslyn/#114 before deciding on FlusyAsync vs DisposeAsync.
But I can also see value in finishing out this stage with FlushAsync, and then once we have an async dispose, let that replace FlushAsync.
But I'd love to hear from @JeremyKuhne etc. as well.

abdulbeard · 2018-07-23T18:22:13Z

@danmosemsft @JeremyKuhne any suggestions for changes to ^, or shall we move forward?

abdulbeard · 2018-09-05T16:47:06Z

Anything I can do to help move this along?

abdulbeard · 2018-12-07T20:30:53Z

crickets..........

JeremyKuhne · 2018-12-07T20:58:45Z

@abdulbeard Sorry, the actual owner here moved on to another project and this got lost in the shuffle. 😦 Thanks for poking on it.

@stephentoub can you give some direction/guidance on the disposal questions? @ericstj do you think this proposal provides what you were looking for?

Presumably IAsyncEnumerable<ZipArchiveEntry> would be the right choice for GetEntriesAsync...

JeremyKuhne · 2018-12-07T21:17:01Z

@ahsonkhan, @ViktorHofer can one of you take this and drive it forward?

stephentoub · 2018-12-07T21:31:56Z

can you give some direction/guidance on the disposal questions?

We have added IAsyncDisposable and C# is adding await using that works with it. If disposing a ZipArchive does work that could be asynchronous (e.g. flushing a stream), having ZipArchive implement it is quite reasonable. Note that Stream itself now implements IAsyncDisposable as well.

stephentoub · 2018-12-07T21:33:27Z

Presumably IAsyncEnumerable would be the right choice for GetEntriesAsync

It depends. Is ZipArchive lazily reading from a stream, e.g. when you go to read the next entry, might that perform a read on the stream? If yes, then having it return an IAsyncEnumerable could make sense. If no, then it should probably return a synchronous enumerable.

TianqiZhang · 2019-03-22T02:30:34Z

Just curious is there any update on this issue? We have a .NET Core project that creates zip files in Azure blobs, and we are forced to use the sync version of ZipArchive apis because the async ones will cause unit tests to hang.

kccsf · 2019-04-09T19:49:20Z

I have some code that is generating and streaming a zip archive on the fly, which fails if AllowSynchronousIO is false, which is the default from core 3.0.0-preview3 onwards:
dotnet/aspnetcore#7644

System.InvalidOperationException: Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead.
at Microsoft.AspNetCore.Server.IIS.Core.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at Microsoft.AspNetCore.Server.IIS.Core.WrappingStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Compression.PositionPreservingWriteOnlyStreamWrapper.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.BinaryWriter.Write(UInt32 value)
at System.IO.Compression.ZipArchiveEntry.WriteCentralDirectoryFileHeader()
at System.IO.Compression.ZipArchive.WriteFile()
at System.IO.Compression.ZipArchive.Dispose(Boolean disposing)
at System.IO.Compression.ZipArchive.Dispose()

m-valenta · 2019-05-28T10:33:21Z

Like @kccsf I have simillar issue with the Kestrel server configured to disallow synchronous IO. With the current implementation I can not use ZipArchive directly with the Kestrel's response stream - because of synchronous disposing.

ahsonkhan · 2019-09-11T04:26:09Z

cc @carlossanlop

juunas11 · 2019-10-23T16:43:26Z

Just hit this today as well. :\ After writing the entries, Dispose() throws the exception. Worked around by allowing sync I/O at that point for now, but would be nice to have some async API that I could call to remove the need for sync I/O.

gragra33 · 2022-08-27T15:17:26Z

I have found a work around for now ...

async Task DoZipFileWorkAsync()
{
     // ZipArchive is a Synchronous Block, so yield asynchronously
    await Task.Yield();

    // Open archive
    using ZipArchive zipArchive = new(File.OpenRead(path));

   // ... do work
}

All work is now done on a .Net TreadPool Worker Thread.

Casper-Olsen · 2022-08-27T17:14:01Z

I have found a work around for now ...

async Task DoZipFileWorkAsync()
{
     // ZipArchive is a Synchronous Block, so yield asynchronously
    await Task.Yield();

    // Open archive
    using ZipArchive zipArchive = new(File.OpenRead(path));

   // ... do work
}

All work is now done on a .Net TreadPool Worker Thread.

This is not really a work around, since all work with the ZipArchive is still done synchronously.
And it's not guaranteed that work will continue on the thread pool, after Task.Yield. Only if there is no SynchronizationContext.

gragra33 · 2022-08-27T17:32:53Z

This is not really a work around, since all work with the ZipArchive is still done synchronously. And it's not guaranteed that work will continue on the thread pool, after Task.Yield. Only if there is no SynchronizationContext.

I do agree to some extent, however testing done works as expected in a worker thread, not the Main Thread. I would still require propper asynchronous support to replace this temporary work around.

synek317 · 2022-10-03T10:51:54Z

I wanted to try adding async and ended up with copy-pasting dotnet source code of ZipFileExtensions.ExtractToDirectory and adding async here and there.

I even added a progress report.

The downside is, I had to use english exception message since the methods to get translated messages from resources are internal.

public static class ZipFileAsyncExtensions
{
  public static async Task ExtractToDirectoryAsync(
    this ZipArchive             source,
    string                      destinationDirectoryName,
    bool                        overwriteFiles,
    Action<IZipExtractProgress> progressCallback  = null,
    CancellationToken           cancellationToken = default
  )
  {
    var progress = progressCallback is null
      ? default
      : new ZipExtractProgress
      {
        EntriesCount = source.Entries.Count(),
        BytesTotal = source.Entries.Sum(e => e.Length)
      };

    foreach (ZipArchiveEntry entry in source.Entries)
    {
      await entry.ExtractRelativeToDirectoryAsync(destinationDirectoryName, overwriteFiles, progressCallback, progress, cancellationToken);
      progress.EntriesExtracted += 1;
    }

    progressCallback?.Invoke(progress);
  }

  private static async Task ExtractRelativeToDirectoryAsync(
    this ZipArchiveEntry       source,
    string                     destinationDirectoryName,
    bool                       overwrite,
    Action<ZipExtractProgress> progressCallback,
    ZipExtractProgress         progress,
    CancellationToken          cancellationToken = default
  )
  {
    string path1 = destinationDirectoryName != null
      ? Directory.CreateDirectory(destinationDirectoryName).FullName
      : throw new ArgumentNullException(nameof(destinationDirectoryName));
    if (!path1.EndsWith(Path.DirectorySeparatorChar))
      path1 += Path.DirectorySeparatorChar.ToString();
    string fullPath = Path.GetFullPath(Path.Combine(path1, source.FullName));
    if (!fullPath.StartsWith(path1))
      throw new IOException("Extracting Zip entry would have resulted in a file outside the specified destination directory.");
    if (Path.GetFileName(fullPath).Length == 0)
    {
      if (source.Length != 0L)
      {
        throw new IOException("Zip entry name ends in directory separator character but contains data.");
      }

      Directory.CreateDirectory(fullPath);
    }
    else
    {
      Directory.CreateDirectory(Path.GetDirectoryName(fullPath));
      await source.ExtractToFileAsync(fullPath, overwrite, progressCallback, progress, cancellationToken);
    }
  }

  private static async Task ExtractToFileAsync(
    this ZipArchiveEntry       source,
    string                     destinationFileName,
    bool                       overwrite,
    Action<ZipExtractProgress> progressCallback,
    ZipExtractProgress         progress,
    CancellationToken          cancellationToken = default
  )
  {
    var initialBytesExtracted = progress.BytesExtracted;
    const int bufferSize     = 81920;

    if (source == null)
      throw new ArgumentNullException(nameof(source));
    if (destinationFileName == null)
      throw new ArgumentNullException(nameof(destinationFileName));
    FileMode mode = overwrite ? FileMode.Create : FileMode.CreateNew;
    using (Stream destination = new FileStream(destinationFileName, mode, FileAccess.Write, FileShare.None, bufferSize, false))
    {
      using (Stream stream = source.Open())
      {
        if (progressCallback != null)
        {
          await stream.CopyToAsyncWithProgress(
            destination,
            bytesCopied =>
            {
              progress.BytesExtracted = initialBytesExtracted + bytesCopied;
              progressCallback(progress);
            },
            bufferSize,
            cancellationToken
          );
        }
        else
        {
          await stream.CopyToAsync(destination, bufferSize, cancellationToken);
        }
      }
    }

    File.SetLastWriteTime(destinationFileName, source.LastWriteTime.DateTime);
  }
}

public interface IZipExtractProgress
{
  int  EntriesCount     { get; }
  int  EntriesExtracted { get; }
  long BytesTotal       { get; }
  long BytesExtracted   { get; }
  bool IsFinished       => EntriesCount == EntriesExtracted && BytesExtracted == BytesTotal;
}

class ZipExtractProgress : IZipExtractProgress
{
  public int  EntriesCount     { get; set; }
  public int  EntriesExtracted { get; set; }
  public long BytesTotal       { get; set; }
  public long BytesExtracted   { get; set; }
}

carlossanlop · 2023-04-25T23:26:50Z

All the ZipFile and ZipArchive async APIs should be implemented together. For that reason, I closed #1560 in favor of this issue. There was a good discussion there we should use for reference.

There are some problems with the way ZipArchive is implemented that would make it complex to implement the async methods:

BinaryReader/BinaryWriter needs to also implement async APIs, so we need the proposal for those first.
We perform many writes and reads using these binary classes. Ideally, we should first reduce the number of calls for each block, similar to how we implemented System.Formats.Tar: Improve performance of Tar library #74281
Additionally, we know where each metadata field is located in the zip blocks, so we should explore rewriting the code in a way that would let us handle unseekable streams. This is something I had to recently investigate for System.Formats.Tar: Tar: Improve unseekable stream handling #84279
And finally: the zip blocks we use were implemented using structs. These are not ideal for async code. We need to consider changing them to classes, but being careful to not affect perf. I had initially fallen into this trap when working on System.Formats.Tar, but we were able to address it before shipping: Convert tar header struct to class #72472

lvde0 · 2024-04-18T09:24:04Z

This issue is now 8 years old... Are there any plans to target this in the near future? 😕

rmunn · 2025-01-15T16:18:27Z

Just wanted to mention that this is blocking me from implementing a feature the most efficient way. I have an ASP.NET Core service that calls an API multiple times and consolidates its results. The API returns zip files, and my service needs to unzip the files, consolidate them, then produce a single zip of the results. I want to do that in streaming fashion, so that the consumer of the service can start receiving data as soon as the first API call returns a zip.

But because of #1560, I can't send the zipped data to the consumer in streaming fashion; instead, I have to wait until all the API calls are finished (there can be several hundred of them), unzip all their zips to a temporary directory, create a new zip that consolidates them all, and only once that is finished will the consumer receive the first byte of data. That's lag time I could get rid of if I could just stream a ZipArchive to the HTTP response body, but #1560 blocks me from doing that.

synek317 · 2025-01-16T11:09:52Z

@rmunn is there anything stopping you from using my "workaround"?

#1541 (comment)

MaxwellDAssistek · 2025-02-05T17:22:09Z

@synek317 You are missing the implementation of the CopyToAsyncWithProgress method. Maybe you could put together a small nuget package?

carlossanlop · 2025-02-13T23:39:29Z

Update on my previous comment: #1541 (comment)

We merged Remove BinaryReader and BinaryWriter references from ZipArchive #103153 , Reduce memory usage when updating ZipArchives #102704 and Fix Zip64ExtraField handling #111802 in .NET 10 preview 1, which addresses the BinaryReader/BinaryWriter limitations I mentioned by removing calls to those types completely. They also added types to represent the metadata field locations and lengths, which makes the code cleaner, more readable, and will help us avoid duplication when implementing the async APIs.
The only item left to address, which can be addressed as part of the implementation of the async APIs, is to change the block structs to classes.
I've been experimenting with the implementation and it should be generally straightforward: https://github.com/carlossanlop/runtime/tree/ZipAsyncImplementation

The API proposals shared by folks above are mostly complete. I am adding my amended proposal at the top description.

bartonjs · 2025-03-11T18:08:07Z

Video

ZipArchive.GetEntriesAsync and GetEntryAsync can be replaced with a single EnsureLoadedAsync as they only do work once
- Since the new CreateAsync can just do the full reading before exiting, this method isn't needed, either.
Since the ZipArchive constructor does I/O it needs a static CreateAsync method to avoid doing synchronous I/O
ZipArchiveEntry.Delete doesn't actually do I/O, so DeleteAsync is not needed.

- public partial class ZipArchive : IDisposable
+ public partial class ZipArchive : IAsyncDisposable, IDisposable

public partial class ZipArchive
{
    public ValueTask DisposeAsync();
    protected virtual ValueTask DisposeAsyncCore();

    public static Task<ZipArchive> CreateAsync(Stream stream, ZipArchiveMode mode, bool leaveOpen = false, Encoding? entryNameEncoding = null, CancellationToken cancellationToken = default);
}

public partial class ZipArchiveEntry
{
    public Task<Stream> OpenAsync(CancellationToken cancellationToken = default);
}

public static partial class ZipFile
{
    public static Task CreateFromDirectoryAsync(string sourceDirectoryName, Stream destination, CancellationToken cancellationToken = default);

    public static Task CreateFromDirectoryAsync(string sourceDirectoryName, Stream destination, CompressionLevel compressionLevel, bool includeBaseDirectory, CancellationToken cancellationToken = default);

    public static Task CreateFromDirectoryAsync(string sourceDirectoryName, Stream destination, CompressionLevel compressionLevel, bool includeBaseDirectory, Encoding? entryNameEncoding, CancellationToken cancellationToken = default);

    public static Task CreateFromDirectoryAsync(string sourceDirectoryName, string destinationArchiveFileName, CancellationToken cancellationToken = default);

    public static Task CreateFromDirectoryAsync(string sourceDirectoryName, string destinationArchiveFileName, CompressionLevel compressionLevel, bool includeBaseDirectory, CancellationToken cancellationToken = default);

    public static Task CreateFromDirectoryAsync(string sourceDirectoryName, string destinationArchiveFileName, CompressionLevel compressionLevel, bool includeBaseDirectory, Encoding? entryNameEncoding, CancellationToken cancellationToken = default);

    public static Task ExtractToDirectoryAsync(Stream source, string destinationDirectoryName, CancellationToken cancellationToken = default);

    public static Task ExtractToDirectoryAsync(Stream source, string destinationDirectoryName, bool overwriteFiles, CancellationToken cancellationToken = default);

    public static Task ExtractToDirectoryAsync(Stream source, string destinationDirectoryName, Encoding? entryNameEncoding, CancellationToken cancellationToken = default);

    public static Task ExtractToDirectoryAsync(Stream source, string destinationDirectoryName, Encoding? entryNameEncoding, bool overwriteFiles, CancellationToken cancellationToken = default);

    public static Task ExtractToDirectoryAsync(string sourceArchiveFileName, string destinationDirectoryName, CancellationToken cancellationToken = default);

    public static Task ExtractToDirectoryAsync(string sourceArchiveFileName, string destinationDirectoryName, bool overwriteFiles, CancellationToken cancellationToken = default);

    public static Task ExtractToDirectoryAsync(string sourceArchiveFileName, string destinationDirectoryName, Encoding? entryNameEncoding, CancellationToken cancellationToken = default);

    public static Task ExtractToDirectoryAsync(string sourceArchiveFileName, string destinationDirectoryName, Encoding? entryNameEncoding, bool overwriteFiles, CancellationToken cancellationToken = default);

    public static Task<ZipArchive> OpenAsync(string archiveFileName, ZipArchiveMode mode, CancellationToken cancellationToken = default);

    public static Task<ZipArchive> OpenAsync(string archiveFileName, ZipArchiveMode mode, Encoding? entryNameEncoding, CancellationToken cancellationToken = default);

    public static Task<ZipArchive> OpenReadAsync(string archiveFileName, CancellationToken cancellationToken = default);
}
  
public static partial class ZipFileExtensions
{
    public static Task<ZipArchiveEntry> CreateEntryFromFileAsync(this ZipArchive destination, string sourceFileName, string entryName, CancellationToken cancellationToken = default);

    public static Task<ZipArchiveEntry> CreateEntryFromFileAsync(this ZipArchive destination, string sourceFileName, string entryName, CompressionLevel compressionLevel, CancellationToken cancellationToken = default);

    public static Task ExtractToDirectoryAsync(this ZipArchive source, string destinationDirectoryName, CancellationToken cancellationToken = default);

    public static Task ExtractToDirectoryAsync(this ZipArchive source, string destinationDirectoryName, bool overwriteFiles, CancellationToken cancellationToken = default);

    public static Task ExtractToFileAsync(this ZipArchiveEntry source, string destinationFileName, CancellationToken cancellationToken = default);

    public static Task ExtractToFileAsync(this ZipArchiveEntry source, string destinationFileName, bool overwrite, CancellationToken cancellationToken = default);
}

stephentoub assigned ianhays Jan 26, 2016

ianhays removed their assignment Oct 10, 2016

ViktorHofer assigned ahsonkhan and ViktorHofer Dec 7, 2018

ahsonkhan removed their assignment Sep 11, 2019

twsouthwick mentioned this issue Jan 6, 2022

Spreadsheet.Open crash in Kestrel web server (.NET 6) since synch reads are not allowed dotnet/Open-XML-SDK#1109

Closed

Symbai mentioned this issue Feb 28, 2022

[API Proposal]: APIs to support tar archives #65951

Closed

jeffhandley modified the milestones: 7.0.0, Future Jul 7, 2022

madelson mentioned this issue Dec 13, 2022

[API Proposal]: make ZipArchive implement IAsyncDisposable #79586

Closed

adamsitnik removed their assignment Dec 19, 2022

carlossanlop mentioned this issue Apr 25, 2023

Add async dispose support to ZipArchive #1560

Closed

scott-moxham mentioned this issue Jun 28, 2023

Intermittent error "Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead." StephenClearyExamples/AsyncDynamicZip#6

Open

campersau mentioned this issue Jan 22, 2025

[browser] [wasm] Make response streaming opt-out #111680

Merged

carlossanlop self-assigned this Feb 13, 2025

carlossanlop added api-suggestion api-ready-for-review and removed api-needs-work api-suggestion labels Feb 13, 2025

carlossanlop modified the milestones: Future, 10.0.0 Mar 5, 2025

bartonjs added api-approved and removed api-ready-for-review labels Mar 11, 2025

carlossanlop mentioned this issue Mar 12, 2025

Change ZipBlocks structs to classes #113453

Merged

Add async ZipFile APIs #1541

Add async ZipFile APIs #1541

Comments

ericstj commented Jan 25, 2016 • edited by carlossanlop Loading

System.IO.Compression

Usage examples:

System.IO.Compression.ZipFile

Usage examples

Plan

danmoseley commented Dec 27, 2016

abdulbeard commented Jun 19, 2018 • edited Loading

ianhays commented Jun 19, 2018

abdulbeard commented Jun 19, 2018

abdulbeard commented Jun 19, 2018

Summary

Proposed API

Expected Use and Benefits

abdulbeard commented Jun 19, 2018

svick commented Jun 19, 2018

abdulbeard commented Jun 27, 2018 • edited Loading

Here's a proposal for ZipArchive class:

svick commented Jun 27, 2018

abdulbeard commented Jun 27, 2018

danmoseley commented Jun 27, 2018

svick commented Jun 28, 2018

abdulbeard commented Jul 2, 2018

abdulbeard commented Jul 23, 2018

abdulbeard commented Sep 5, 2018

abdulbeard commented Dec 7, 2018

JeremyKuhne commented Dec 7, 2018

JeremyKuhne commented Dec 7, 2018

stephentoub commented Dec 7, 2018 • edited Loading

stephentoub commented Dec 7, 2018

TianqiZhang commented Mar 22, 2019

kccsf commented Apr 9, 2019

m-valenta commented May 28, 2019

ahsonkhan commented Sep 11, 2019

juunas11 commented Oct 23, 2019

gragra33 commented Aug 27, 2022

Casper-Olsen commented Aug 27, 2022

gragra33 commented Aug 27, 2022

synek317 commented Oct 3, 2022

carlossanlop commented Apr 25, 2023

lvde0 commented Apr 18, 2024

rmunn commented Jan 15, 2025

synek317 commented Jan 16, 2025

MaxwellDAssistek commented Feb 5, 2025

carlossanlop commented Feb 13, 2025 • edited Loading

bartonjs commented Mar 11, 2025 • edited by dotnet-api-review bot Loading

ericstj commented Jan 25, 2016 •

edited by carlossanlop

Loading

abdulbeard commented Jun 19, 2018 •

edited

Loading

abdulbeard commented Jun 27, 2018 •

edited

Loading

stephentoub commented Dec 7, 2018 •

edited

Loading

carlossanlop commented Feb 13, 2025 •

edited

Loading

bartonjs commented Mar 11, 2025 •

edited by dotnet-api-review bot

Loading