Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem writing a large file to ZipArchive #101238

Closed
Bo0m21 opened this issue Apr 18, 2024 · 9 comments
Closed

Problem writing a large file to ZipArchive #101238

Bo0m21 opened this issue Apr 18, 2024 · 9 comments

Comments

@Bo0m21
Copy link

Bo0m21 commented Apr 18, 2024

Description

The problem occurs when writing a large file to an archive.

Reproduction Steps

  1. Run this code
  2. Have error
  3. Fine
using (FileStream zipToOpen = new FileStream("release.zip", FileMode.OpenOrCreate))
{
    using (ZipArchive archive = new ZipArchive(zipToOpen, ZipArchiveMode.Update))
    {
        ZipArchiveEntry readmeEntry = archive.CreateEntry("Readme.txt");
        using (StreamWriter writer = new StreamWriter(readmeEntry.Open()))
        {
            for (int i = 0; i < 250000000; i++)
            {
                writer.WriteLine("Information about this package.");
                writer.WriteLine("========================");
            }
        }
    }
}

Expected behavior

The code will run without errors

Actual behavior

The code worked with an error
System.IO.IOException: 'Stream was too long.'

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Apr 18, 2024
@dotnet-policy-service dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label Apr 18, 2024
@PaulusParssinen
Copy link
Contributor

What error?

@Bo0m21
Copy link
Author

Bo0m21 commented Apr 18, 2024

What error?

Updated issue.

@PaulusParssinen
Copy link
Contributor

PaulusParssinen commented Apr 18, 2024

What error?

Updated issue.

Thanks!

It seems like you're hitting StreamWriter's internal buffer limit by trying to write over 2GB in one go. Depending on your you may want to periodically issue StreamWriter.Flush or chunk the data accordingly. You can find similar solutions to this simply by googling the error.

edit: it's #1544 once again

@vcsjones vcsjones added area-System.IO.Compression and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Apr 18, 2024
Copy link
Contributor

Tagging subscribers to this area: @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.

@Bo0m21
Copy link
Author

Bo0m21 commented Apr 18, 2024

What error?

Updated issue.

Thanks! It seems like you're hitting StreamWriter's internal buffer limit by trying to write over 2GB in one go. Depending on your you may want to periodically issue StreamWriter.Flush or chunk the data accordingly. You can find similar solutions to this simply by googling the error.

Before posting the error here, I researched other similar problems and, of course, used Google.
Could you please provide working code?

@PaulusParssinen
Copy link
Contributor

PaulusParssinen commented Apr 18, 2024

...

Before posting the error here, I researched other similar problems and, of course, used Google. Could you please provide working code?

edit: nvm. this is #1544

@Bo0m21
Copy link
Author

Bo0m21 commented Apr 18, 2024

...
Before posting the error here, I researched other similar problems and, of course, used Google. Could you please provide working code?

This entirely depends on your actual use-case/shape of your data/performance needs but my first reaction would be do something along the lines:

using (FileStream zipToOpen = new FileStream("release.zip", FileMode.OpenOrCreate))
{
    using (ZipArchive archive = new ZipArchive(zipToOpen, ZipArchiveMode.Update))
    {
        ZipArchiveEntry readmeEntry = archive.CreateEntry("Readme.txt");
        using (StreamWriter writer = new StreamWriter(readmeEntry.Open()))
        {
            for (int i = 0; i < 250000000; i++)
            {
                writer.WriteLine("Information about this package.");
                writer.WriteLine("========================");
                if (i % 100_000 /* or something */ == 0) writer.Flush();
            }
        }
    }
}

Could you please provide working code?

@PaulusParssinen
Copy link
Contributor

...

Could you please provide working code?

So, this seems to essentially be blocked by #1544.

IF you can rebuild the existing archive, you can avoid the in-memory MemoryStream buffer caused by readMeEntry.Open() by overwriting existing archive in write-only manner. Otherwise, you're blocked by #1544

using (FileStream zipToOpen = new FileStream("release.zip", FileMode.OpenOrCreate, FileAccess.Write))
{
    using (ZipArchive archive = new ZipArchive(zipToOpen, ZipArchiveMode.Create))
    {
        ZipArchiveEntry readmeEntry = archive.CreateEntry("Readme.txt");
        using (StreamWriter writer = new StreamWriter(readmeEntry.Open()))
        {
            for (int i = 0; i < 250000000; i++)
            {
                writer.WriteLine("Information about this package.");
                writer.WriteLine("========================");
            }
        }
    }
}

@ericstj
Copy link
Member

ericstj commented Jul 24, 2024

Duplicate of #1544

@ericstj ericstj marked this as a duplicate of #1544 Jul 24, 2024
@ericstj ericstj closed this as completed Jul 24, 2024
@dotnet-policy-service dotnet-policy-service bot removed the untriaged New issue has not been triaged by the area owner label Jul 24, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Aug 24, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants