Header checksum is invalid on IOS downloading tar.gz file with UnityWebRequest (2019.4.8f1)

[silviaPixowl]

Steps to reproduce

1.Make a request to the server to download the gziped file
2. Use all possible examples from GZip and Tar Samples to try to decompress/extract contents
3. On unity editor and Android all works fine using the method below

public static void ExtractNestedTar(string gzArchiveName, string destFolder, bool deleteOriginal, Func<string, string> fileVersionExtractor) 
{
    var tmp = Path.Combine(destFolder, "tmp");

    using (Stream inStream = File.OpenRead(gzArchiveName))
    using (TarArchive tarArchive = TarArchive.CreateInputTarArchive(inStream)) {

        tarArchive.ExtractContents(tmp);

        System.Array.ForEach(new DirectoryInfo(tmp).GetDirectories(),
            fl => Debug.Log($"Compressor: TMP file name {fl.Name}"));

        foreach (var f in new DirectoryInfo(tmp).GetFiles("*.tar")) {
            using (var s = f.OpenRead())
            using (var t = TarArchive.CreateInputTarArchive(s)) {
                string fileName = fileVersionExtractor.Invoke(f.Name);
                t.ExtractContents(Path.Combine(destFolder, fileName));
            }
        }
    }
        
    if (!deleteOriginal) return;

    File.Delete(gzArchiveName);
    Directory.Delete(tmp, true);
}

Expected behavior

Be able to extract contents from tar.gz file on IOS

Actual behavior

When trying to parse header on IOS devices the checksum is invalid since checksum obtained from ParseOctal and the one obtained from MakeCheckSum are different.
I'm attaching the tar.gz file I got from unity and from IOS (strangely both have different sizes) I'm thinking that maybe from IOS side the gzipped file gets decompressed and encoded in some way not supported, but it's just a hypothesis.
unityFile.tar.gz
IOSFile.tar.gz

Version of SharpZipLib

v1.2.0

Obtained from (only keep the relevant lines)

• Downloaded from GitHub

[piksel]

As you pointed out, the sizes of the files are different, which means that the "source" file is not the same on iOS. Running the files through ArchiveDiag shows that IOSFile.tar.gz is indeed corrupt, but unitfyFile.tar.gz looks OK.

I think the problem lies outside of SharpZipLib, probaby in the way you are getting those files (you mention UnityWebRequest in the title?)

[silviaPixowl]

Thank you for your answer! I'm using UnityWebRequest to download the files from the server, it seems that on IOS side the gzip files gets decompress natively and encoded, but I'm not sure what type of encoding are they using to be able to decode it. What is weird though is that if I open the IOS file I attached on my Mac it will work fine (decompress it and expose the contents) even on IOS, if I move it to some folder where I can preview it, it can show me all the files contained and also a preview of each.

[piksel]

I found the reason!

$ file IOSFile.tar.gz
IOSFile.tar.gz: gzip compressed data, from Unix, original size modulo 2^32 2505102

$ gunzip -vk IOSFile.tar.gz
IOSFile.tar.gz:   3.4% -- created IOSFile.tar

$ file IOSFile.tar
IOSFile.tar: gzip compressed data, max speed, from Unix, original size modulo 2^32 2504704

$ mv IOSFile.tar IOSFile-inner.tar.gz

$ gunzip -vk IOSFile-inner.tar.gz
IOSFile-inner.tar.gz:    -0.0% -- created IOSFile-inner.tar

$ file IOSFile-inner.tar
IOSFile-inner.tar: POSIX tar archive

$ tar -tvf IOSFile-inner.tar
-rw-r--r-- 0/0          163840 2020-08-26 20:37 100_326_256_png.tar
-rw-r--r-- 0/0           92160 2020-08-26 20:37 78_326_256_png.tar
-rw-r--r-- 0/0          153600 2020-08-26 20:37 502_326_256_png.tar
-rw-r--r-- 0/0          256000 2020-08-26 20:37 74_326_256_png.tar
-rw-r--r-- 0/0          952320 2020-08-26 20:37 756_326_256_png.tar
-rw-r--r-- 0/0          102400 2020-08-26 20:37 542_326_256_png.tar
-rw-r--r-- 0/0          122880 2020-08-26 20:37 922_326_256_png.tar
-rw-r--r-- 0/0          194560 2020-08-26 20:37 99_326_256_png.tar
-rw-r--r-- 0/0          204800 2020-08-26 20:37 265_326_256_png.tar
-rw-r--r-- 0/0          256000 2020-08-26 20:37 45_326_256_png.tar

The IOSFile is not just a Gzipped TAR, it's a Gzipped Gzipped TAR 😁

bsdtar (which is used by macOS) identifies the double gzip header and correctly extracts it, but GNUtar fails wih 'invalid header'.

As for your solution (if you cannot change whatever procedure is producing the archives, as it is compressing the archive twice, which just increases the resulting file size by at least 10 bytes), you should be able to just add another layer of GZipInputStream like this:

    using (var inStream = File.OpenRead(gzArchiveName))
    using (var gzipStreamOuter = new GZipInputStream(inStream))
    using (var gzipStreamInner = new GZipInputStream(gzipStreamOuter))
    using (var tarArchive = TarArchive.CreateInputTarArchive(gzipStreamInner))
    {
        tarArchive.ExtractContents(destFolder);
    }

But this will of course not work for the unityFile.tar.gz as it is just compressed once.

[silviaPixowl]

Thank you very much, I'll try this and let you know :)

[silviaPixowl]

It worked , I'm really grateful to you!