Add Zstd as a ZIP compression method

[ghost]

Please refer to p7zip-project/p7zip#25 (comment) for the detailed description.

Key points:

• Method ID for Zstd (20) has been included in PKWARE's ZIP specification.
• You can use Zstd in ZIP (ZIPX) with WinZip 24, although it seems that WinZip uses Method ID 93 instead.
• ZIP allows random access.

[mcmilk]

I will add both IDs ... what ID should be preferred, when creating an .zip file with zstd by 7-Zip ZS ?

[mcmilk]

I have written an email to PKWARE and WinZip. Maybe I get some response for the two IDs issue ;-)

The code is nearly finished, but I will wait for the response of them.

[fcharlie]

Minizip adopts a compatible solution, that is, it supports magic number 20/93 at the same time, which is used to support ZIP (zstd). Magic number 20 is used when creating compressed files.

https://github.com/nmoinvaz/minizip/blob/432fcc712b8fa5f6cb0dc72475704246f8aabd55/minizip.c#L607

https://github.com/nmoinvaz/minizip/blob/dev/mz_strm_zstd.c

[ghost]

@fcharlie

Many thanks for your patches. When there are more open-source programs supporting Zstd in ZIP, hopefully more users will want to try using Zstd in ZIP.

May I know if you are interested in adding Zstd support in libzip? It is used by various projects such as PHP and KDE Ark.

[fcharlie]

@ipaucek4680 If I have time this weekend, I will give it a try.

[fcharlie]

@mcmilk APPNOTE.TXT Update:

        0 - The file is stored (no compression)
        1 - The file is Shrunk
        2 - The file is Reduced with compression factor 1
        3 - The file is Reduced with compression factor 2
        4 - The file is Reduced with compression factor 3
        5 - The file is Reduced with compression factor 4
        6 - The file is Imploded
        7 - Reserved for Tokenizing compression algorithm
        8 - The file is Deflated
        9 - Enhanced Deflating using Deflate64(tm)
       10 - PKWARE Data Compression Library Imploding (old IBM TERSE)
       11 - Reserved by PKWARE
       12 - File is compressed using BZIP2 algorithm
       13 - Reserved by PKWARE
       14 - LZMA
       15 - Reserved by PKWARE
       16 - IBM z/OS CMPSC Compression
       17 - Reserved by PKWARE
       18 - File is compressed using IBM TERSE (new)
       19 - IBM LZ77 z Architecture 
       20 - deprecated (use method 93 for zstd)
       93 - Zstandard (zstd) Compression 
       94 - MP3 Compression 
       95 - XZ Compression 
       96 - JPEG variant
       97 - WavPack compressed data
       98 - PPMd version I, Rev 1
       99 - AE-x encryption marker (see APPENDIX E)

[ghost]

@fcharlie, Thanks for the update.

I don't think there are users who have Zstd ZIP files with the deprecated ID 20 (by using minizip) and therefore we can consider removing the usage of ID 20.

However, if backward compatibility is desired, how should we name the two IDs now? OldZstd for 20 and Zstd for 93?

[fcharlie]

@ipaucek4680
I have added support for zstd in minizip/libzip/archiver(golang) and my own project baulkarc(golang) bali(golang). At present, except for minizip, others are not using 20. In fact, the ZIP standard changes very quickly It seems that everyone has no time to use the magic number 20, and the magic number 20 is deprecated.

If necessary, I might create a new PR for minizip

[ghost]

By the way, I think PKWARE forgot to add an entry for Zstd in APPNOTE Section 4.4.3.2 (version needed to extract). It should be 6.3 since Zstd is added to the specification in revision 6.3.7.
However, it looks like WinZip just sets the version needed to extract to 20 (i.e. 2.0) for Zstd ZIP(X).

[gvollant]

Supporting both 20 and 93 at decompression is probably the better idea for me

[fcharlie]

@gvollant I have some differences from yours. In a short period of time, the compression method magic number 20 was granted to zstd, and then it was not recommended to use, but 93 was assigned to zstd. This time is too short to be available in many software. Support'zstd(20)' in a timely manner, so, in my opinion, support 20 is no longer necessary, PKWARE has abandoned the difference, and chose to be consistent with WinZip, so we don't need to do too much compatibility.

My PR for archiver, libzip and this project no longer support 20, and I will also initiate a new PR to fix minizip support for zip(zstd)

Thanks.

[mcmilk]

It's in now ;)

[stefano2734]

So it is possible to convert older slow zip-files to zip-93-zstd without changing names?
This can reduce size and more performance for user is possible.

very interesting for administrators and users with many old zip files.
this can reduce costs a lot.
Actual zip software with zstd support like here is here only necessary.

great work and thanks by a little user.