Regarding LZMA1 (LZMA Alone, Method ID 14) in libzip (Follow-up of #195)

To use the LZMA library (XZ Utils) for Method ID 14, I guess we need to refer to libarchive's implementation: archive_read_support_format_zip.c

So, in order to use the "lzma alone" decoder for the zipx lzma
stream, we simply need to shuffle around some fields, prepare a new
lzma alone header, feed it into lzma alone decoder so it will
initialize itself properly, and then we can start feeding normal
zipx lzma stream into the decoder.

That means the following code from zip_algorithm_xz.c is probably not enough (incomplete) and extra work/magic is needed to deal with LZMA1 ZIP files.

        if (ctx->method == ZIP_CM_LZMA)
            ret = lzma_alone_encoder(&ctx->zstr, filters[0].options);

        if (ctx->method == ZIP_CM_LZMA)
            ret = lzma_alone_decoder(&ctx->zstr, UINT64_MAX);

CC: @jinfeihan57

---

EDIT: See also libarchive/libarchive#1417
EDIT 2: See also https://sourceforge.net/p/lzmautils/discussion/708858/thread/663db6473d/#2919

[0-wiz-0]

Thanks for the libarchive link, that is helpful.
Can you please provide a zip archive with the same contents as regress/testfile-stored-dos.zip compressed with lzma (id 14)? I don't even know a tool that supports this.

Test file, using LZMA, created by WinZip 24:
lzma-winzip-24.zip

Test file, using LZMA, created by vanilla p7zip 16.02:
lzma-vanilla-p7zip-1602.zip

CC: @mholt
(mholt/archiver#226)

[jinfeihan57]

@jloqfjgk
I furged it out yesterday. LOL，about lzma header. it different between lzma format and ZIP lzma format.
in lzma1 format it has 13-byte header. but in ZIP lzma format,It's only have 9 byte header.and
the preceding 4 bytes are irrelevant to the algorithm. it's like that:

puls 1 byte Properties, plus 4-byte Dictionary Size. Total 9 bytes .then comes the LZMA Compressed Data.
NO Uncompressed Size(8 bytes).

so we have to rebuild the lzma header in ZIP.

But it not a easy job

1. the lzma_code() is a stream compress API. It only finished when lzma_action action = LZMA_FINISH.
   before is finished we can't change the head.
2. I try to move the head_rebuild_code to function compress_read(***).but It's abstraction layer . it should not contain a specific algorithm.
   Now I try to Separating the LZMA compressed header and compressed data .(Working on)

Anyone have any suggestion？Please help.

One more thing :
I just finished update p7zip17.02, Please try it. Thanks! : )

[jinfeihan57]

@jloqfjgk This(archive_read_support_format_zip.c) is helped. I will check it out.
@0-wiz-0 Are you working on it? if not I want to help.

[0-wiz-0]

Please try 750a78e!

@0-wiz-0

If the compressed LZMA stream contains the optional end-of-stream (EOS) marker (APPNOTE Section 5.8.9 and 4.4.4), will libzip handle it properly?

[0-wiz-0]

Right now, libzip does not have any particular handling for an lzma EOS marker but just passes the compressed stream as provided by liblzma (after handling the header).

What is the specification for the EOS marker? Does liblzma write it itself or does libzip have to add it manually? In which cases should libzip do that? How can libzip recognize if liblzma wrote it or not? What does it look like? The format seems woefully underspecified.

Neither of your two examples has the bit set, so I assume neither comes with such an EOS marker. Do you know a tool that writes one?

[0-wiz-0]

If we're talking about the PK\07\010 marker, that one was already handled correctly before.

I found this comment by the author of XZ Utils, Lasse Collin:

The lzma_alone_encoder() API in liblzma always adds the end marker. Currently there is no public API in liblzma to encode without the end marker. If you don't want the end marker, you need to use LZMA SDK or XZ for Java.

In 2008, he also said:

Note that ZIP allows LZMA with and without "end of stream (EOS)" marker (I usually call it "end of payload marker (EOPM)"). lzma_alone_decoder() expects to find this marker if and only if the uncompressed size is known. So you cannot unconditionally insert the uncompressed size from ZIP headers to the stream going to LZMA_Alone format decoder.

And in the same thread, Igor Pavlov (7-Zip author) said:

There are two types of LZMA streams:

• Stream with end mark. That end mark adds about 6 bytes to compressed size.
• Stream without end mark. You must know exact uncompressed size to decompress such stream.

LZMA EOS marker is additional data sequence in compressed stream. But only lzma decoder can recognize that EOS marker. It must unpack whole stream before marker.

---

Do you know a tool that writes one?

I figured out that we could enable the marker in 7-Zip by specifying the eos switch, which means:
7z a with-eos.zip abac-repeat.txt -mx=1 -meos+ -mm=LZMA
7z a without-eos.zip abac-repeat.txt -mx=1 -mm=LZMA

Test files:

Producer	File	Bit 1 set? (APPNOTE Section 4.4.4)	Extracts in WinZip?	Extracts in 7-Zip?	Extracts in p7zip?
7-Zip 19.00	lzma-with-eos-7z-1900.zip	Yes	Yes	Yes	Yes
Vanilla p7zip 16.02	lzma-with-eos-vanilla-p7zip-1602.zip	No	Yes	No (Headers Error: abac-repeat.txt)	Yes

Explanation

From the above table:

• p7zip is buggy and does not set the required bit, although it accepts both files.
• WinZip is forgiving and does not warn / throw error when extracting the buggy file by p7zip.
• 7-Zip seems to follow the specification and produces a correct file, and it refuses to open the buggy file.

---

It will be a good idea to contact Lasse Collin regarding the use of EOS marker with the API.

To my understanding, liblzma should be responsible for writing the EOS marker, and libzip should set the bit.

Test file produced by @jinfeihan57's p7zip (used version: commit b07be46):
lzma-with-eos-szcnick-p7zip-b07be46.zip

His version seems better than the vanilla p7zip - the output file has Bit 1 set and can be extracted by 7-Zip.