Codecs dir

[cielavenir]

Closes #25 #53 #141 #156

This build (1a4e078) should have the same capability as 7-Zip-zstd (except GUI):

• compression and decompression for Brotli, Lizard, LZ4, LZ5 and Zstandard within the 7-Zip container format
• compression and decompression of Lizard (.liz), LZ4 (.lz4), LZ5 (.lz5) and Zstandard (.zst) files
• handling of ZIP files with Zstandard compression
• included lzip decompression support, patch from: https://download.savannah.gnu.org/releases/lzip/7zip/
• squashfs files with LZ4 or Zstandard compression can be handled
• these hashes can be calculated: CRC32, CRC64, MD2, MD4, MD5, SHA1, SHA256, SHA384, SHA512, XXH32, XXH64, BLAKE2sp, BLAKE3 (lowercase or uppercase)

Additionally:

• It does support macOS/Linux :)
• Add PKImplode codec (can be disabled via -DDISABLE_PKIMPLODE_COMPRESS)
• Add Lzham codec
• Can build external Rar/PKImplode codecs
• Can build external hashers
• Codecs are now submodules
• Can build mingw edition on Linux
• External Zstd codec work with stock (p)7zip zip (specify -mm=zstd-pk or zstd-wz)

[teoberi]

git clone --recurse-submodules -b CodecsDir https://github.com/cielavenir/p7zip.git
cd p7zip/CPP/7zip/Bundles/Alone && make -f ../../cmpl_gcc_x64.mak

make: *** No rule to make target 'b/g_x64/liblz5.a', needed by 'b/g_x64/7za'. Stop.

[cielavenir]

@teoberi fixed (I tested only Alone2 and Format7zF)

[teoberi]

What will happen if @jinfeihan57 stops working on this project?

[cielavenir]

@trounticonan https://github.com/jinfeihan57/p7zip/blob/7zip_21.02/CPP/7zip/7zip_gcc.mak uses cmake.
however i see master uses each c file. let me integrate it

[teoberi]

master is dead.
https://github.com/jinfeihan57/p7zip/issues/156#issuecomment-891433297

[teoberi]

If this PR is accepted, CMake will become a build dependency, as it is required by some of the additional codecs.

I don't know who should accept this PR, it seems that jinfeihan57 still doesn't answer.

[teoberi]

Amazing work! Thank you!
https://github.com/cielavenir/p7zip/actions

[cielavenir]

ok, so, even additionally:

• Added Balz and Bcm codecs (man, the performance does not look good...)
• Now CI builds linux(x64)/linux(x86)/macos(x64/arm64)/win32/win64
  • 7z.so/7zz are now built with DISABLE_RAR_COMPRESS=1 and DISABLE_PKIMPLODE_COMPRESS=1
• 7zz can access external codecs
• macos AES/CRC/SHA1 are assembled (Lz code is not assembled)

[cielavenir]

• And fixed make -C CPP/7zip/Bundles/Alone2 -f makefile.gcc CXX=aarch64-linux-android-g++ CC=aarch64-linux-android-gcc CFLAGS_ADDITIONAL="-std=c99 -Wno-error -DNOTIMESPEC"

[edit] now CI build android-aarch64 binary automatically

[teoberi]

Although there are different releases (21.07.2, 21.07.3, 21.07.4) the version for 7zz, 7za and 7zr is the same, for example:

7-Zip (z) 21.07 (x64)

It would be very easy to add this in the p7zip/C/7zVersion.h file line 4
#define MY_VERSION_NUMBERS "21.07.4"

This change may cause problems in addition to the benefit of making it easier to identify the release?

[teoberi]

Maybe it could be implemented for the UASM build:
sed -i 's|CC = gcc|CC = gcc -fcommon -fstack-protector-strong -D_FORTIFY_SOURCE=2|g' gccLinux64.mak
and
sed -i 's|-Wl,-Map,$(OUTD)/$(TARGET1).map $(LDFLAGS)|-Wl,-Map,$(OUTD)/$(TARGET1).map -Wl,-z,now $(LDFLAGS)|g' gccLinux64.mak
The source of the changes is openSUSE (with small changes).
The result using checksec:
checksec --file=GccUnixR/uasm

RELRO STACK CANARY NX PIE RPATH RUNPATH Symbols FORTIFY Fortified Fortifiable FILE
Full RELRO Canary found NX enabled No PIE No RPATH No RUNPATH No Symbols Yes 11 20 GccUnixR/uasm

[teoberi]

And for 7zip build:
sed -i 's|CFLAGS = $(MY_ARCH_2) $(LOCAL_FLAGS) $(CFLAGS_BASE2) $(CFLAGS_BASE) -std=c99 $(CC_SHARED) -o $@|CFLAGS = $(MY_ARCH_2) $(LOCAL_FLAGS) $(CFLAGS_BASE2) $(CFLAGS_BASE) -fstack-protector-strong -std=c99 $(CC_SHARED) -o $@|g' CPP/7zip/7zip_gcc.mak

sed -i 's|CXXFLAGS = $(MY_ARCH_2) $(LOCAL_FLAGS) $(CXXFLAGS_BASE2) $(CFLAGS_BASE) $(CXXFLAGS_EXTRA) -std=c++11 $(CC_SHARED) -o $@ $(CXX_WARN_FLAGS)|CXXFLAGS = $(MY_ARCH_2) $(LOCAL_FLAGS) $(CXXFLAGS_BASE2) $(CFLAGS_BASE) $(CXXFLAGS_EXTRA) -fstack-protector-strong -std=c++11 $(CC_SHARED) -o $@ $(CXX_WARN_FLAGS)|g' CPP/7zip/7zip_gcc.mak

sed -i 's|LD_arch += -Wl,--no-undefined -Wl,-z,noexecstack|LD_arch += -Wl,--no-undefined -pie -z now -z noexecstack|g' CPP/7zip/7zip_gcc.mak

The result using checksec:
checksec --file=CPP/7zip/Bundles/Alone2/b/g_x64/7zz

RELRO STACK CANARY NX PIE RPATH RUNPATH Symbols FORTIFY Fortified Fortifiable FILE
Full RELRO Canary found NX enabled PIE enabled No RPATH No RUNPATH No Symbols No 0 14 CPP/7zip/Bundles/Alone/b/g_x64/7zz

[tansy]

Closes #25 #53 #141 #156

How can you say it closes those issues mentioned when it just tweaks existing state, and definitely does not solve those.

Also you would do better if you separated these pull requests into separate topics.
I just stumbled upon someone saying it better than I could:
The benefit of having multiple PRs is to be able to merge them more gradually, as opposed to all-or-nothing. There are multiple topics in this PR, each one with its own points of attention.
I believe we can be more focused if we take care of them one by one.

Also "your" LzHandler is a copy of Diaz'es handler which is no different form p7zip's, with change of:

 REGISTER_ARC_I(
-  "lzip", "lz tlz", "* .tar", 0xC6,
+  "lzip", "lz tlz", "* .tar", 0xC4,

What's the point of it?

And also convension of files to DOS new lines (CRLF), again, what's the point of it?

Like above.

[tansy]

What will happen if @jinfeihan57 stops working on this project?

Seems like he already did.

[cielavenir]

Also "your" LzHandler is a copy of Diaz'es handler

I just followed https://github.com/mcmilk/7-Zip-zstd/blob/master/CPP/7zip/Archive/LzHandler.cpp, so ask mcmilk.

And also convension of files to DOS new lines (CRLF)

Original 7-zip source has CRLF new lines. So 7-zip derivatives (non-p7zip) should follow it.

[tansy]

Original 7-zip source has CRLF new lines. So 7-zip derivatives (non-p7zip) should follow it.

Yeah, right, unix source should have dos line endings...
Even ms visual studio uses LF only, unix new lines.
And the last one - it is p7zip, posix, *nix port of 7zip. If 7zip author wanted *nix port he would created it. He never did. Until 2021 never did *nix port nor anything like that. So, no. The original, windows source is not a codex, it's just a guidance and under *nix we can follow unix phylosophy and make it simple and good for us.

I just followed ..., so ask mcmilk.

Original Diaz'es source has 0xC6, just like p7zip today, so ask him.

[jinfeihan57]

Thank you very much for your work, but there is so much code to review that my computer can't display them all. So I did the update of p7zip21.07 myself. Please follow this branch. If you have any ideas you can submit the code on the new branch. I will close this merge request.

[cielavenir]

@jinfeihan57 I wanted to start from clean state so that git 3-way merge works well. So I asked for https://github.com/jinfeihan57/p7zip/issues/179#issuecomment-1255673136 , but it is not done yet. It seems I cannot follow anything now.

One of my original goals was to provide Codecs/*.so including Balz and Bcm and they were resolved. It is also possible to copy them to jinfeihan57/p7zip distribution.