Support BCJ method

[bodgit]

Decompression method: 3.3.1.3

The GameCube homebrew "Swiss" is distributed in a 7z that uses BCJ for some of the binaries.

[rusq]

Hey, Matt, firstly, I'd like to thank you for your hard work on this.

Secondly, I'd like to upvote this issue, as MAME archive uses BCJ as well.

$ 7z i
...
Codecs:
 0  ED  3030103 BCJ
...

I think I found the original C implementation in the 7z source code, but I'm sure you've seen it already.

[bodgit]

I think I found the original C implementation in the 7z source code, but I'm sure you've seen it already.

I think there's some additional scaffolding around that code but yes, that's essentially the implementation. It should be easier than BCJ2, I just need to find some time to do it.

[bodgit]

@rusq I've created #83 which seems to work with my limited testing; it works with the small test cases I created and it seems to work fine on the Swiss GameCube homebrew. I tried to test it on the MAME archive you linked but noticed that this is a self-extracting archive which my package currently doesn't support.

I can probably add support for self-extracting archives doing something similar to https://github.com/hillu/go-zipsfx which searches through the file until it finds the magic archive header rather than assume it's at the head of the file.

[rusq]

Thank you @bodgit ! Great work! I'll test it out.

I cheated on that MAME file, in my program I'm using a specific version, so I get the archive offset by running 7z test:

[8:09:54] 0:_experiments/tetris/1 (runner)> 7z t mame0253b_64bit.exe 

7-Zip [64] 17.05 : Copyright (c) 1999-2021 Igor Pavlov : 2017-08-28
p7zip Version 17.05 (locale=utf8,Utf16=on,HugeFiles=on,64 bits,16 CPUs x64)

Scanning the drive for archives:
1 file, 90468973 bytes (87 MiB)

Testing archive: mame0253b_64bit.exe
--       
Path = mame0253b_64bit.exe
Type = 7z
Offset = 205824
Physical Size = 90263149
Headers Size = 21231
Method = Delta LZMA2:26 BCJ
Solid = +
Blocks = 3

Everything is Ok              

Folders: 256
Files: 2243
Size:       621080252
Compressed: 90468973

Which gives the offset, then I use this value to Seek, and open a SectionReader: https://github.com/corkscrewey/soviet-tetris/blob/runner/mame.go#L70-L74. I did think of implementing a search within the file for the 7z signature, but that would be an overkill for such a simple program that just needs to unpack it.

Thanks again!

[bodgit]

I've just created #85 which adds support for self-extracting archives. It scans the first 1 MiB of the file which should be enough to find the signature without scanning the whole file.

Annoyingly on some platforms, the signature can appear more than once (usually twice) and the first occurrence is in the executable that is at the beginning of the file so there's some heuristics to work out which offset is the correct one.

I've tested it on that MAME archive and it works correctly.

[rusq]

That's perfect @bodgit , thank you! I'll aim to try it out on the weekend.

I'm really amazed how you put so much effort to implement the 7z in go, tbh, kudos!

[rusq]

Hey @bodgit, I have tried it with mame, it works (for both: SFX scan and BCJ compressed files). Thanks again!

[bodgit]

Thanks for testing and confirming!