When compact compression is not enough - deduplication with Wimlib + Wimboot!

[theChaosCoder]

Could be a feature in compactGUI2 😄

I found an additional way in storing/compressing files. It is similar to what compact.exe does but "the old way". I think with Windows 8.1 MS introduced Wimboot to save space on devices with 16/32gb hdd space an we are gonna use that technique to save some space!

A Wim Image does not only compress files but also uses deduplication to avoid saving the same file multiple times, it can store multiple snapshot inside one wim image (or make a diff wim file) and calculate hashes for each file. (with --check)

So I created a WIM image of my Adobe folder like this:
.\wimlib-imagex.exe capture 'C:\Program Files\Adobe' Adobe.wim --chunk-size=32K --compress=LZX:80 --snapshot --check
(You can use dism.exe for that, but I like the open source wimlib more https://wimlib.net/)

it's similar like a zip file. Adobe.wim contains all files from the Adobe folder + metadata etc.
Wimlib has more compression options but because we are using wimboot we are limited to what MS supports.

The compression formats supported by wof.sys are: XPRESS 4K, XPRESS 8K, XPRESS 16K, XPRESS 32K, and LZX 32K.

With Wimboot you can create so called sparse files or hard-links to a wim file. These files are actually 0kb in size but looks like regular files for explorer and all other apps. (Use WizTree to see the allocated size on your disk)

Now I copied the Adobe.wim to my Program Files Folder and created the links to the wim file.
.\wimlib-imagex.exe apply --wimboot 'C:\Program Files\Adobe32K100.wim' 1 "C:\Programme\Adobe"

This is the result:
"Adobe_" is compressed with compactGUI Xpress16K (~4,6GB ~54% saved VS. ~3,16GB (LZX 32K) ~65%, (Xpress 32K = 3,39GB) saved with wim/wimboot )

I tried this also with a Games (Cuphead) but there was no benefit in using a wim file. Maybe 1-2% better compression.

/!\ If you edit a linked file it becomes a regular file which is not linked anymore. So you would need to re-create you wim after an Adobe update.

You can mount a wim file with dism or just open it with 7-zip.
Oh, and the creating of the wim file was much much faster than compressing it with compact.exe even at max compression 100.

[xalender]

This is another great idea of compressing files, i couldn't find any information on it, but i was wondering, can WIM files be repaired? Something along the lines of how RAR adds a recovery record to allow repairing of (a percentage of) the file?

This could be helpful for preservation of the files in question in case you don't have the originals.

[theChaosCoder]

The wim format does not specify any recovery options. But you could use something like par2, which is btw more robust than the winrar recovery + can be used with any files without compressing them first.

https://multipar.org/
https://github.com/Yutaka-Sawada/MultiPar

[xalender]

Excellent, I've never heard of this par files before, thank you very much, program works great.

[xalender]

Ok, i've been using this kind of compression and it works great, reduced Cuphead file size from 10.9 GB to 2.2 GB! Almost all my games are compressed this way now, quick question though, whenever there's a file bigger than 4 GB inside a WIM, it cannot be linked, says it needs to be extracted because of wof.sys limitations.

Is there a way to bypass this? There are a lot of games which have bigger than 4 GB files (RE2 Remake, DragonBall FighterZ, My Hero One's Justice, Watch Dogs, Shenmue III, among others).

[theChaosCoder]

I never tried it with very large files. But I doubt what the games will have a 4gb duplicate file 😄
So the best way would be to exclude files bigger than 4gb while creating your WIM.

Is it really worth the effort? You'll need re-create the WIM after a game update every time theoretically.

Alternative http://zipmagic.co/drive-press.html
http://zipmagic.co/doublespace.html