iOS 11 LZFSE compression

[horrorho]

iOS 11 introduced lzfse compression which is not yet supported by InflatableDonkey. Database files (sqlite) are particularly affected.

As a workaround you'll need to manually decompress files using the above tool. The header magic string is bvx2.

I hope to have this issue patched over the next week. Please see #66 for additional details.

[ghost]

Thank you for updating the code. Last night I had made a backup for the second IOS 11 beta that came out and everything is still working.

Just in case anyone needs to use the tool to access sqlite files and are having issues, here is what I did to get it working:

(Note: currently only works for macOS)

Step 1. Download lzfse

Step 2. cd into lzfse-master and issue this command: xcodebuild install DSTROOT=/tmp/lzfse.dst

Step 3. Issue this cd command: cd /tmp/lzfse.dst/usr/local/bin/

Step 4. The binary that needs to be used will be in that bin directory.
Usage: ./lzfse -encode|-decode [-i input_file] [-o output_file] [-h] [-v]

Just by executing this command below, I was able to read my notes.sqlite:

./lzfse -decode -i /Users/Mathew/Desktop/notes.sqlite -o /Users/Mathew/Desktop/decoded.sqlite

[ghost]

Update: After extended testing with backups, I can confirm that ALL sqlite databases are encoded in lzfse which includes sqlite files from third party apps.

[horrorho]

Windows users: I've forked lzfse and uploaded a binary executable here. Please unzip before use. I only just about managed to refrain myself from compressing it with lzfse itself to bug you all...

If you would like to build it yourself and I encourage you to do so, then grab Visual Studio Community if you need a compiler and the lzfse source.

Then either:

• use cmake using the guide here and load into Visual Studio and build (release mode).
• create an empty project, import all of the .c and .h files from the src folder and build (release mode).

Usage:

lzfse -decode -i my.sqlite -o my.decoded.sqlite

[horrorho]

I've pushed a new patch that will decompress lzfse files but it requires an external lzfse binary (as above).

Either place the lzfse binary in the command path/ update path and use --lzfse:

java -jar InflatableDonkey.jar elvis@lives.com uhhurhur --lzfse

Or let InflatableDonkey know where you've placed lzfse with --lzfse path:

java -jar InflatableDonkey.jar elvis@lives.com uhhurhur --lzfse /this/is/where/i/put/lzfse

InflatableDonkey will look for and test the lzfse compressor. If all is well you should get the output line:

LZFSE external compressor configured.

I have written a decompressor in Java but it needs testing and it doesn't handle all of the compression types at present. I'm away with work from mid-week so it may get delayed significantly.

[michaljirman]

Yaldo425 commented 3 days ago
Update: After extended testing with backups, I can confirm that ALL sqlite databases are encoded in lzfse which includes sqlite files from third party apps.

Based on my tests not all sqlite databases are compressed (e.g. consolidated.db).

[ghost]

@jirmi Thanks for pointing that out, I did not get to test every database file but there were many that were encoded. Not sure what consolidated.db is for but maybe we could say that at least all the important database files are encoded.

[tmyroadctfig]

IIRC LZFSE is quite a bit more than just LZVN, but I've ported over that part to Java already here: https://github.com/tmyroadctfig/jnode/blob/master/fs/src/fs/org/jnode/fs/hfsplus/compression/LzvnDecode.java

It might be helpful if you want to move the LZFSE decoding into Java.

[horrorho]

@tmyroadctfig Thank you! I have a half finished LZFSE decompressor that's untested. I was secretly hoping that someone who knows how to program properly would have created their own Java port and I wouldn't need to bother with it. I'll certainly take a look at your LZVN code.

LZFSE source:

#define LZFSE_NO_BLOCK_MAGIC             0x00000000 // 0    (invalid)
#define LZFSE_ENDOFSTREAM_BLOCK_MAGIC    0x24787662 // bvx$ (end of stream)
#define LZFSE_UNCOMPRESSED_BLOCK_MAGIC   0x2d787662 // bvx- (raw data)
#define LZFSE_COMPRESSEDV1_BLOCK_MAGIC   0x31787662 // bvx1 (lzfse compressed, uncompressed tables)
#define LZFSE_COMPRESSEDV2_BLOCK_MAGIC   0x32787662 // bvx2 (lzfse compressed, compressed tables)
#define LZFSE_COMPRESSEDLZVN_BLOCK_MAGIC 0x6e787662 // bvxn (lzvn compressed)

Basically:

• uncompressed blocks
• LZFSE blocks (+- compressed tables)
• LZVN blocks.

In short, block types are selected on the input size. LZFSE is unsuitable for very small input sizes where LZVN or even uncompressed blocks may be emitted.

I have a LZFSE decompressor coded but not a LZVN decompressor. I'll try and spare the time to either write the LZVN component or patch your code in.

[horrorho]

Ok. I've ported the LZFSE LZVN decoder over to Java. The Gist is here. It needs testing.

I've made a number of assumptions regarding it's usage as a fallback decompressor so I've stripped out things like state management. Unless I'm mistaken it will be used for small (under 4096 byte) blocks only and we'll know the input and output buffer sizes in advance.

My LZFSE code looks more horrible than I recall, I'll need time to tidy and test it.

🐙

[horrorho]

So! After months of sitting on it, I've finally tidied and pushed my completed but untested Java LZFSE decompressor here. It's fairly simple as far as these things go and I've not really optimised it, we are decompressing a handful of small sqlite files not gigabytes of data.

I'll test it over the weekend and if all is well I'll pull it into InflatableDonkey. It would probably make sense to leave it as a fallback decompressor if no local lzfse binary is available. Apple's reference implementation will certainly be faster and less prone to bugs.

If anyone wants to play with it:

//  in = LZFSE compressed input
// out = decompressed output

    static void decode(Path in, Path out) {
        try (InputStream is = new BufferedInputStream(Files.newInputStream(in));
                OutputStream os = new BufferedOutputStream(Files.newOutputStream(out))) {
            new LZFSEDecoder().decode(is, os);

        } catch (IOException ex) {
            System.out.println("IO error: " + ex.getMessage());
        } catch (LZFSEDecoderException ex) {
            System.out.println("Decoder error: " + ex.getMessage());
        }
    }

🌴

[horrorho]

Heya. Sorry for the delay. I have been testing the LZFSE decompressor I promise!

I've just added unit tests with well formed data and it's passing them all. It's possible we can rely on it as the main compressor and make things simpler for everyone. I need to conduct tests on malformed data and we'll go from there.

☯️

[horrorho]

Update. RagingMoose is performing better than expected. I've rebuilt InflatableDonkey around RagingMoose and the requirement for lzfse binaries no longer exists.

However, the laughable situation of me not owning an iPhone continues and I can't test the new build. It's here. If anyone wants to test it on iOS11 backups it would be wonderful. The files that are LZFSE compressed are typically sqlite / sqlitedb files.

🍃

[marksims]

hi, how can i donate an iphone to you for test

[horrorho]

@marksims Hi! Thank you for the offer! However I must decline, partly because I don't want to feel indebted and largely because it would be an endless time sink. I've spent far too much time as it is, usually late into the night on this project.

[marksims]

hi, beauty, you deserved it. we have a project based on your code.

[ghost]

@horrorho I can have it tested sometime later today and report back here.

[horrorho]

@Yaldo425 Heya. That would be wonderful. Thank you!

[ghost]

@horrorho

Alright, so I got around to testing it. While downloading backups, the terminal outputted something like this for at least 24 sqlite files:

22:34:27.481 [ForkJoinPool-2-worker-0] WARN c.g.h.i.file.FileAssembler - -- write() - no decompressor: HomeDomain Library/SMS/sms.db -> 2

>> HomeDomain Library/Preferences/com.apple.imessage.bag.plist
22:34:26.457 [ForkJoinPool-2-worker-3] WARN  c.g.h.i.file.FileAssembler - -- write() - no  decompressor: HomeDomain Library/com.apple.itunesstored/kvs.sqlitedb -> 2
>> HomeDomain Library/com.apple.itunesstored/kvs.sqlitedb

Additionally, I could not open files that did not output the WARN error, such as this one:

> HomeDomain Library/Accessibility/VOPronunciation/VoiceOverPronunciation.sqlite

If needed, I can provide my iOS 11 account again if needed

[horrorho]

@Yaldo425 Thanks, but I think you're using the main branch and not the new build which is here.

[ghost]

@horrorho Sorry about that. I just tried it again with that branch and so far it looks like it works well. I've tried numerous .db and .sqlite files and all of them were able to be opened without any issues.

[horrorho]

@Yaldo425 Thank you! I just emailed you, which you can ignore now I guess.

I'll leave the new branch up for a couple of days and if there are no reports of problems I'll go ahead and merge it.

[ghost]

@horrorho No problem! I had emailed it to you anyways just in case you might want to test something in the future

[horrorho]

Ok. I've updated the master build. I'll go ahead and close this issue.

Summary: InflatableDonkey is handling LZFSE decompression via RagingMoose.

🍊