Error reading certain pak (incomplete or truncated stream)

[BlueAmulet]

Game: Conan Exiles (Mod)
UE4 Version: 4.15

File: https://steamcommunity.com/sharedfiles/filedetails/?id=1765361591

Description:
When I try to read modinfo.json from SandstormFix.pak, I receive the following error message:

Traceback (most recent call last):
  File "modconflict.py", line 46, in <module>
    record.sendfile(fp, stream)
  File "u4pak.py", line 463, in sendfile
    block_decompress = zlib.decompress(block_content)
zlib.error: Error -5 while decompressing data: incomplete or truncated stream

Minor debugging into the issue shows that it attempts to seek past the end of the file, reads an empty byte array, and then crashes.

I've noticed that umodel also has issues opening this pak file, running into memory allocation errors.

[panzi]

Hm, if another program also has problems with this maybe the file is corrupted? Do you know that the file is not corrupted?

[BlueAmulet]

I've been informed by gildor2 that conan has a nonstandard pak format, I'll just modify my copy of u4pak locally for conan support.

[BlueAmulet]

The problem here actually seems to be that the pak files uses absolute offsets, which gildor's explanation didn't quite make sense to me given that I was able to extract some files successfully, only the ones at the beginning of the pak since their base offset was 0.

The QuickBMS script seems to detect this by seeing if any of the compressed block offsets are less than the file offset, and interpreting them as relative. I don't know if this is part of the conan modifications, but the only modification I read about was extra data before the file entry's filename, which doesn't seem to be happening here.

[panzi]

Does it work with the newest version? Or does it work with this new program (download compiled binary) that I've wrote that doesn't need Python, is faster, and has features that might help Windows users that don't want to use a terminal? :D

[panzi]

Also what version is the pak file? I have seen a pak of version 7 now that has that change of offsets (relative vs absolute). So I know version 1, 2, 3, 4 have absolute offsets, version 7 has compression block offsets relative to the file records. That's probably why some files work to extract and some not: The ones that work are not compressed in the pak.

[BlueAmulet]

All of the mod pak's are giving me this now with the new version of u4pak.py

Traceback (most recent call last):
  File "u4pak.py", line 1872, in <module>
    main(sys.argv[1:])
  File "u4pak.py", line 1794, in main
    pak = read_index(stream, args.check_integrity, args.ignore_magic, args.encoding, args.force_version, args.ignore_null_checksums)
  File "u4pak.py", line 917, in read_index
    record   = read_record(stream, filename)
  File "u4pak.py", line 631, in read_record_v3
    blocks_bin = st_unpack('<%dQ' % (block_count * 2), stream.read(16 * block_count))
MemoryError

The rust version gives me memory allocation of # bytes failed

The older u4pak.py code as of when this issue was initially created worked fine for the conan mod pak's if the handling of offsets was changed to absolute offsets. The error suggests that it's a V3 pak

[panzi]

Hmmm, what happens if you try it with --force-version=7?

[BlueAmulet]

Nothing has changed, same errors. I did link the workshop file in the first issue comment if you would like to download and test the pak yourself.

[panzi]

Ah yes! Took a look at that and made a new release: https://github.com/panzi/rust-u4pak/releases/tag/v1.1.0
Does that work for you?