[UNLINKER] Add Support to Fastfile for PS3 and Xbox 360

[Darktreivor]

Hello, how are you? Thank you for the project!
I would like to translate the games in the franchise for PS3 and Xbox 360 into Brazilian Portuguese, since only BO2 and later games originally came with a translation. Would it be possible to add support for PS3 and Xbox 360 fastfile, or guide me on how I can extract the ff files from these consoles to access the .str files for this purpose?
I'll leave a sample of the code_post where the subtitles are supposedly located.

code_post_gfx.zip

[JezuzLizard]

Adding support for console versions is not trivial as OAT is not currently endian aware.
Also console structs may be different than PC structs which means there will need to be zone code generation per game, per platform.

[Darktreivor]

Adding support for console versions is not trivial as OAT is not currently endian aware. Also console structs may be different than PC structs which means there will need to be zone code generation per game, per platform.

I understand, but would it be possible to help me understand the file based on your experience with PC so that I can at least create an extractor and inserter for the .str?
My focus isn't on map or graphics mods, but solely on translating the game into other languages that lack support.

[diamante0018]

No matter how small the change you want to make, everything Jezuz said needs to happen. You can't just say, let me insert a string into a fastfile because OAT does not even know how to open the file. A lot of work needs to be done to make OAT endian aware and then at the end, treat the console counter parts for each PC game like a completely separate game because the assets changed. Not to mention that to test anything to even confirm OAT works you need to rely of incomplete emulation tools or a modded console.

[Laupetin]

Hey, thanks for the suggestion. I guess there is a bunch of reasons to support console fastfiles, it would be nice to be able to dump for example beta builds and also release builds.

The issue I see with what you are saying though is the following:
Idk how console fastfiles work in detail, but for PC the MP fastfiles are cryptographically signed which means that the game will refuse to load any fastfile that is not signed by Infinity Ward.
This is why only custom clients for MP can load custom fastfiles.
For SP this is not a requirement, so even Steam SP can load custom fastfiles.

Idk if that also applies to consoles so whether they are also signed. If they are though, it will not be possible to load custom fastfiles without a modified game.

Other than that supporting console fastfiles requires a lot of work to be done as the others mentioned above.

• It requires a solution for supporting big endian fastfiles and assets.
• It requires a solution for having multiple load/write codes with different slightly different assets for the same game to be able to reuse code

As it stands currently it doesn't have a very high priority i'm afraid since it requires rewriting large chunks of the zone code generator and other load/write components.
And there are still a lot of lower hanging fruits to solve 😅

[Darktreivor]

Hey, thanks for the suggestion. I guess there is a bunch of reasons to support console fastfiles, it would be nice to be able to dump for example beta builds and also release builds.

The issue I see with what you are saying though is the following: Idk how console fastfiles work in detail, but for PC the MP fastfiles are cryptographically signed which means that the game will refuse to load any fastfile that is not signed by Infinity Ward. This is why only custom clients for MP can load custom fastfiles. For SP this is not a requirement, so even Steam SP can load custom fastfiles.

Idk if that also applies to consoles so whether they are also signed. If they are though, it will not be possible to load custom fastfiles without a modified game.

Other than that supporting console fastfiles requires a lot of work to be done as the others mentioned above.

• It requires a solution for supporting big endian fastfiles and assets.
• It requires a solution for having multiple load/write codes with different slightly different assets for the same game to be able to reuse code

As it stands currently it doesn't have a very high priority i'm afraid since it requires rewriting large chunks of the zone code generator and other load/write components. And there are still a lot of lower hanging fruits to solve 😅

Thank you for the clarifications! Yes, the MP has this verification hash, which was exploited for consoles years ago, but I'm not sure if it's valid in the latest versions of the game. What I found out about the console's FF is that it's in zlib format, and I managed to decompress the various chunks of the .ff file. However, my knowledge stops there because what I'd like to understand at this point is how the .zone file is unpacked, as it’s possible to view plain text via hex for strings and gcs/scripts. Understanding the extraction logic behind the .zone would likely make it easier to insert modified data for testing.

[michaeloliverx]

I've been researching tools regarding console FF's lately. I'd love to see some console support in OAT!

Some info might be of use to someone:

IW3

PS3

Both the SP and MP FFs are unsigned.

Python code to extract zone from FF

import io
import zlib

ZLIB_HEADER = b"\x78\xda"
CHUNK_SIZE = 0x10000
FF_HEADER = b"IWffu100\x00\x00\x00\x01"


def decompress_ff(ff: bytes):
    reader = io.BytesIO(ff)
    writer = io.BytesIO()

    # Skip the header
    reader.seek(len(FF_HEADER))

    while True:
        compressed_data_size = int.from_bytes(reader.read(2), byteorder="big")
        if compressed_data_size == 1:  # End of the file
            break
        compressed_data = reader.read(compressed_data_size)
        decompressed_data = zlib.decompress(ZLIB_HEADER + compressed_data)
        writer.write(decompressed_data)
    return writer.getvalue()

Xbox 360

Build	SP	MP
253	Unsigned	Unsigned
270	Unsigned	Unsigned
290	Unsigned	Unsigned
328	Unsigned	Signed
Retail	Unsigned	Signed

Build 328 and Retail MP FFs are signed but the executable can be patched to ignore the signature.

Python code to extract zone from FF

import io
import zlib

CHUNK_SIZE = 0x200000
AUTH_DATA_SIZE = 0x2000
FF_HEADER = b"IWffu100\x00\x00\x00\x01"
COMPRESSED_ZONE_DATA_START = len(FF_HEADER) + 0x4000


def decompress_ff(ff: bytes):
    writer = io.BytesIO(ff)
    writer.seek(COMPRESSED_ZONE_DATA_START)

    zone_data = bytearray()

    while True:
        block = writer.read(CHUNK_SIZE)
        if not block:
            break
        zone_data.extend(block)
        writer.seek(writer.tell() + AUTH_DATA_SIZE)

    return zlib.decompress(zone_data)

---

I'd love to learn more about zone files and how they work. Could OAT add some technical docs or a markdown file explaining how zone files and OAT work? It’d be a helpful resource for newcomers, giving them a quick overview of the architecture and internals without having to dive straight into the code.

[Laupetin]

A technical documentation of both, how fastfiles are structured/how they work, and the design of OAT would be helpful, you are right.
I should definitely work on that at some point, when I have the time and energy 😅

While the exact details on how fastfiles work of course depend on the game, the rough concept is that are more or less a memory dump of the asset structs of the game with pointers being replaced by offsets to the beginning of the memory they will be loaded in.
That memory dump is compressed in multiple chunks (starting with T6 also encrypted).

So to be able to parse the data correctly you have to know the structs of the game assets and how they generally refer to another. Like for example the struct for an image (GfxImage) being X bytes big, having a string (its name) in offset 0 and a pointer to an array of texture definitions at offset Y with the amount of textures being defined in an uint16 at offset Z.
There is a bit more to it in practice but that's the rough idea the format follows.

After a preamble of headers, the root of that logic is a struct "XAssetList" that contains a list of all assets inside the fastfile.
After loading a struct, one pointer in that struct is resolved after another.
When loading a file the pointer is like I mentioned either an offset to already loaded data, or -1 (0xFFFFFFFF) to indicate, that the data follows.
From there on the rest of the fastfile data is being read as a depth-first search resolving the pointers.

Since the loading code is heavily based on the game structs, the original devs very obviously generated the code which is what I also did with OAT.
There is a component called the "ZoneCodeGenerator" which parses the game structs in form of a C header file and a "commands" file in a custom format specifying the rules that the structs follow.

It outputs code for loading and writing the respective assets from and to fastfiles.
The generated code is not in the repository but you can see if it you compile and take a look in the folder src/ZoneCode/Game/IW3/XAssets/gfximage for example.

So all in all it's unfortunately not a very easy format to parse, while I must definitely overengineered a bit since this started as a fun project I experimented on, trying to parse fastfiles does come with a good amount of complexity.

Since you cannot skip forward in the file really, it is also not very easy to make something reliable that only implements a subset of the assets of the game since there are no markers to where an asset starts and where it ends (some other tools can dump rawfiles and stringtables though by searching for recognizable ASCII sequences).

I hope that gives a rough idea of how these files work and why big endian and code duplication are challenges that need tackling to be able to implement console files as well 😅

[JezuzLizard]

Hey, thanks for the suggestion. I guess there is a bunch of reasons to support console fastfiles, it would be nice to be able to dump for example beta builds and also release builds.

The issue I see with what you are saying though is the following: Idk how console fastfiles work in detail, but for PC the MP fastfiles are cryptographically signed which means that the game will refuse to load any fastfile that is not signed by Infinity Ward. This is why only custom clients for MP can load custom fastfiles. For SP this is not a requirement, so even Steam SP can load custom fastfiles.

Idk if that also applies to consoles so whether they are also signed. If they are though, it will not be possible to load custom fastfiles without a modified game.

Other than that supporting console fastfiles requires a lot of work to be done as the others mentioned above.

• It requires a solution for supporting big endian fastfiles and assets.
• It requires a solution for having multiple load/write codes with different slightly different assets for the same game to be able to reuse code

As it stands currently it doesn't have a very high priority i'm afraid since it requires rewriting large chunks of the zone code generator and other load/write components. And there are still a lot of lower hanging fruits to solve 😅

ALL T6 fastfiles from the earliest known build onwards for all platforms, and binaries are signed.
I believe only dedicated fastfiles are always unsigned. It's likely that local test builds use unsigned fastfiles, but otherwise any distributable build would use signed fastfiles, whether for release, or inhouse QA.

Technically the T6 console (debug only?) builds can load unsigned fastfiles under certain circumstances so it might be possible to use OAT to generate an unsigned fastfile that one of those builds could load. Depending on the build however, it may be required to implement XCompress to handle Xbox 360 fastfiles that used it, instead of zlib.

T6 PC builds cannot load unsigned fastfiles under any circumstances without modifying the binary, even for SP, primarily due to CEG being utilized to protect fastfile authentication.

[JezuzLizard]

There is more reason to look into this now that there are console exclusive maps that could be ported to PC.