Asobo BigFile Format Specification

Overview

Introduction

Treat this document as if it were a tattered notebook you found while rummaging through my desk drawers; just because it’s formatted well doesn’t mean it’s true. Any information on this page is subject to change. Variable and structure names are completely arbitrary and may not properly represent the purpose of the field, read the comments under each element for a more complete understanding of what it does and what values are legal.

It is strongly recommended that you read this document in its entirety before using it as a reference because the minimal comments and arbitrary names are better understood in the context of the whole specification. I often exclude the full sequence of structs containing a variable for the sake of brevity; this is most common with size and count variables. If you are unsure of where a variable is oncoming from, it is most likely in the struct's associated header member or a description struct, such as Block Description, that is not directly in its ancestry. For this reason, a passing familiarity with the structures specified in this document is required and I would encourage you to read the whole document.

What Is A BigFile File?

The BigFile files used in FUEL are asset archives that contain serialized class objects such as textures, models, animations, sounds, particle systems, etc. Internally these types of files are called BigFiles.

History of the BigFile

A fully specified older version of the BigFile format can be found on the Chum World wiki. A more recent, but still old, version of the BigFile format can be found at zouna-templates-docs/templates/Rat/PC/Asobo DXX v1.06.63.0X.bt.

See also: Engine History and Zouna BigFile Format Timeline

BigFile Binary Template/Pattern

An ImHex pattern for the BigFile format can be found in the ImZouna repository.

BigFile Tool

I have created a tool to work with BigFiles: https://github.com/widberg/dpc.

BigFile Associated File Types

File types related to BigFiles in FUEL and other Asobo games are specified at the end of this document.

BigFile File Versions

This page only specifies BigFile versions v1.220.50.07 to v1.381.67.09 inclusive.

Other versions of the BigFile format are partially specified on the XeNTaX Wiki.

BigFile Platforms

Platform	Extension	Endianness
PC	DPC	Little
UWP	DUA	Little
Mac i386	DMC	Little
Mac PPC	DBM	Big
PlayStation 2	DPS	Little
PlayStation 3	DP3	Big
PSP	DPP	Little
Xbox	DXB	Little
Xbox 360	D36	Big
GameCube	DGC	Big
Wii	DRV	Big
Switch	DNX	Little

Considerations For CRC32 And LZ

All references in this specification to the CRC32 hashing function refer to the Asobo CRC32 implementation. Similarly, all references in this specification to the LZ compression algorithm refer to the Asobo LZ implementation.

What’s Left?

For more information about what work still needs to be done, see the BigFile Needs Work entry.

Uniform Object Locator

A path to an object in a BigFile.

The fuel-brute repository is being used to research possible ways to recover the Uniform Object Locator from the CRC32 hash.

[a-zA-Z0-1_]+|DB:>[a-zA-Z0-1_>\-]+[.][a-zA-Z0-1_\-]+

Shared Structures

Asobo File Format Idioms

Classes

The word class is used to refer to one of the classes defined on the Asobo Classes page.

Padding

All top level structures (PrimaryHeader, Block, Pool Manifest, and Object) are padded with either 0x00 or 0xFF to a size that is a multiple of 2048. Blocks are padded with 0x00 and all other top level structures are padded with 0xFF. The functions to calculate the padded size of a section and the size of the padding are:

std::uint32_t calculatePaddedSize(std::uint32_t unpaddedSize)
{
    return (unpaddedSize + 0x7ff) & 0xfffff800;
}

std::uint32_t calculatePaddingSize(std::uint32_t unpaddedSize)
{
    return calculatePaddedSize(unpaddedSize) - unpaddedSize;
}

Load Procedure

See the BigFile Load Procedure entry for more information about how the game loads BigFiles.

BigFile File

struct BigFile
{
    PrimaryHeader primaryHeader;
// Block Sector
    Block blocks[primaryHeader.blockCount];
// The following fields only appear if primaryHeader.poolManifestPaddedSize is not 0
// The pool is exclusive to BigFiles opened with the OpenBFS command; this is the streaming version of the OpenBF command.
// Pool Sector
    PoolManifest poolManifest;
    PoolObject objects[poolManifest.objectsCRC32s.size];
};

Primary Header

struct PrimaryHeader
{
    char versionString[256];
        // Human readable version information
        // The length of this string MUST match the same format as bellow
        // It is better to use the latest version string than make your own
        // If this is an invalid value, the game will get stuck in a loop
        // Values used in FUEL:
        //   * "v1.381.67.09 - Asobo Studio - Internal Cross Technology"
        //   * "v1.381.66.09 - Asobo Studio - Internal Cross Technology"
        //   * "v1.381.65.09 - Asobo Studio - Internal Cross Technology"
        //   * "v1.381.64.09 - Asobo Studio - Internal Cross Technology"
        //   * "v1.379.60.09 - Asobo Studio - Internal Cross Technology"
        //   * "v1.325.50.07 - Asobo Studio - Internal Cross Technology"
        //   * "v1.220.50.07 - Asobo Studio - Internal Cross Technology"
    std::uint32_t isNotRTC;
        // RTC = real time cinematics
        // 1 if the BigFile is not located in the RTC directory
        // 0 if the BigFile is located in the RTC directory
    std::uint32_t blockCount;
        // Number of blocks in the BigFile
        // This field cannot be 0
        // Empty BigFiles are illegal and when one is encountered an error screen is displayed and the game is hard locked
    std::uint32_t blockWorkingBufferCapacityEven;
        // Used as the capacity of an internal byte vector for BigFileLoader_Z
        // uses raw malloc
        // A heap allocated buffer of size blockWorkingBufferCapacityEven is used to load even indexed blocks to be operated on
        // This is effectively the padded size of the largest even indexed block with the workingBufferOffset added to it
        // The pool might borrow this buffer for processing the pool manifest
        // Must be a multiple of 2048
        // This field cannot be 0
    std::uint32_t blockWorkingBufferCapacityOdd;
        // Same as above but for odd indexed blocks
        // The buffer is created with a wrapper vector class around it
        // 0 when there are no odd indexed blocks
        // can be 0 even with a pool
    std::uint32_t paddedSize;
        // Sum of padded sizes specified in the block descriptions
        // sizeof(PrimaryHeader) + paddedSize will be the starting address of the pool manifest
    std::uint32_t versionPatch;
        // Unused
        // Corresponds to the patch version in the version string
        // versionPatch values and the corresponding versionString values:
        //   * 272
        //      * "v1.381.67.09 - Asobo Studio - Internal Cross Technology"
        //      * "v1.381.66.09 - Asobo Studio - Internal Cross Technology"
        //   * 271
        //      * "v1.381.65.09 - Asobo Studio - Internal Cross Technology"
        //      * "v1.381.64.09 - Asobo Studio - Internal Cross Technology"
        //   * 269
        //      * "v1.379.60.09 - Asobo Studio - Internal Cross Technology"
        //   * 262
        //      * "v1.325.50.07 - Asobo Studio - Internal Cross Technology"
        //      * "v1.220.50.07 - Asobo Studio - Internal Cross Technology"
    std::uint32_t versionMinor;
        // Unused
        // Corresponds to the minor version in the version string
        // versionMinor values and the corresponding versionString values:
        //   * 380
        //      * "v1.381.67.09 - Asobo Studio - Internal Cross Technology"
        //      * "v1.381.66.09 - Asobo Studio - Internal Cross Technology"
        //      * "v1.381.65.09 - Asobo Studio - Internal Cross Technology"
        //      * "v1.381.64.09 - Asobo Studio - Internal Cross Technology"
        //      * "v1.379.60.09 - Asobo Studio - Internal Cross Technology"
        //   * 326
        //      * "v1.325.50.07 - Asobo Studio - Internal Cross Technology"
        //   * 221
        //      * "v1.220.50.07 - Asobo Studio - Internal Cross Technology"
    std::uint32_t versionMajor;
        // Unused
        // Values used in FUEL:
        //   * 144
        //      * "v1.220.50.07 - Asobo Studio - Internal Cross Technology"
        //   * 146
        //      * "v1.325.50.07 - Asobo Studio - Internal Cross Technology"
        //   * 211
        //      * "v1.379.60.09 - Asobo Studio - Internal Cross Technology"
        //   * 249
        //      * "v1.381.64.09 - Asobo Studio - Internal Cross Technology"
        //      * "v1.381.65.09 - Asobo Studio - Internal Cross Technology"
        //   * 252
        //      * "v1.381.66.09 - Asobo Studio - Internal Cross Technology"
        //   * 253
        //      * "v1.381.67.09 - Asobo Studio - Internal Cross Technology"
    BlockDescription blockDescriptions[64];
        // Array of BlockDescriptions
        // Only the first blockCount number of block descriptions are populated with information
        // The remaining descriptions are filled with null bytes
    std::uint32_t poolManifestPaddedSize;
        // poolManifestPaddedSize << 0xB is the size of the pool manifest
        // Equals 0 when there is no pool manifest
    std::uint32_t poolManifestOffset;
        // poolManifestOffset << 0xB is the offset from the beginning of the file to the beginning of the pool manifest
        // Always equals primaryHeader.paddedSize + sizeof(PrimaryHeader) when there is a pool manifest
        // Equals 0 when there is no pool manifest
    std::uint32_t poolManifestUnused0;
        // NEEDS WORK
    std::uint32_t poolManifestUnused1;
        // NEEDS WORK
    std::uint32_t poolObjectDecompressionBufferCapacity;
        // Always equals calculatePaddedSize(max(poolObjects.decompressedSize)) >> 11
        // Regardless of whether the object is compresssed
        // If this value is larger than the existing pool object decompression buffer, the buffer is resized to poolObjectDecompressionBufferCapacity << 11 bytes
        // Equals 0 when there is no pool manifest
    std::uint32_t blockSectorPaddingSize;
        // Unused
        // Number of pad bytes in the block sector
        // 0xFFFFFFFF if there is no incredibuilder data
    std::uint32_t poolSectorPaddingSize;
        // Unused
        // Number of pad bytes in the pool sector
        // 0xFFFFFFFF if there is no incredibuilder data
    std::uint32_t fileSize;
        // Unused
        // Size of the file on disk
        // 0xFFFFFFFF if there is no incredibuilder data
    char incrediBuilderString[128];
        // https://en.wikipedia.org/wiki/Incredibuild
        // Filled with 0xFF if there is no incredibuilder data
        // Irrelevant to decoding the BigFile format
        // All variables prefixed by incrediBuilder may safely be filled with null bytes
    std::uint8_t padding[64];
        // 64 0xFF bytes
};

Block Description

struct BlockDescription
{
    std::uint32_t objectCount;
        // Number of objects in the object array
        // This field cannot be 0
    std::uint32_t paddedSize;
        // Size of the block data including padding
    std::uint32_t dataSize;
        // Size of the block data excluding padding
    std::uint32_t workingBufferOffset;
        // NEEDS WORK
    std::uint32_t crc32;
        // CRC32 hash of the name of the first object in the block
    std::uint32_t checksum;
        // This used to be the Alternative CRC32 hash of the block without padding in previous versions
        // Always 0
};

Block

struct Block
{
    BlockObject blockObjects[objectCount];
    std::uint8_t padding[paddedSize - dataSize];
};

Block Object

struct BlockObject
{
    ObjectHeader objectHeader;
    std::uint8_t data[dataSize];
};

Object Header

struct ObjectHeader
{
    std::uint32_t dataSize;
        // Size of the object in the file excluding padding
    std::uint32_t classObjectSize;
        // Size of the class object data from the beginning of the data
        // Offset from the beginning of the data to the beginning of the object data
        // Always 0 in non-block objects
    std::uint32_t decompressedSize;
        // Size of the object data after decompression
        // Equal to 0 when object data is stored in the object sector
    std::uint32_t compressedSize;
        // Size of the compressed object data
        // Equal to dataSize when data is LZ compressed
        // Equal to 0 when data is not LZ compressed
    std::uint32_t classCRC32;
        // CRC32 value of the class name
        // See the appendix CRC32 Reverse Lookup table
    std::uint32_t crc32;
        // CRC32 hash of the object name string
};

Pool Manifest

struct PoolManifest
{
    PoolManifestHeader poolManifestHeader;
    PascalArray<std::uint32_t> objectsCRC32s;
        // Index in crc32s of the CRC32 hash of the object at the current index
        // Not unique from 0 to count-1
        // Same size as object array
    PascalArray<std::uint32_t> crc32s;
        // Array of the CRC32 values of the object name strings
    PascalArray<std::uint32_t> referenceCounts;
        // Number of times the CRC32 hash at the same index in crc32s is referenced in the pool sector
    PascalArray<std::uint32_t> objectPaddedSize;
        // objectPaddedSize << 0xB is the padded size of the object
    PascalArray<std::uint32_t> referenceRecordsIndices;
        // Each value is unique between 1 and count inclusive
        // Values of 0 are skipped in processing
        // 1 is subtracted from the value and used as an index in referenceRecords
    PascalArray<ReferenceRecord> referenceRecords;
        // Describes relationships between objects
        // It is safe to remove duplicates and reorder this array as long as referenceRecordsIndices is updated accordingly
    ReferenceRecord terminal;
        // The non-placeholder value fields are always 0
    std::uint8_t padding[poolManifestPaddedSize - sizeof(/* other fields */)];
};

Pool Manifest Header

struct PoolManifestHeader
{
    std::uint32_t equals524288;
        // Unused
        // Always 524288 = 256 << 11
        // Must be a multiple of 2048
    std::uint32_t equals2048;
        // Unused
        // Always 2048 = 1 << 11
        // Could be related to the address of the first block or size of the primary header or padding
        // Must be a multiple of 2048
    std::uint32_t objectsCRC32CountSum;
        // Unused
        // Sum of the objectsCRC32Count values of all the reference records
};

Reference Record

struct ReferenceRecord
{
    std::uint32_t startChunkIndex;
        // Equals poolManifestOffset + poolManifestPaddedSize + sum(objectPaddedSize[all objects before the one referenced by objectsCRC32StartingIndex])
        // Often equal to the endChunkIndex of the structure right before this one
        // The objectPaddedSize is added to this to get a temporary value
        // Idk how the first startChunkIndex index is calculated but its probably related to the number of objects
    std::uint32_t endChunkIndex;
        // Unused
        // Equals poolManifestOffset + poolManifestPaddedSize + sum(objectPaddedSize[all objects before the one referenced by objectsCRC32StartingIndex + objectsCRC32Count])
        // Often equal to the startChunkIndex of the structure right after this one
        // End chunk index
    std::uint32_t objectsCRC32StartingIndex;
        // Corresponds to an index of objectsCRC32s in the pool manifest
        // Overwritten at runtime with a calculated value
        // Some number is added to each objectsCRC32index
        // The same number is added to all objectsCRC32indexs in the same BigFile
        // The added number is constant across multiple runs
    std::uint16_t placeholderBigFileIndex;
        // Always 0
        // Over written at runtime with the BigFile's index
        // A BigFile's index represents when it was loaded in the load order
        // First BigFile to load will have 0, second will have 1, and so on
    std::uint16_t objectsCRC32Count;
        // Never 0
        // Number of elements in objectsCRC32 to iterate over starting at objectsCRC32StartingIndex
    std::uint32_t placeholderTimesReferenced;
        // Always 0xFFFFFFFF
        // Over written at runtime to the total number of times this structure has been referenced
    std::uint32_t placeholderCurrentReferencesShared;
        // Always 0xFFFFFFFF
        // Overwritten at runtime to the number of shared current references to this structure
    std::uint32_t placeholderCurrentReferencesWeak;
        // Always 0xFFFFFFFF
        // Overwritten at runtime to the number of weak current references to this structure
};

PoolObject

struct PoolObject
{
    ObjectHeader objectHeader;
        // If objectHeader.crc32 is identical to an objectHeader.crc32 in another BigFile then the two objectHeaders MUST be identical.
    std::uint8_t data[dataSize];
    std::uint8_t padding[calculatePaddingSize(sizeof(ObjectHeader) + dataSize)];
};

Associated File Types

NXX

Names XX

Plaintext table of Asobo CRC32 hashes and their human readable strings. CRC32 Hashes are represented as signed 32bit integers despite internally being used as unsigned 32bit integers. The distinction between signed and unsigned is important because the hashing algorithm makes use of the right shift operation which compiles to different machine code instructions for signed and unsigned integers. It is unclear why the NXX format uses signed integers when the BigFile format uses unsigned integers but my guess is that fprintf is used with the %d format specifier in their internal build tools because somebody didn’t want to lookup the unsigned integer format specifier.

An NXX file consists of a list of names formatted as follows:

<std::int32_t hash> "<std::string name>"<\n>

Sample:

0 ""
349401345 "DB:>SOUNDS>SFX>MENU>I_ME_CHANGE.WAV"
1202944770 "DB:>LEVELS>FONTES>FONTES.TOTEMBITMAP"
-681630461 "DB:>TEXTURES>FONTES>FNT_FONT1.TGA"
1223335941 "SMALL_FONT"
<...>
1404141044 "DB:>SOUNDS>SFX>MENU>I_IT_2.WAV"
-683453185 "DB:>LEVELS>FONTES>FONTES>FONTES_001.TBITMAP"

LXX

Layout XX

Describes the contents of a BigFile archive. Analogous to a PDB file for an EXE.

BlockSectorPaddingSize 1344
PoolSectorPaddingSize 0
BigFileSize 59392
ObjectsSize 55904
NbObject 4
EndDiskSize
SortByBlock
  0/  0	   55498	RTC_Z					"DB:>RTC>TRAFFIC2_RTC.TRTC"
  0/  1	     272	NODE_Z					"DB:>RTC>TRAFFIC2_RTC.TRTC_BF_NODE_Z_-33686019"
  0/  2	     146	CAMERA_Z					"DB:>RTC>TRAFFIC2_RTC.TRTC_BF_CAMERA_Z_0"
  0/  3	      84	LIGHTDATA_Z					"DEFAULTSUNLIGHT"
EndSortByBlock
SortByDiskSpace
   55498	"DB:>RTC>TRAFFIC2_RTC.TRTC"
     272	"DB:>RTC>TRAFFIC2_RTC.TRTC_BF_NODE_Z_-33686019"
     146	"DB:>RTC>TRAFFIC2_RTC.TRTC_BF_CAMERA_Z_0"
      84	"DEFAULTSUNLIGHT"
EndSortByDiskSpace
SortByClassSize
CLASS RTC_Z           	55498
   55498	"DB:>RTC>TRAFFIC2_RTC.TRTC"
CLASS NODE_Z          	272
     272	"DB:>RTC>TRAFFIC2_RTC.TRTC_BF_NODE_Z_-33686019"
CLASS CAMERA_Z        	146
     146	"DB:>RTC>TRAFFIC2_RTC.TRTC_BF_CAMERA_Z_0"
CLASS LIGHTDATA_Z     	84
      84	"DEFAULTSUNLIGHT"
EndSortByClassSize
SortByInfos
	"DB:>RTC>TRAFFIC2_RTC.TRTC"  Static 4 Stream 55470 FriendlyName "TRAFFIC2" 
	"DB:>RTC>TRAFFIC2_RTC.TRTC_BF_NODE_Z_-33686019"  Static 4 Stream 244 FriendlyName "ASOBO001" 
	"DB:>RTC>TRAFFIC2_RTC.TRTC_BF_CAMERA_Z_0"  Static 110 Stream 12 FriendlyName "TRAFFIC2" 
	"DEFAULTSUNLIGHT"  Static 4 Stream 56 FriendlyName "DEFAULTSUNLIGHT" 
EndSortByInfos
SortDependencies
1	"DB:>RTC>TRAFFIC2_RTC.TRTC"
		"DB:>RTC>TRAFFIC2_RTC.TRTC_BF_NODE_Z_-33686019"
EndSortDependencies
SortShaders
EndSortShaders

DXX.LAYOUT

Data XX Layout

Older format of LPC files.

Sample:

00	O	       0	S	   18414	P	       0	[DB:>SOUNDS>SFX>MENU>I_IT_1.WAV]
00	O	   18430	S	   44814	P	   28118	[DB:>SOUNDS>SFX>MENU>I_IT_3.WAV]
00	O	   46564	S	   65550	P	   32458	[DB:>SOUNDS>SFX>MENU>I_ME_INVALID.WAV]
00	O	   79038	S	  106510	P	   42542	[DB:>SOUNDS>SFX>MENU>I_ME_VALID.WAV]
01	O	       0	S	   53614	P	   34344	[DB:>SOUNDS>SFX>MENU>I_IT_GO.WAV]
01	O	   34360	S	   90126	P	   26922	[DB:>SOUNDS>SFX>MENU>I_ME_SHIFT.WAV]
02	O	       0	S	   51214	P	   18247	[DB:>SOUNDS>SFX>MENU>I_ME_BACK.WAV]
02	O	   18263	S	   65550	P	    9590	[DB:>SOUNDS>SFX>MENU>I_ME_CHANGE.WAV]
02	O	   27869	S	   48814	P	   31391	[DB:>SOUNDS>SFX>MENU>I_IT_2.WAV]
Sort by Disk Space
   42542	[DB:>SOUNDS>SFX>MENU>I_ME_VALID.WAV]
   34344	[DB:>SOUNDS>SFX>MENU>I_IT_GO.WAV]
   32458	[DB:>SOUNDS>SFX>MENU>I_ME_INVALID.WAV]
   31391	[DB:>SOUNDS>SFX>MENU>I_IT_2.WAV]
   28118	[DB:>SOUNDS>SFX>MENU>I_IT_3.WAV]
   26922	[DB:>SOUNDS>SFX>MENU>I_ME_SHIFT.WAV]
   18414	[DB:>SOUNDS>SFX>MENU>I_IT_1.WAV]
   18247	[DB:>SOUNDS>SFX>MENU>I_ME_BACK.WAV]
    9590	[DB:>SOUNDS>SFX>MENU>I_ME_CHANGE.WAV]

IWR

Information World Reference (I could stand for anything. Internal/Input/Include)

Describes TWORLDREF dependencies. Only USA1.DPC in FUEL has one of these that isn't empty.

An IWR file consists of a list of entries as follows:

<std::string linkName><\n>
<std::string worldReferncePath><\n>
<for each dependency>
<\t><std::string dependencyPath><\n>
<end for each>

Sample:

CR2
DB:>LEVELS>HUBS>QUARTIER_1>3DNODEGEOMETRY>CR2.TWORLDREF
	DB:>LEVELS>COURSES3>QUARTIER_1>3DNODEGEOMETRY>PROJECTEURROT_007.TROTSHAPE
CR36
DB:>LEVELS>HUBS>QUARTIER_1>3DNODEGEOMETRY>CR36.TWORLDREF
	DB:>LEVELS>COURSES1>QUARTIER_1>3DNODEGEOMETRY>PROJECTEURROT_005.TROTSHAPE
	DB:>LEVELS>HUBS>QUARTIER_1>3DNODEGEOMETRY>EVENT5_001.TVOLUME
	DB:>LEVELS>HUBS>QUARTIER_1>3DNODEGEOMETRY>EVENT4_001.TVOLUME
	DB:>LEVELS>HUBS>QUARTIER_1>3DNODEGEOMETRY>EVENT3_001.TVOLUME
	DB:>LEVELS>HUBS>QUARTIER_1>3DNODEGEOMETRY>EVENT2_001.TVOLUME
<...>
CR69
DB:>LEVELS>COURSES1>QUARTIER_1>3DNODEGEOMETRY>CR69.TWORLDREF
	DB:>LEVELS>COURSES1>QUARTIER_1>3DNODEGEOMETRY>PROJECTEURROT_013.TROTSHAPE
CR04
DB:>LEVELS>COURSES1>QUARTIER_1>3DNODEGEOMETRY>CR04.TWORLDREF

JPG

It's just a jpg image file. Idk why it's there. Identical except for the filename which changes to match the BigFile it goes to. Only in the world folder for Monopoly.

Sample:

A 2048px by 2048px black square image with a 1px thick white outline of a 601px by 1862px rectangle 93px away from the edge of the image top, left, and bottom.