BigFile Needs Work

Questions I still want answers to:

General

• When does an object get compressed?
  • Size?
  • Type?
  • Ratio? - anything bellow or equal to 20%/25% is not compressed floor the ratio before the comparison ✅
  • The first object in a block is rarely compressed. Is this to minimize the offset? probably not

Header

Primary Header

poolManifestUnused0 and poolManifestUnused1

• These fields are copied to the internal DPC header representation but those copies do not appear to be used
• Always equal to each other
• Equals 0 when there is no pool manifest

BlockDescription

workingBufferOffset

• This is often extremely large for the last block
• This may be non-zero even when the block contains no compressed objects and the DPC contains no pool
  • HUB_YELLOWS and TELEPORT_HELICO are good examples of the previous statement
  • TELE_01 has 1 object in the last block and a massive offset
  • THE_INTRO has a block with 0 offset
  • USA1 has all 0 offsets except for the last block which is >10,000,000
• When isNotRTC == 1 this value is used as the workingBufferOffset otherwise it is unused but will still have a value
• This is the size of the area used to decompress/process blockObjects aka how far from the beginning of the working buffer to offset the block data which may or may not have compressed objects in it
• The previous statement is only half true; for the last block this field is exceptionally large
• The last one might be big because it has to accommodate pool stuff sometimes (not true, big even if no pool)
• Indexing data similar to the pool manifest can be found in this space while the last block is processed
• This could be the reason last block has a big value
• Might be where the lookup table for objects in the dpc winds up
• This may be 0 and contain compressed objects if the compressed data is sufficiently far enough in the block that the space before it is enough to decompress
• If there is at least one compressed objects then this may be equal to calculatePaddedSize(largestCompressedObject.objectHeader.decompressedSize - (&largestCompressedObject.data - &block))
• following the previous rule the game will load without crashing although the original value is usually larger
• It is possible that some classes may use this buffer while loading (probably not)
• Stages 7, 8, and 9 call ResourceObject_Z virtual methods
• Is it constructing the classes in place in this buffer? (no, this pointer isn't in the buffer)
• Always divisible by 2048

Block Sector

• How is the number of blocks determined?
  • All of the objects may safely be placed into a single block or moved between any number of blocks. ✔️
  • Seem to be split to have a similar size so that loading doesn't stall too long on a single block
• How is the order of blocks determined?
  • The blocks may occur in any order so long as blockWorkingBufferCapacityEven and blockWorkingBufferCapacityOdd are appropriate. ✔️
  • Maybe related to the lexicographical order of the path strings before hashing?
  • The decompressed size of the largest compressed object in each block seems to decrease as the block index increases
• How is the order of objects in a block determined?
  • The objects may occur in any order so long as the block offset is appropriate. ✔️
  • Maybe related to the lexicographical order of the path strings before hashing?
  • Maybe ordered to reduce the block offset?
  • Maybe ordered by dependency excluding duplicates?
  • Maybe an unordered_map-like container was used and it’s effectively random?
  • Maybe ordered by the hash value signed/unsigned/string? NO.
  • In P_MOTO.DPC and P_BUGGY.DPC the objects are ordered by dependency with the root of the dependency graph being first.

Pool Sector

• When is an object bumped to the pool?
  • References?
  • Size? NO.
  • Type?
  • Is it mandatory or an optimization? It is an optional optimization.
  • Other DPCs share the object? NO.
• Why are there duplicate objects in the pool?
  • Something to do with the references needing continuous intervals ✅
  • Locality on disk for console BigFiles
• When does an object reference another?
  • CRC used in objects data?
  • CRC used in base class object header?
  • Whenever LoadLinkID is used to read the field ✅
• How are the associative arrays in the pool manifest generated?
  • Maybe an unordered_map-like container was used and it’s effectively random?
  • Maybe related to the lexicographical order of the path strings before hashing?