Proposed RFC Feature New O3DE Archive System

[lemonade-dm]

Summary:

To add a modern and performant archive format to O3DE, that can overcome the existing limitations of the current .pak file format based on CryPak.

Newer specs for storage technology such as NVMe.4 and NVMe.5 drives has led to the hardware being able to support throughput speeds of 16GiB/s in 4-lane configuration which is common for storage drives. Speeds of up to 64GiB would be supported in a 16-lane configuration, but that is not common nor affordable among consumer level storage drives.

Currently CryPak is decompression limited to zlib-based compression algorithms. Alternative compression libraries such as Oodle and Snappy. Furthermore the compression is file based, instead of block based, which makes it hard to parallelize decompression and reads on multiple threads.
Currently within the AZ Streamer stack decompression of files from .pak files is the bottleneck when it comes to throughput.
Finally the CryPak that O3DE uses only supports 32-bit zip files and have a 4GiB limit on the archive file itself.

The new O3DE Archive format will address these issues, by providing a Compression gem that allows performing block based decompression, whille also supporting 64-bit offsets to allow for archive files of up to 172 TiB.

The full RFC for the new Archive TDD is linked at https://github.com/o3de/sig-core/blob/daa015f/rfcs/rfc-core-2023-04-20-new-archive-tdd.md