Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Game data loading optimizations. #1161

Merged
merged 36 commits into from
Mar 5, 2023

Conversation

VladiStep
Copy link
Member

@VladiStep VladiStep commented Jan 23, 2023

Description

  1. Added new game loading stage that allows to set an initial capacity for the object pools (UndertaleReader.objectPool /objectPoolRev). [most of the LOC additions count from that]
    Basically, now it reads through the file twice - the first pass only counts all objects that would be in the object pool.
    This new stage is much faster than a full data reading, but in return it significantly reduces RAM usage.
  2. All Undertale- lists are inherited from UndertaleListBase that has SetCapacity() and InternalAdd() (adds an item without ObservableCollection-related overhead).
  3. Game data loads by copying it to a buffer chunk by chunk.
    This should increase overall data reading performance, especially if there's some antivirus or the game data file is on HDD.
    UndertaleReader now inherits from AdaptiveBinaryReader - a new type of binary reader that uses re-implemented BufferBinaryReader if the current chunk is not bigger than 12 MB (the size of the new chunk buffer), and FileBinaryReader otherwise (then it would read the chunk directly from file, as before).
  4. UndertaleReader.Position is long instead of uint.
    This also improves game data loading performance a little bit, because it prevents the conversion from long FileStream.Position to uint.
  5. Set initial capacity for Lists where it was necessary.
  6. Various unserialization optimizations:
    a) UndertaleGameObject - using static eventTypesLength.
    b) UndertaleEmbeddedTexture - checking for PNG end without .GetString().
    c) UndertaleSimpleListShort - simplified the length check.
    d) UndertaleAnimationCurve - moved GMS 2.3.1 check from "UndertaleAnimationCurve.Point" to the "ACRV" chunk..
  7. Replaced every instance of dummy reader.ReadByte() with Position++.
  8. Skipped redundant unserialization of duplicate instructions of "gml_Script_..." (gray code entries).
    Closes Certain games' data.win will get UTMT to load seemingly indefinitely #1185.
  9. Fixed room unserialization performance regression that was introduced 11 months ago.
  10. Fixed "audiogroup*.dat" files cannot be loaded. #1194.
  11. Simplified UndertaleRoom.GameObjects list unserialization (GMS 2.2.2.302 detection) - removed UndertalePointerListLenCheck list type.
  12. Reworked UndertaleSequence.TrackKeyframes type - now it's a generic type, so "UndertaleSequence.cs" file is smaller.
  13. Extracted all GM(S) version checks from UndertaleChunkXXXX.UnserializeChunk() to separate methods (UndertaleChunkXXXX.CheckForGM...()).
  14. A little performance improvement of embedded texture name generation (see UndertaleChunkTXTR.UnserializeChunk()).

All of these improvements made game data loading both faster (even in comparison to 0.4.0.4 that copied all game data to RAM) and less memory expensive.

Here are the comparisons ("Release" build mode):
[first loading time, in milliseconds]

0.4.0.4 "Bleeding Edge" This PR Relative to "BE" Relative to 0.4.0.4
Undertale 2208 2669 2115 -554 ms (26%) -93 ms (4%)
Deltarune Ch. 1 736 893 770 -123 ms (15%) +34 ms
Deltarune 1&2 (1.07) (unsupported) 3345 2648 -697 ms (26%) ---
Stoneshard 0.7.0.23 6473 8326 5240 -3086 ms (58%) -1233 ms (23%)
Pizza Tower (SAGE 2019) 771 942 774 -168 ms (21%) +3 ms

[second loading time (with the same data already loaded), in milliseconds]

0.4.0.4 "Bleeding Edge" This PR Relative to "BE" Relative to 0.4.0.4
Undertale 1962 2073 1385 -688 ms (49%) -577 ms (41%)
Deltarune Ch. 1 570 573 432 -141 ms (32%) -138 ms (31%)
Deltarune 1&2 (1.07) (unsupported) 2576 1925 -651 ms (33%) ---
Stoneshard 0.7.0.23 6168 7596 4649 -2947 ms (63%) -1519 ms (32%)
Pizza Tower (SAGE 2019) 559 623 511 -112 ms (21%) -48 ms (9%)

[peak RAM usage, in MB]

"Bleeding Edge" This PR Peak RAM usage diff.
Undertale +433 +299 -134 MB (44%)
Deltarune Ch. 1 +117 +107 -10 MB (9%)
Deltarune 1&2 (1.07) +498 +376 -122 MB (32%)
Stoneshard 0.7.0.23 +588 +482 -76 MB (15%)
Pizza Tower (SAGE 2019) +336 +347 +11 MB

Caveats

The scripts that are using UndertaleReader.Position probably should be fixed.

@github-actions
Copy link

github-actions bot commented Jan 23, 2023

@VladiStep VladiStep force-pushed the dataLoadingOptimizations branch from 3512619 to 394a18a Compare January 23, 2023 19:29
@VladiStep VladiStep force-pushed the dataLoadingOptimizations branch from 20cc73f to 0d6055a Compare January 24, 2023 12:12
@VladiStep VladiStep force-pushed the dataLoadingOptimizations branch from c185688 to 0bc4fed Compare February 1, 2023 11:17
…s for the object pool dictionaries optimization.
@VladiStep VladiStep force-pushed the dataLoadingOptimizations branch from 0bc4fed to 8d43b4f Compare February 1, 2023 11:54
@VladiStep VladiStep force-pushed the dataLoadingOptimizations branch from ea11852 to b25ae9d Compare February 8, 2023 15:59
1) Fixed count unserialization of some lists.
2) Added "IStaticChildObjectsSize".
3) Improved count unserialization error handling.
4) Implemented count unserialization for some models.
5) Little optimizations of code unserialization.
@VladiStep VladiStep force-pushed the dataLoadingOptimizations branch from b25ae9d to 9359326 Compare February 8, 2023 16:01
…r changes.

1) Removed leftover variables in "UndertaleCode".
2) Renamed "UndertaleListBase<T>.AddDirect()" to "...InternalAdd()".
3) More improvements of count unserialization error handling.
4) Fixed "UndertaleReader.GetUnserializeCountFunc()" for generic types.
5) Fixed count unserialization of some chunks and lists.
6) Moved GMS 2.3.1 check from "UndertaleAnimationCurve.Point" to the "ACRV" chunk.
7) More of little optimizations of some models unserialization.
8) Reorganized track keyframes related code of "UndertaleSequence".
@VladiStep VladiStep force-pushed the dataLoadingOptimizations branch from ce4ac18 to 892ec5c Compare February 13, 2023 18:11
…r changes (2).

1) Improved room game object unserialization, removed related redundant classes.
2) More of little unserialization optimizations (2).
3) Un-hardcoded "UndertaleChunkOBJT.CheckFor2022_5()" event count.
4) Updated a TODO of "UndertaleGameObject.cs" for EventHandlerFor() overloads.
@VladiStep VladiStep force-pushed the dataLoadingOptimizations branch 2 times, most recently from 7628bb0 to b8e2a2a Compare February 18, 2023 21:40
@VladiStep VladiStep force-pushed the dataLoadingOptimizations branch from b8e2a2a to eb001ad Compare February 19, 2023 12:44
@VladiStep VladiStep force-pushed the dataLoadingOptimizations branch from 33467f0 to 6cd0be0 Compare February 26, 2023 11:42
@VladiStep VladiStep force-pushed the dataLoadingOptimizations branch from 6cd0be0 to 394fc35 Compare February 26, 2023 11:57
1) "Position" of the binary readers is now "long" (prevents casting to "uint").
2) Moved boundary checks of "BufferBinaryReader" to "ChunkBuffer".
1) Increased size of the chunk buffer to 12 MB.
2) Fixed all unserialization errors.
3) Slightly improved performance.
@VladiStep VladiStep marked this pull request as ready for review March 2, 2023 15:22
@Grossley Grossley merged commit e922fd7 into UnderminersTeam:master Mar 5, 2023
@VladiStep VladiStep deleted the dataLoadingOptimizations branch March 5, 2023 17:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants