-
Notifications
You must be signed in to change notification settings - Fork 237
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Game data loading optimizations. #1161
Merged
Grossley
merged 36 commits into
UnderminersTeam:master
from
VladiStep:dataLoadingOptimizations
Mar 5, 2023
Merged
Game data loading optimizations. #1161
Grossley
merged 36 commits into
UnderminersTeam:master
from
VladiStep:dataLoadingOptimizations
Mar 5, 2023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Download the artifacts for this pull request here: GUI:
CLI: |
VladiStep
force-pushed
the
dataLoadingOptimizations
branch
from
January 23, 2023 19:29
3512619
to
394a18a
Compare
VladiStep
force-pushed
the
dataLoadingOptimizations
branch
from
January 24, 2023 12:12
20cc73f
to
0d6055a
Compare
VladiStep
force-pushed
the
dataLoadingOptimizations
branch
from
February 1, 2023 11:17
c185688
to
0bc4fed
Compare
…s for the object pool dictionaries optimization.
VladiStep
force-pushed
the
dataLoadingOptimizations
branch
from
February 1, 2023 11:54
0bc4fed
to
8d43b4f
Compare
…erializeObjectCount()" for every "UndertaleSingleChunk<T>".
VladiStep
force-pushed
the
dataLoadingOptimizations
branch
from
February 8, 2023 15:59
ea11852
to
b25ae9d
Compare
1) Fixed count unserialization of some lists. 2) Added "IStaticChildObjectsSize". 3) Improved count unserialization error handling. 4) Implemented count unserialization for some models. 5) Little optimizations of code unserialization.
VladiStep
force-pushed
the
dataLoadingOptimizations
branch
from
February 8, 2023 16:01
b25ae9d
to
9359326
Compare
…r changes. 1) Removed leftover variables in "UndertaleCode". 2) Renamed "UndertaleListBase<T>.AddDirect()" to "...InternalAdd()". 3) More improvements of count unserialization error handling. 4) Fixed "UndertaleReader.GetUnserializeCountFunc()" for generic types. 5) Fixed count unserialization of some chunks and lists. 6) Moved GMS 2.3.1 check from "UndertaleAnimationCurve.Point" to the "ACRV" chunk. 7) More of little optimizations of some models unserialization. 8) Reorganized track keyframes related code of "UndertaleSequence".
VladiStep
force-pushed
the
dataLoadingOptimizations
branch
from
February 13, 2023 18:11
ce4ac18
to
892ec5c
Compare
…r changes (2). 1) Improved room game object unserialization, removed related redundant classes. 2) More of little unserialization optimizations (2). 3) Un-hardcoded "UndertaleChunkOBJT.CheckFor2022_5()" event count. 4) Updated a TODO of "UndertaleGameObject.cs" for EventHandlerFor() overloads.
VladiStep
force-pushed
the
dataLoadingOptimizations
branch
2 times, most recently
from
February 18, 2023 21:40
7628bb0
to
b8e2a2a
Compare
VladiStep
force-pushed
the
dataLoadingOptimizations
branch
from
February 19, 2023 12:44
b8e2a2a
to
eb001ad
Compare
VladiStep
force-pushed
the
dataLoadingOptimizations
branch
from
February 26, 2023 11:42
33467f0
to
6cd0be0
Compare
VladiStep
force-pushed
the
dataLoadingOptimizations
branch
from
February 26, 2023 11:57
6cd0be0
to
394fc35
Compare
1) "Position" of the binary readers is now "long" (prevents casting to "uint"). 2) Moved boundary checks of "BufferBinaryReader" to "ChunkBuffer".
1) Increased size of the chunk buffer to 12 MB. 2) Fixed all unserialization errors. 3) Slightly improved performance.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
UndertaleReader.objectPool
/objectPoolRev
). [most of the LOC additions count from that]Basically, now it reads through the file twice - the first pass only counts all objects that would be in the object pool.
This new stage is much faster than a full data reading, but in return it significantly reduces RAM usage.
Undertale-
lists are inherited fromUndertaleListBase
that hasSetCapacity()
andInternalAdd()
(adds an item withoutObservableCollection
-related overhead).This should increase overall data reading performance, especially if there's some antivirus or the game data file is on HDD.
UndertaleReader
now inherits fromAdaptiveBinaryReader
- a new type of binary reader that uses re-implementedBufferBinaryReader
if the current chunk is not bigger than 12 MB (the size of the new chunk buffer), andFileBinaryReader
otherwise (then it would read the chunk directly from file, as before).UndertaleReader.Position
islong
instead ofuint
.This also improves game data loading performance a little bit, because it prevents the conversion from
long FileStream.Position
touint
.List
s where it was necessary.a)
UndertaleGameObject
- using staticeventTypesLength
.b)
UndertaleEmbeddedTexture
- checking for PNG end without.GetString()
.c)
UndertaleSimpleListShort
- simplified the length check.d)
UndertaleAnimationCurve
- moved GMS 2.3.1 check from "UndertaleAnimationCurve.Point" to the "ACRV" chunk..reader.ReadByte()
withPosition++
.Closes Certain games' data.win will get UTMT to load seemingly indefinitely #1185.
UndertaleRoom.GameObjects
list unserialization (GMS 2.2.2.302 detection) - removedUndertalePointerListLenCheck
list type.UndertaleSequence.TrackKeyframes
type - now it's a generic type, so "UndertaleSequence.cs" file is smaller.UndertaleChunkXXXX.UnserializeChunk()
to separate methods (UndertaleChunkXXXX.CheckForGM...()
).UndertaleChunkTXTR.UnserializeChunk()
).All of these improvements made game data loading both faster (even in comparison to 0.4.0.4 that copied all game data to RAM) and less memory expensive.
Here are the comparisons ("Release" build mode):
[first loading time, in milliseconds]
[second loading time (with the same data already loaded), in milliseconds]
[peak RAM usage, in MB]
Caveats
The scripts that are using
UndertaleReader.Position
probably should be fixed.