Potential lead regarding savefile encoding (Rewritten data remains in the file)

[AffectedArc07]

Don't know a better place to file this, so here we are.

When attempting to use the savefile decoding script in this repo for another project, I noticed something weird.

BYOND would read this data from the savefile

name = "DTX-PAI"
description = "A pAI version of DTX"
role = "Socialise with host, probably break down doors."
comments = "N/A"
version = 1

However, the script read this (using just description key as an example)

data entry
 unk: 01
 raw: 02 00 00 00 00 00 00 00 0b 37 39 16 0d 05 e9 f9 e6 f2 cb c3 17 00 00 00 3b 57 4c 14 7e 17 31 30 a2 fd f1 ef d5 c6 d7 af ea bc ba c5 aa a3 58 67 77 35 04 05 7f 51 2b 3d 2f 07 05 1b 19 a7 e4 f1 c7 8b dc d2 b5 bb f1 c2 9c 8e 6f 67 77 3b 46 58 42 1f 3c 39 3f 43 04 1a 0d f3
  key: description
  dat: 01 14 00 41 20 70 41 49 20 76 65 72 73 69 6f 6e 20 6f 66 20 44 54 58 6e 65 2e 20 28 49 6e 63 6c 75 64 69 6e 67 20 74 68 65 20 68 6f 73 74 29 23 76 7d 93 62 79 2c 66 71 70 24 78 74 69 1c 6c 6b 77 70
  dac: . . . A pAI version of DTXne. (Including the host)#v}�by,fqp$xti. lkwp

The important thing to note here, is the (Including the host) part. This used to be part of the description before I rewrote it and made it shorter, but it still exists in the savefile as seen with the decoder.

This means that BYOND doesn't rewrite a savefile when its edited, only changes are appended, which means there is history stored in bytes, presumably waiting to be written over.

This also means there must be "markers" for the data starting and ending. This exact savefile is available at https://affectedarc07.co.uk/pai.sav

I do not possess the knowledge myself to investigate this, but I am flagging it up here as you may be able to figure something out with it.

Edit: This image makes it much easier to visualise

[20kdc]

Quite interesting, thank you!

Some key notes:

• The end of the string is known based on the length (the 14 00).
• The savefile analyzer script shows all data to the end of the RAD entry (see https://github.com/20kdc/byond-data-docs/blob/master/formats/RAD.md ) even if there's less data described by the savefile entry stored within it.
• The more interesting point here is that it seems as if at least that BYOND version (appears to be 510) will not split RAD blocks (at least beyond a certain point, but perhaps even at all?), using more space for the entry than is strictly necessary rather than shrinking it.
• While it doesn't have an effect on the file format, as a matter of implementation behaviour knowing what RAD block structures are caused by various insertion/deletion sequences might be useful in studying ways to either prevent and/or cause bloated savefiles.

[20kdc]

Oh, also, if you just want to take an existing savefile of a relatively short length and salvage whatever strings you can out of it, try the xor-jump-9 bruteforcer, it tends to do well enough.

a4/9a
N/A RP fairly well, providing a good enough scenario is given.

[20kdc]

Ah! I found a bug in my savefile analyzer where it would still try to read those invalid RAD blocks. This failed, as could be expected. It's fixed now.

[KirbyDaMaster]

This information is very useful, can you contact me through discord so I can ask a few questions regarding save files? Kirbsuke #0857