feat: SQLite+Compressed save format (Phase 2 of v2 save format)

[randombk]

Checklist

Required

• I wrote the PR title in conventional commit format.
• I ran the code formatter. => Awaiting CI bot
• I linked any relevant issues using github keyword syntax so it can be closed automatically.
• I have committed my changes to new branch that isn't main so it won't cause conflict when updating main branch later.

Purpose of change

This PR is Phase 2 of #5771, and supersedes #5772.

• This builds on top of refactor: Clean up world saving/loading codepath (Phase 1 of v2 save format) #5774 to implement the new save format. It should only be merged after refactor: Clean up world saving/loading codepath (Phase 1 of v2 save format) #5774 is merged.

This patch implements a new V2 save layout based on SQLite3 and gzip.

• A save can either be V1 (current layout) or V2.
• Existing V1 saves will continue to use V1.
• V1 saves can be converted to V2 using a new option in the "World" tab of the main menu.
• Newly created saves can opt-in to use V2.

Describe the solution

The new save format consolidates and compresses certain high-volume files into a set of sqlite3 database files with gzip compression. Specifically,

map.sqlite3 replaces:

• maps/ directory (map tile info)
• o.* files (overmap info)

<player_id>.sqlite3 replaces:

• <player_id>.seen.* files (player overmap visibility data)
• <player_id>.mm1/ directory (player map tile memory)

All other files will remain unchanged. This should cover all the worst offenders while leaving the more manual-edit-friendly files untouched.

Each database consists of a single table:

CREATE TABLE IF NOT EXISTS files (
    path           TEXT PRIMARY KEY NOT NULL,
    parent         TEXT NOT NULL,  -- Currently unused, but adding here for future compat considerations
    compression    TEXT DEFAULT NULL,  -- NULL for uncompressed. 'zlib' if `data` is compressed.
    data           BLOB NOT NULL
);

Benefits:

• Greatly reduced size-on-disk.
• Greatly reduced number of files in a save. This should benefit backups, cross-device copies, and other systems requiring file watchers.
• Improved consistency. Crashes in the middle of a save no longer cause save corruption.

Costs:

• Slightly degraded save/load performance.
• For large worlds, replaced a large number of very small files with a small number of fairly large files.

There is an interesting trade-off between file size and save performance. Even in the old format, save/load was not a particularly big issue. Only changed chunks were modified, and autosave triggered often enough that most save events only covered a small number of chunks.

The new format does introduce an additional delay to the save time, though events remained sub-second. All of this delay can be attributed to the compression process - turning off compression resulted in no measurable changes in save performance. This is to be expected - we're introducing an extra CPU-intensive step into a single-threaded save process.

Benchmarks & Comparisons

The below are some rough figures I captured during my testing. All of these were done on a Ryzen 9 7900X system running Arch Linux with the save directory located on a btrfs partition stored on a NVME SSD.

Some figures using a small test save (after ~3hrs gameplay):

mode	size	# files	Save Time
Baseline V1 (current)	70.5 MB	4,174	119ms
V2, gzip level 1 (this PR)	10.0 MB	15	170ms
V2, gzip level 6	7.2 MB	15	320ms

Using a medium-sized test save:
Figures were taken after saving at one end of a big town, taking a helicopter across the town to generate new tiles, then saving again.

mode	size	# files	Save Time
Baseline V1 (current)	128 MB	11,714	105ms
V2, gzip level 1 (this PR)	21.8 MB	15	162ms
V2, gzip level 6	17.2 MB	15	302ms

(Not sure why the bigger map was a faster save; could be multiple things, but the point is that there's a small increase)

Gzip compression level 1 ("fast") seems to be optimal. I suspect a more modern compression scheme like zstd would be even better, but we'll need to look into the packaging story.

Describe alternatives you've considered

Testing

Additional context

[autofix-ci]

Autofix has formatted code style violation in this PR.

I edit commits locally (e.g: git, github desktop) and want to keep autofix

1. Run git pull. this will merge the automated commit into your local copy of the PR branch.
2. Continue working.

I do not want the automated commit

1. Format your code locally, then commit it.
2. Run git push --force to force push your branch. This will overwrite the automated commit on remote with your local one.
3. Continue working.

If you don't do this, your following commits will be based on the old commit, and cause MERGE CONFLICT.

[scarf005]

turning off compression resulted in no measurable changes in save performance. This is to be expected - we're introducing an extra CPU-intensive step into a single-threaded save process.

while out of scope for this PR, I'm curious whether it's possible to

• ...add in settings to enable/disable save compression? would there be issue in save/loading from compressed/uncompressed state? this way players could choose tradeoff between save speed and file compression.
• ...parallelized without being too hard to implement?
• ...delegate this task to another thread(s)?

sequenceDiagram
	Note left of BN: log "saving..."
	BN-)saveThread: send save request
    activate saveThread
    alt on failure
      saveThread--)BN: log "failed saving"  
    else compress save
      saveThread--)BN: log "save success"
    end
    deactivate saveThread

Loading

[randombk]

I'm going to tweak this to make this opt-in during world creation (+ the convert-existing-world functionality).
I'll need to understand the perf implications some more, but I want to avoid adding in the multithreading complexity (particularly cross-platform) to save a couple hundred milliseconds.

Same goes for tweaking compression algo. The database has the metadata to change compression or toggle it on/off even at the map tile grain, but it's more the config UI and adding in so much complexity into something that most players shouldn't have to worry about.

[scarf005]

makes sense, to most players keeping their saves not being corrupt would be much more important than saving 100ms.

[randombk]

The V2 format is now opt-in during world creation.

There is an option to convert V1 saves to V2 in the main menu.

I tested creating/loading both V1 and V2 worlds. I converted one of my V1 worlds to V2 and have been playing on it for the past ~week without issues.

I still need to figure out the dependency/build situation for MSVC and Android - this is the last remaining item before the PR is ready. For folks on Linux, you can try out these changes already.

[randombk]

Additional TODOs: add in the required libraries into the CI environment, and update the relevant docs & wikis to document the V2 system and the new build dependencies.

[randombk]

Any idea how to trigger the MSVC and Android checks? I'm not sure why the CI skipped those

[Coolthulhu]

Looks good to me, except for the compiler warnings.

[Coolthulhu]

Not needed here, but it might be a good idea to extract it to an option later.

[scarf005]

also related: #3150

[randombk]

This now builds across all platforms, and I've tested that the Linux, Windows (MSVC), and Android binaries work and can save/load successfully.

The MXE build is still broken and will likely remain so until BrettDong/MXE-GCC#1 is resolved, assuming we want to keep MXE in the first place.

A batch of working binaries can be found at https://github.com/randombk/Cataclysm-BN/releases/tag/v0.6.0

[RobbieNeko]

The MXE build is still broken and will likely remain so until BrettDong/MXE-GCC#1 is resolved, assuming we want to keep MXE in the first place.

Oh, very good thing we fixed the MSVC builds then.
As for whether we want to keep MXE or not: It's mostly just kept around as a backup for the sake of added stability and helping to prevent releases where no Windows builds succeeded. For a while it was actually broken itself for various reasons, so we have gotten by without it in the past. (Not even mentioning the fact that Linux is getting by just fine with only one tiles build period, although part of that is probably due to just how many linux devs there are on the team and thus Linux getting a more first-class experience)

[scarf005]

let's goooooooo

(but @chaosvolt please playtest just one more time before we ship it)
(also we should probably make a major release first)
(also we should make an announcement a week before merging it first)

[chaosvolt]

Re-running tests due to that error that kept popping up, but will test in a bit.

[RobbieNeko]

That Clang-tidy check has been failing on a bunch of PRs recently.

[chaosvolt]

I might go ahead and update the PR from upstream before I test and that should fix it since I think we tackled this in a recent PR, but that will cancel Scarf's review. ;~;

[chaosvolt]

EDIT: I feex, just had to update vcpkg, compile-testing now.

[chaosvolt]

1. Compiled and load-tested.
2. Started up an empty world created before compiling.
3. Saved and loaded, no odd behaviors found.
4. Started a save in build 2025-01-07 then transferred it over.
5. Loading the save works without issue, saving the old world in the new build also works fine.
6. Hit convert to V2 format, Avast freaked the fuck out at me but conversion went as expected after I told it to fuck off.

This is uh...confusing though, and you can convert a save to V2 even after it's already been converted? Uhhh...

EDIT: Ah I see, you can just spam infinite backups by trying to convert the V1 backups made from converting, lel.

[chaosvolt]

It implodes in a relatively safe manner so that's prolly okay.

[scarf005]

That Clang-tidy check has been failing on a bunch of PRs recently.

it's due to esm.sh not returning same file, this might be related: esm-dev/esm.sh#614

EDIT: how strange, it works on my machine

scarf@fedora /v/h/s/r/c/Cataclysm (sqlite-saves-phase2)> deno task affected-files 5777
Task affected-files deno run -RWEN --allow-run scripts/affected_files.ts "5777"
{
  src: [
    "src/compress.cpp",
    "src/main_menu.cpp",
    "src/world.cpp",
    "src/worldfactory.cpp"
  ],
  headers: [ "src/compress.h", "src/world.h", "src/worldfactory.h" ]
}
Set(30) {
  "src/compress.cpp",
  "src/main_menu.cpp",
  "src/world.cpp",
  "src/worldfactory.cpp",
  "src/diary.cpp",
  "src/overmap.cpp",
  "src/worldfactory.h",
  "src/mapbuffer.cpp",
  "src/auto_note.cpp",
  "src/artifact.h",
  "src/clzones.cpp",
  "src/auto_pickup.cpp",
  "src/map_memory.cpp",
  "src/overmapbuffer.cpp",
  "src/safemode_ui.cpp",
  "src/artifact.cpp",
  "tests/filesystem_test.cpp",
  "src/options.cpp",
  "tests/test_main.cpp",
  "src/init.cpp",
  "src/catalua.cpp",
  "src/mod_manager.cpp",
  "src/newcharacter.cpp",
  "src/game.cpp",
  "src/distribution_grid.cpp",
  "src/handle_action.cpp",
  "src/language.cpp",
  "src/main_menu.h",
  "src/sdltiles.cpp",
  "src/debug.cpp"
}

[randombk]

This is uh...confusing though, and you can convert a save to V2 even after it's already been converted? Uhhh...

That's an unfortunate side effect of how the main UI is laid out - every menu option has to have the same submenu options, and the main UI doesn't know what's V1 or V2 until you actually start the conversion process.

It's fixable, but would require a major rewrite of the main UI

[scarf005]

let's goooooooooo

[RoyalFox2140]

Item

Identity.