New Immortal mod crashes to desktop with out of memory error

[hemebond]

The Immortal Lock mod crashes to desktop when you enter the first level (after start). I'm using the latest master.

Apparently even Ironwail crashes due to memory usage. vkQuake is recommended by the author.

53 memory pools, totalling 2220852746 bytes (2117.970MB)
total allocated size: 2223056920 bytes (2120.072MB)
Memory pool 0x5629dc1e1d60 has sprung a leak totalling 503018517 bytes (479.716MB)!  Listing contents...
 503015997 bytes allocated at ../../../fs.c:3499
      1024 bytes allocated at ../../../image.c:1890
       736 bytes allocated at ../../../cmd.c:1565
       736 bytes allocated at ../../../cmd.c:1565
        24 bytes allocated at ../../../cmd.c:1602
Engine Aborted: Mem_Alloc: out of memory (alloc of size 18446744073474398208.000000 (17592186044191.740MB) at ../../../model_brush.c:2996)
DarkPlaces-Quake Linux, buildstring: v20140513-1451-gd7763f6f Jul 10 2024 release

Larger snippet of start and end of error dump

SpawnServer: immortal
Mem_Alloc: pool Temporary Memory, file ../../../fs.c:3499, size 503015997.000000 bytes (479.713437 MB)
Mem_Alloc: pool maps/immortal.bsp, file ../../../model_brush.c:2270, size 23350656.000000 bytes (22.268921 MB)
Mem_Alloc: pool maps/immortal.bsp, file ../../../model_brush.c:2333, size 30157968.000000 bytes (28.760880 MB)
Mem_Alloc: pool maps/immortal.bsp, file ../../../model_brush.c:3219, size 29415888.000000 bytes (28.053177 MB)
Mem_Alloc: pool Temporary Memory, file ../../../fs.c:3499, size 99031611.000000 bytes (94.443904 MB)
Mem_Alloc: pool maps/immortal.bsp, file ../../../model_brush.c:2127, size 99031602.000000 bytes (94.443895 MB)
Mem_Alloc: pool maps/immortal.bsp, file ../../../model_brush.c:3235, size 29415000.000000 bytes (28.052330 MB)
Mem_Alloc: pool maps/immortal.bsp, file ../../../model_brush.c:2371, size 33070884.000000 bytes (31.538853 MB)
Mem_Alloc: pool maps/immortal.bsp, file ../../../model_brush.c:2572, size 167235744.000000 bytes (159.488434 MB)
Mem_Alloc: pool maps/immortal.bsp, file ../../../model_brush.c:2573, size 111490496.000000 bytes (106.325623 MB)
Mem_Alloc: pool maps/immortal.bsp, file ../../../model_shared.c:976, size 546508824.000000 bytes (521.191429 MB)
Mem_Alloc: pool maps/immortal.bsp, file ../../../model_brush.c:2187, size 33661981.000000 bytes (32.102567 MB)
Mem_Alloc: pool maps/immortal.bsp, file ../../../model_brush.c:2989, size 91291424.000000 bytes (87.062286 MB)
Mem_Alloc: pool maps/immortal.bsp, file ../../../model_brush.c:2996, size 18446744073474398208.000000 bytes (17592186044191.740234 MB)
memory pool list:
size    name
    950841k (    950900k actual) maps/immortal.bsp (+973660871 byte change) 
  91291424 bytes allocated at ../../../model_brush.c:2989

...

         2 bytes allocated at ../../../cvar.c:670
        17 bytes allocated at ../../../cvar.c:669
    491230k (    491235k actual) Temporary Memory (+503018517 byte change) TEMP
 503015997 bytes allocated at ../../../fs.c:3499
      1024 bytes allocated at ../../../image.c:1890
       736 bytes allocated at ../../../cmd.c:1565
       736 bytes allocated at ../../../cmd.c:1565
        24 bytes allocated at ../../../cmd.c:1602
53 memory pools, totalling 2220852746 bytes (2117.970MB)
total allocated size: 2223056920 bytes (2120.072MB)
Memory pool 0x5629dc1e1d60 has sprung a leak totalling 503018517 bytes (479.716MB)!  Listing contents...
 503015997 bytes allocated at ../../../fs.c:3499
      1024 bytes allocated at ../../../image.c:1890
       736 bytes allocated at ../../../cmd.c:1565
       736 bytes allocated at ../../../cmd.c:1565
        24 bytes allocated at ../../../cmd.c:1602
memory pool list:
size    name
    950841k (    950900k actual) maps/immortal.bsp (+0 byte change) 
      4625k (      4626k actual) q3shaders (+0 byte change) 
      1335k (      1338k actual) MESH_UI (+0 byte change) 
         0k (         1k actual) MESH_SCENE (+0 byte change) 
        11k (        12k actual) progs/beam.mdl (+0 byte change) 
        98k (       104k actual) progs/metal.mdl (+0 byte change) 
     15554k (     15557k actual) maps/spinny.bsp (+0 byte change) 
       399k (       400k actual) progs/dog.mdl (+0 byte change) 
       406k (       407k actual) progs/enforcer.mdl (+0 byte change) 
        79k (        83k actual) progs/backpack.mdl (+0 byte change) 
         8k (         9k actual) progs/spike.mdl (+0 byte change) 
        19k (        21k actual) progs/grenade.mdl (+0 byte change) 
        13k (        14k actual) progs/missile.mdl (+0 byte change) 
        25k (        26k actual) progs/bolt3.mdl (+0 byte change) 
         8k (         9k actual) progs/bolt2.mdl (+0 byte change) 
         8k (         9k actual) progs/bolt.mdl (+0 byte change) 
        13k (        14k actual) progs/plasma.mdl (+0 byte change) 
        59k (        60k actual) progs/c_lance.mdl (+0 byte change) 
        37k (        38k actual) progs/v_light.mdl (+0 byte change) 
        29k (        30k actual) progs/v_rock2.mdl (+0 byte change) 
        14k (        16k actual) progs/v_rock.mdl (+0 byte change) 
        43k (        44k actual) progs/v_nail2.mdl (+0 byte change) 
        45k (        46k actual) progs/v_nail.mdl (+0 byte change) 
        33k (        34k actual) progs/v_shot2.mdl (+0 byte change) 
        27k (        28k actual) progs/v_shot.mdl (+0 byte change) 
        54k (        55k actual) progs/c_ham.mdl (+0 byte change) 
        40k (        41k actual) progs/v_axe.mdl (+0 byte change) 
        20k (        22k actual) progs/gib.mdl (+0 byte change) 
         9k (        10k actual) progs/h_player.mdl (+0 byte change) 
       158k (       165k actual) progs/s_vp_pll.spr (+0 byte change) 
       505k (       507k actual) progs/player.mdl (+0 byte change) 
      5449k (      5667k actual) Server Progs (+0 byte change) 
     91999k (     92024k actual) maps/start.bsp (+0 byte change) 
         0k (         1k actual) FONT (+0 byte change) 
    220517k (    220535k actual) texture management (+0 byte change) 
      3813k (      3814k actual) client (long term memory) (+0 byte change) 
      7836k (      7838k actual) client (per-level memory) (+0 byte change) 
     19671k (     19689k actual) sound (+0 byte change) 
         0k (         1k actual) MESH_UI (+0 byte change) 
         0k (         1k actual) MESH_SCENE (+0 byte change) 
       293k (       293k actual) FONTS (+0 byte change) 
    348786k (    349643k actual) Renderer (+0 byte change) 
      1433k (      1433k actual) server (+0 byte change) 
        71k (        72k actual) collision cache (+0 byte change) 
       356k (       357k actual) modelinfo (+0 byte change) 
       418k (       418k actual) network connections (+0 byte change) 
         0k (         1k actual) crypto (+0 byte change) 
       203k (       212k actual) file management (+0 byte change) 
         2k (         5k actual) commands (+0 byte change) 
        19k (        41k actual) commands (+0 byte change) 
        28k (        46k actual) Command buffer (+0 byte change) 
      2188k (      3029k actual) Zone (+0 byte change) 
    491230k (    491235k actual) Temporary Memory (+0 byte change) TEMP
53 memory pools, totalling 2220852746 bytes (2117.970MB)
total allocated size: 2223056920 bytes (2120.072MB)
Memory pool 0x5629dc1e1d60 has sprung a leak totalling 503018517 bytes (479.716MB)!  Listing contents...
 503015997 bytes allocated at ../../../fs.c:3499
      1024 bytes allocated at ../../../image.c:1890
       736 bytes allocated at ../../../cmd.c:1565
       736 bytes allocated at ../../../cmd.c:1565
        24 bytes allocated at ../../../cmd.c:1602
Engine Aborted: Mem_Alloc: out of memory (alloc of size 18446744073474398208.000000 (17592186044191.740MB) at ../../../model_brush.c:2996)
DarkPlaces-Quake Linux, buildstring: v20140513-1451-gd7763f6f Jul 10 2024 release
Client "player" dropped (Server aborted!)

There is also a very strange physics bug in the start.bsp elevator where the player will bounce and and lose a lot of life by the time they get to the bottom.

[bones-was-here]

The most obvious issue is signed integer overflow, easily fixed with a (size_t) cast, at which point the real problem appears: DP's PVS (at least for Q1BSP) uses several orders of magnitude more memory than vkquake's.

Adding a debug print to vkquake's Mod_DecompressVis() reveals it allocates 102348 bytes for numleafs: 818760.
DP's Mod_Q1BSP_LoadLeafs() allocates 51732336555 bytes (~49GB) for num_leafs: 818760.
DP's Mod_Q1BSP_LoadLeafs() has an allocation multiplier that other engines don't seem to need, introduced in f2fbe3c.
43c93c6 may also be relevant.
@ladyhavoc could you shed some light please on why this is used (and how it can be avoided)?

[hemebond]

Also affects the newly released Meat Jam

49 memory pools, totalling 3055927274 bytes (2914.359MB)
total allocated size: 3058726990 bytes (2917.029MB)
Memory pool 0x55bdcdcbbd30 has sprung a leak totalling 480125809 bytes (457.884MB)!  Listing contents...
 480123289 bytes allocated at ../../../fs.c:3514
      1024 bytes allocated at ../../../image.c:1890
       736 bytes allocated at ../../../cmd.c:1463
       736 bytes allocated at ../../../cmd.c:1463
        24 bytes allocated at ../../../cmd.c:1500
Engine Aborted: Mem_Alloc: out of memory (alloc of size 18446744073525950464.000000 (17592186044240.904MB) at ../../../model_brush.c:3011)
DarkPlaces-Quake Linux v20140513-1537-g2f25af65~ Aug 11 2024 release GCC

[ladyhavoc]

Decompressing PVS data was done to simplify support for q3bsp which doesn't use compressed pvs, and to make it faster to check pvs in several places per frame (e.g. for dynamic lights), that said in any case that uses the majority of the data it is probably faster to decompress on access because of memory bandwidth being lower that way. I'm curious if these maps are legitimately using an absolutely gigantic number of leafs or if they're just sitting on a lot of blank space in the PVS data. Reusing the same PVS data for multiple leafs when loading q1bsp/q2bsp leafs would be an option even if it's uncompressed, but I had not encountered maps like this when I made that decision.