RangeError: Array buffer allocation failed

[KyleMaas]

Running into an issue in ssb-browser-demo which seems to be stemming from this line:

https://github.com/ssb-ngi-pointer/jitdb/blob/c143df20db1f08a74f1a7cf54bb9c7058a394865/index.js#L186

bundle-ui.js:73176 Uncaught (in promise) RangeError: Array buffer allocation failed
    at new ArrayBuffer (<anonymous>)
    at new Uint32Array (<anonymous>)
    at growTarrIndex (bundle-ui.js:73176)
    at updateSeqIndex (bundle-ui.js:73184)
    at write (bundle-ui.js:73346)
    at Object.o.write (bundle-ui.js:115586)
    at Stream._writeToSink (bundle-ui.js:8280)
    at Stream._handleBlock (bundle-ui.js:8313)
    at bundle-ui.js:8355
    at Object.getBlock (bundle-ui.js:7449)

(See the initial report at arj03/ssb-browser-demo#90 )

My system's nowhere near 100% RAM consumption, so I did a bit of experimenting. On Firefox I get the following results:

var x = new ArrayBuffer(1.9 * 1024 * 1024 * 1024)
undefined
var x = new ArrayBuffer(2 * 1024 * 1024 * 1024)
Uncaught RangeError: invalid array length
    <anonymous> debugger eval code:1
var x = new ArrayBuffer(2 * 1000 * 1000 * 1000)
undefined

So there's a rough 2GB limit (<2GiB). If I do the same on Chromium:

var x = new ArrayBuffer(2 * 1000 * 1000 * 1000)
undefined
var x = new ArrayBuffer(2 * 1024 * 1024 * 1024)
VM279:1 Uncaught RangeError: Array buffer allocation failed
    at new ArrayBuffer (<anonymous>)
    at <anonymous>:1:9

Pretty similar results.

[KyleMaas]

Limit appears to be even lower for larger types. On Chromium:

var x = new Uint32Array(0.5 * 1000 * 1000 * 1000)
undefined
var x = new Uint32Array(0.5 * 1024 * 1024 * 1024)
VM849:1 Uncaught RangeError: Array buffer allocation failed
    at new ArrayBuffer (<anonymous>)
    at new Uint32Array (<anonymous>)
    at <anonymous>:1:9

So we're still at about a 2GB allocation limit (4 bytes per entry).

[KyleMaas]

Additional debugging. I can consume more memory if I break it into smaller allocations:

var a = new ArrayBuffer(2 * 1000 * 1000 * 1000)
undefined
var b = new ArrayBuffer(2 * 1000 * 1000 * 1000)
undefined
var c = new ArrayBuffer(2 * 1000 * 1000 * 1000)
undefined
var d = new ArrayBuffer(2 * 1000 * 1000 * 1000)
undefined
var e = new ArrayBuffer(2 * 1000 * 1000 * 1000)
VM41:1 Uncaught RangeError: Array buffer allocation failed
    at new ArrayBuffer (<anonymous>)
    at <anonymous>:1:9

[KyleMaas]

Any progress on this? It rather severely limits the maximum size of the data that you can have in the database if you're running in a browser (like in ssb-browser-demo).