Experimenting with an LMDB-backed Array(like) implementation

[KyleAMathews]

The idea being that LMDB can free up memory for huge arrays of data which would otherwise OOM the process (or need a really large machine).

We have some Gatsby sites that can put 10GB+ of memory in arrays.

The idea is that LMDB is fast enough that the tradeoff of lower memory for slower iteration speed won't be so bad.

This is all grand in theory — the only problem I'm having so far is my test code so far isn't freeing up memory so it OOMs a bit earlier than a normal JS array would.

My benchmark code reads in objects from an LMDB db (up to 1 million) & then pushes them onto the lmdb-backed array.

This is what the push method looks like:

  push(element: any) {
    const index = this._nextKey
    this.db.put(index, element)
    this._nextKey++
    this.length++
    return this.length
  }

I assume/hope I'm just holding onto memory somewhere inadvertently? Would appreciate some quick looks through the code.

• array class https://github.com/KyleAMathews/lmdb-backed-array/blob/main/lib/index.ts
• benchmark https://github.com/KyleAMathews/lmdb-backed-array/blob/main/benchmark/index.mjs

[kriszyp]

When I run the lmdb-array benchmark, it seems to run with very little stress on the heap at all:

cpu: 298.6% | rss: 8.6 GB (25.7%) | heap: 7.7 MB / 44 MB (17.3%) | ext: 625 MB | delay: 670 ms | 00:00:15 | loadavg: 1.25, 1.07, 0.98
flushed: 263.445ms
{ chunkI: 18 }
cpu: 304.0% | rss: 8.9 GB (26.8%) | heap: 7.6 MB / 45 MB (17.1%) | ext: 627 MB | delay: 690 ms | 00:00:16 | loadavg: 1.25, 1.07, 0.98
flushed: 277.58ms
{ chunkI: 19 }

real	0m16.808s
user	0m46.741s
sys	0m6.263s

This was on node v16 and v18 (on ubuntu). What version of node are you running? I wonder if I am doing something wrong? (it does indeed OOM on the other benchmarks).

[KyleAMathews]

But disabling the LRU expirer means you can't get inflight puts again 😬😂

[KyleAMathews]

is it possible the weak-lru-cache is just not releasing memory somehow? I can't get it to change its behavior (i.e. release memory) regardless of what I do e.g. reducing cacheSize, reducing cleanupInterval — I added logging to the exprierer and verified it is doing something.

[kriszyp]

I think this should be fixed in v2.7.9, if you want to try that.

[KyleAMathews]

that works! Now as soon as the data is flushed, the memory is cleared 🙏

haha — even if nothing else happens with this project, I feel very good about helping get two bugs get found & fixed!

This is looking very cool now — it's only 1.2-2x slower or so and uses far far less memory 💥

[kriszyp]

Good to hear! And thank you for the detailed repo for reproducing this, that was super helpful!

[kriszyp]

BTW, one other thing I was going to mention is that you can actually open an lmdb store without a path, and will do almost exactly what you are doing, create a store with a random name in the OS temp directory (and will also open it with noSync and auto-delete it on close). I know that is peripheral to the main issue of memory, but perhaps a nice convenience.

[KyleAMathews]

oh nifty! Yeah that's perfect. useWritemap & noMemInit look useful too.