Use IndexedDB as cache storage on Chromium platform

[gorhill]

Currently, IndexedDB is used only on Firefox platform to implement cache storage. Chromium uses extension API storage (chrome.storage.local) as a cache storage, i.e. cache items are sharing the same physical storage as user setting ones.

A benefit of using IndexedDB is that the items stored in it are LZ4-compressable (which is enabled by default for that storage). LZ4 compression is not trivially possible with extension API storage, as this requires the storage to support the saving of Blob resources. (An approach would be to create a special encoding/decoding step, but that is just not practical.)

There is no technical obstacle to also use IndexedDB on Chromium. The main obstacle of doing so is that there is no code to automatically transfer cached items from one storage to the other, which would cause stale cache items to linger in the old storage should there be a switch to a new one for cache purpose.

Provide code to move cache items stored in chrome.storage.local to IndexedDB when uBO launches and detect that IndexedDB is being used as cache storage.

[gwarser]

Isn't extension storage already compressed natively in Chromium? Files in Local Extension Settings/cjpalhdlnbpafiamejdnhcphjbkeiagm/ are already small and not compressible.

---

Snappy https://en.wikipedia.org/wiki/LevelDB#Features
and "LevelDB is widely noted for being unreliable and databases it manages are prone to corruption."

[gorhill]

Could be.

However I found that with IndexedDB (seems to be at IndexedDB/chrome-extension_[id]_0.indexeddb.leveldb on my side) I could definitely see less issues with garbage collection after loading filter lists or selfie.

With a valid selfie, when using chrome.storage.local, the baseline memory seems stuck at a higher level than expected. I found that with IndexedDB, the baseline memory usage at launch quickly snaps well under 40 MB on my side, something I am not able to get with chrome.storage.local. With a valid selfie (default lists), uBO's baseline memory usage at launch is now lower than it ever has been I am pretty sure.

I can only speculate but it seems the browser will keep entries from chrome.storage.local in memory more aggressively -- even forcing garbage collection in dev tools does not help, while this does not happen with IndexedDB. There is no point to keep cache-related entries in memory once they are loaded.

[uBlock-user]

My storage usage on Chromium on 1.17.3rc5 was 27 MB and now with the upgrade to 1.17.5b0 it went down to 10.3 MB, WOW! Is this because of IndexedDB being used now ?

[gwarser]

Measured by uBO?

[uBlock-user]

Yes, the one listed in settings.html ?

[gwarser]

On disk for all non-regional list both use ~17MB (needs some time to stabilize and restart to remove overgrown 000*.log)

[uBlock-user]

what do you mean ?

[uBlock-user]

@gorhill The storage value seems to be jumping back and forth on my end. Once I see 10 MB in use and then after few minutes/hours, it's 17 MB in use, is this to be expected ?

[gorhill]

Probably uBO created a selfie, used for fast loading next time uBO is launched. The selfie is destroyed whenever a filter list is updated, and it will re-create itself after ~17 minutes following the reload of all filter lists.

[uBlock-user]

next time uBO is launched.

So for future use ? Thats odd as I don't close Chromium for hours together(not disabling uBO either) and this happens randomly.

[gwarser]

what do you mean ?

Both stable and dev uses comparable amount of disk storage (after stabilisation). Stable uses more initially, releases excess after restart (very lazy, can take hours).

storage.local probably uses *.log files for journaling, and clean this file after long delay. Other files are also keept.

[uBlock-user]

Stable uses more initially, releases excess after restart (very lazy, can take hours).

and what about dev builds as I'm on that ?

[gwarser]

Check IndexedDB/chrome-extension_cgbcahbpdhpcegmbfconppldiemgcoii_0.indexeddb.* folders in your Chromium profile.

[uBlock-user]

I did, I see lots of files inside the first folder, the last two have files name LOCK and other files which are created by chromium I suppose.

[gorhill]

I've run a lot of benchmarks and measurements. Here is the summary: IndexedDB + LZ4 is beneficial to uBO in Chromium, even if whatever is stored on disk is independently compressed by Chromium itself. So this will stay.

If ever there is a demonstration that the overhead of LZ4 compression uBO-side is questionable, I can always trivially add code to turn off compression while keeping IndexedDB.

I used the advanced setting cacheStorageCompression to turn off compression for my measurements. When toggling this setting, it is important to force an update of all filter lists.

Two screenshots of performance benchmark using dev tools to measure launch timings.

No selfie, IndexedDB + compression:

No selfie, IndexedDB + no compression:

Now it seems the "IndexedDB + compression" did better, but we shouldn't draw such conclusion given the small difference. What is important is to be able to conclude that it does not appear using compression adds undue overhead.

Now, memory baseline at launch time (i.e. memory usage after garbage collection, I waited 5 minutes before screenshots).

uBO 1.17.4: selfie + 2nd-gen hntrie + chrome.storage.local:

uBO dev: selfie + 3rd-gen hntrie + chrome.storage.local + no compression:

uBO dev: selfie + 3rd-gen hntrie + IndexedDB + compression:

I tried many different combos and I conclude that what actually reduces baseline memory usage is uBO-side compression (hopefully I didn't lose myself in all the measurements I ran). If I disable compression but still use IndexedDB, I get the 50 MB baseline. Forcing garbage collection in dev tools accomplishes nothing. I don't know the explanation of this but this is what I have observed.

Thus, given that there is seemingly no undue overhead incurred as a result of using compression, and given that the baseline memory usage is much better when using compression, and given that actually reporting a more accurate on-disk bytes used to users (assuming Chromium does compress itself) is desirable, I will keep compression enabled.

[gwarser]

I just discovered (reddit) that Firefox also have compression in indexedDB:

https://hg.mozilla.org/releases/mozilla-release/annotate/0a91847e50e6/dom/indexedDB/ActorsParent.cpp#l1441

But when it's used and for what type of data??