Cache resources through localStorage/indexedDB on the backend side

[mossroy]

This would be an alternative to #414.
@Jaifroid has already done some hard work in branch https://github.com/kiwix/kiwix-js/tree/Persist-cache-in-localStorage

We have to decide if we prefer to implement this or #414.
An advantage for this solution is that it could probably be made to work the same for both jQuery and ServiceWorker modes.

[mossroy]

On the other hand, there are some drawbacks :

• the cached content would have to go through app.js, instead of being directly retrieved by the ServiceWorker

And some things to pay attention to :

• I think we need to refactor the code, so that this cache is handled by an extra layer between app.js and the ZIM backend. It would make things less complicated in app.js, while allowing to replace our current ZIM backend with something compiled from libzim (ZIM backend : Try to cross-compile the libZIM library in webassembly with emscripten #116). It would also allow to use the same cache for any kind of content (and to remove the existing CSS cache implementation)
• the branch code is perfect for comparing different cache implementations (memory, localstorage, indexeddb). But, if we want to be able to cache binary objects (like images), I suppose indexeddb is more suitable. https://caniuse.com/#search=indexeddb seems to say that there are only a few limitations in Microsoft browsers : it might be worth a try. If not, something like https://hacks.mozilla.org/2014/02/localforage-offline-storage-improved/ might help us having only one cross-browser implementation. There are interesting pointers for the cache topic in https://developers.google.com/web/fundamentals/instant-and-offline/web-storage/offline-for-pwa

And things to do, that are the same if we choose #414 :

• we need to choose and implement a cache retention strategy, so that the cache does not grow up forever. I suppose we should not try to reinvent the wheel. It looks like IndexedDB and the Cache API of ServiceWorkers implement it for us, if we use a "temporary" storage. See https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Browser_storage_limits_and_eviction_criteria
• we also need to be able to separate caches coming from different ZIM files. Each ZIM file should have its own cache

[Jaifroid]

I think we need to refactor the code, so that this cache is handled by an extra layer between app.js and the ZIM backend

Do you mean something different from cache.js, which is the current cache layer I'm using? I tried to put most of the code in there, but what remains in app.js are mostly the intercept hooks. Maybe more of the code can be pushed to cache.js?

the branch code is perfect for comparing different cache implementations (memory, localstorage, indexeddb)

When I was developing this, Firefox didn't support indexedDB from the file:/// protocol. However, looking at it now, it seems that it does. I even put a comment in the code about this! It's actually the main reason I made the cache support localStorage as a fallback. Seems that's been fixed recently in FF. Internet Explorer supports indexedDB only in localhost mode, but doesn't support any cache protocol in file:// protocol, hence the fallback to memory cache.

But, if we want to be able to cache binary objects (like images), I suppose indexeddb is more suitable

Indeed, localStorage can't, AFAIK, store BLOBs. But would we really want to cache images? It would only be useful if someone is reloading the same page. The only objects I can foresee us needing to cache are stylesheets and JavaScript, because these are likely to be the same ZIM-wide.

we need to choose and implement a cache retention strategy, so that the cache does not grow up forever

The cache only supports CSS currently and a copy of the HTML retrieved from the ZIM for the current page. The latter is overwritten for every page visited. CSS is the same ZIM-wide, so we'd want to retain indefinitely, until the user empties the cache on purpose. User can empty the cache using the radio button, then all elements are cleared.

we also need to be able to separate caches coming from different ZIM files

They are currently separated by a database key that is specific to each ZIM. Even if the CSS file is the same in two different ZIMs, it will be stored under its own ZIM-specific key. You can inspect the content of the cache using the F12 tools Here is an example from Edge:

[Jaifroid]

I'd like to keep this issue open for one main reason: as mentioned, I need to implement (have already implemented in another branch) indexedDB caching for jQuery mode, to use in Kiwix JS Windows. I want to implement it in a way that fits fully with the Kiwix JS codebase, and that needs me to make it work on current master with #556. Once that is working properly in a Kiwix JS branch, I'll close this issue. I'm not expecting that what I implement will be merged in Kiwix JS.

[Jaifroid]

Just to say that I have ported all changes in #556, and all latest changes to master to the branch Persist-cache-in-localStorage, and am therefore closing this issue. To be clear, this has been implemented as a basis for a Version 2.0 of Kiwix JS Windows (Version 1.0 is close, and will use the existing codebase). I do not intend to implement this in Kiwix JS.

For info, this branch implements a similar caching strategy to #556 in jQuery mode, using a separate module called cache.js. In Service Worker mode, it uses the same caching strategy as is provided by service-worker.js. It works in the following way:

• If CacheAPI is available, it will be used, even if the browser is in jQuery mode (see screenshot below);
• If no CacheAPI is available, the module will try to use IndexedDB to cache requests and responses;
• If IndexedDB is unavailable, then the module will try to use LocalStorage;
• If LocalStorage is unavailable, then the module defaults to the standard Memory Cache.

Included in this implementation is extraction and running of JavaScript in jQuery mode. It is able to run simple JS, and can also run PhET experiments, but it cannot run React JS interfaces such as those provided by Gutenberg ZIMs.