Handling bookmarks from previous zim files

[BPerlakiH]

If we update a ZIM file, the bookmarks created formerly remains linked to to old ZIM file, and stop working.
In case of custom apps, this means losing all bookmarks at each update.

Create a plan how we can improve this on the iOS/macOS side.

Depends on kiwix/libkiwix#1037

[kelson42]

@BPerlakiH Please save ASAP (and update DWDS) app, the ZIM name in your bookmarks. Once libkiwix ready, implementing this ticket will imply relying on the libkiwix for that AND create a migration plan.

[BPerlakiH]

Bookmarks

The problem

If a newer version of a zim file is added to an app the bookmarks created with the old zim file won't work anymore.

How the bookmarks are working on iOS/macOS?

It is an entity in the local database, based on the articleURL, eg:
kiwix://A992BF76-CA94-6B60-A762-9B5BC89B5BBF/wb/Saftladen
or
kiwix://A992BF76-CA94-6B60-A762-9B5BC89B5BBF/wb/08_.15
where kiwix is our custom namespace
the A992BF76-CA94-6B60-A762-9B5BC89B5BBF is the fileID of the ZimFile (also stored as an entity/table).

The fileID is comming from the zim file metadata itself, and it is not changing. It has been tested:

• across devices
• with a fresh install
• even with renaming the file itself, and re-adding to the project

The Bookmark entity is also referencing the ZimFile in the DB via a unique (auto-incremented) primary key.

Custom apps problem

Our current solution is enforcing to always use a single file per custom app. It is imported into the DB on launching the app.

When we update the zim file and create a new app release, the new zim file will be inserted into the DB next to the old one.

The problem we are facing is that the bookmarks will still point to the old file, both in referencing the primary key of the old file (which is no longer loaded),
and their url will contain the old value as the first part of their path (see above).

If there's a new app release, but the zim file (it's fileID) remains the same, the DB would not change, and the bookmarks should work as before.

Custom apps solution

In case of an update, we can assume that if a new file is imported, the old entries in the DB should be updated.

The following solution has been tested (by manually editing the DB):

1. change the old bookmarks url, by replacing the first part of the path to the value comming from the fileID of the new zim file, eg changing the one above to:
   kiwix://64C3EA1A-5161-2B94-1F50-606DA5EC0035/wb/Saftladen
2. update the old bookmarks zim file primary key reference, so they are pointing to the new file inserted, eg changing primary key from 1 to 2.

Alternatively, since we will only want to use 1 zim file, we could also replace the old zim file in the DB, keeping the bookmark primary key references to the same entry.

Kiwix app solution

When it comes to zimfiles fetched from our catalog / downloaded by the user, we have even more data available, at the same time we need to support multiple zim files and multiple bookmarks. The Kiwix application starts with around ~3000 zim files, non of them being downloaded (physically available on disk).

The user can select and download one of those zim files. They might have different flavours, depending on their content, eg based on language of the content and/or cut down versions such as "nopic" (no pictures included to get a smaller size).

To identify a ZimFile and eventually apply an update to it, we have the following information at hand:

• primary key of the zim file (auto incremented in the DB)
• the download url eg: https://download.kiwix.org/zim/other/t4-wiki_de_all_nopic_2022-12.zim.meta4
• the file system path (if it was downloaded), eg: "file:///Users/user/Library/Developer/CoreSimulator/Devices/43204F1F-9C78-407C-9288-BA978ECFDF36/data/Containers/Data/Application/27D8B5FB-3D76-4E72-848A-28A8BD731249/Documents/t4-wiki_de_all_nopic_2022-12.zim"
• the persistentID eg: t4-wiki_de_all

Based on the above we should be able to determine if the newly fetched / downloaded file is a successor of an older DB entry, and apply similar changes as in case of a custom app.

Assumptions

We are assuming that the bookmarks created both in the old file and the new file will have the same articleURL "ending", like so:

• kiwix://64C3EA1A-5161-2B94-1F50-606DA5EC0035/wb/Saftladen
• kiwix://A992BF76-CA94-6B60-A762-9B5BC89B5BBF/wb/Saftladen
  Currently this should be true for most of the cases. Therefore such bookmarks can be migrated from one version of the zim file to another. If for some reason the articleURL of a page will completely change in the newer version of a zim file compared to the old one (or won't be available in the new one), the end user will be alerted, that the page cannot be opened. In these rare cases the user can manually delete the old bookmark, search for the page in the new zim file and create a new bookmark.

TLDR

Currently we have enough data collected and available already to apply a bookmark fixing solution in our future releases, therefore we do not need any additional preparation steps carried out right now.

[rgaudin]

• the persistentID eg: t4-wiki_de_all

For clarity's sake, I'll add some details with openZIM wording:

• a ZIM file has a single ID set by libzim at creation. That's the UUID we see above.
• ZIM spec requires some basic metadata which includes (among others): Name, Flavour, Publisher
• The string t4wiki_de_all could be the Name metadata of that ZIM or part of its filename. Filename should not be relied on.
• It's important to acknowledge that neither the ID or the Name is presented in any reader's UI so from a user's perspective there is only a Title, a Description and an Icon. This leaves room for user confusion regardless of the technical implementation.

In other part of the project, we use a combination of those metadata (as {Publisher}:{Name}:{Flavour} to identify a ZIM Content over time. It might be wise to reuse it.

Questions:

• If we rely on some non-UUID identifier to match ZIMs to retrieve bookmarks, wouldn't it be more efficient to store the bookmarks using this identifier instead of the UUID?
• Do we want bookmarks to work accros flavors? If I bookmark on wikipedia_en_all_nopic and also have wikipedia_en_all_maxi, would it use the same bookmarks?

[rgaudin]

@kelson42 there's a question here about whether to await libkiwix 14 and handle migration via libkiwix or implement a similar swift-only fix in the mean time.

At the moment, apple app doesn't use the Bookmark feature of libkiwix at all.

[kelson42]

@rgaudin @BPerlakiH Sorry if it was unclear. The fix of this issue is blocked by kiwix/libkiwix#1043 which is itself last blocker for libkiwix14 release. That said, I hope the fix can already be developed once the fix is in libkiwix main branch.

[BPerlakiH]

@kelson42 I just re-read this ticket. Can you please explain which part of this process (from my list above) is meant to be covered by libkiwix ?
Is it part the part of comparing the zim files (which one is the new) or is it a complete change how we should store and read the bookmarks?

[kelson42]

@BPerlakiH Not absolutly sure to fully understand your question but the full bookmark handling is provided by the libkiwix. Regarding update through file update, there is still a bit of discussion how it will look like. But the migration from current system to libkiwix based one is the responsability of the Swift code.

Maybe this ticket can be done in two steps: move to libkiwix handled bookmarks and then integrate bookmark updates through ZIM versions?

[kelson42]

@BPerlakiH Upstream ticket has been implemented and is available in libkiwix nightly. No blocker anymore. If you face difficulties with libkiwix API, please request support of @mgautierfr.

[BPerlakiH]

@kelson42 @mgautierfr please see my observations in #679.
Maybe I am missing something here, but at the moment it very much seems to me the route of using libkiwix for Bookmarks is not the ideal one to be taken at the moment on Apple platform.

[rgaudin]

Thank you for all the details in the PR. I have the feeling we're not clear on the objectives.
This ticket is related to the migration/reconciliation of bookmarks from one version of a ZIM/Book to another one.
Because we don't want to re-implement that logic in every reader, it has been implemented in libkiwix.

I recall asking about this: do we want to handle everything-bookmark with libkiwix or do we only want to do the bookmark-reconciliation (if it's possible) ?
@kelson42 said we want the full thing but maybe that was overlooked.
That PR should allow us to reevaluate. I think the solution will be hybrid anyway if we want to keep the features (snippet for instance).

[mgautierfr]

Is it fixed by #688 ?

[kelson42]

Yes, but not really in the right way (using the libkiwix). Main reason why this is still open.