Folder/Message handling: current problems & further ideas

[ChristophWurst]

This ticket is only meant to document my thought on the problems and potential solutions for the folder and message handling we have. This should give other developers an overview of what I'm planning to work on in the near future.

Message synchronization

Currently, we have two message synchronization mechanisms. First, Loading and reloading a folder's messages. Second, the background fetch timer that pulls in new messages.

(Re)load messages

Messages of a specific folder are loaded in pages. The client sends from=0 and to=20 to get the first page, the initial data. Whenever the folder should be reloaded, we fire the same request, throw away all existing messages and replace them with the newly fetched ones.

This raises the following problems:

• The message list is shortened, e.g. when you scrolled down and refreshed, you're forced to start over again.
• All data of those 21 messages is fetched, even if it does not change.

Possible countermeasures include:

• Use horde sync. It's based on uid next and uid validity. It more or less gives you a diff of the current folder state, e.g. which messages have been added, which ones have been deleted. This allows an efficient data sync, without having to neither re-fetch redundant data, nor discard local pages.

Background sync

New messages of all inboxes are queried each five minutes. Technically, this implementation is pretty efficient, as it's based on the same technique as horde sync. However, the implementation has a little but significant issue: uid next and uid validity aren't updated whenever the user reloads the folder, only when the background check is performed. This means in certain situations the background check might detect changes the client already knew, resulting in wrong/duplicate notifications.

Possible countermeasures include:

• Merge this functionality with the other folder synchronization above, using horde sync. This was it's efficient and we can probably speed up the client as the data structures can be leveraged.

Pagination

As mentioned above, we load messages of a folder as pages. Pagination is great and needed, but it's also implemented wrongly for this use case. The problem here is that the data on the IMAP server is likely to change often, therefore we run into inconsistency issues quickly. The main problem is that the client only tells the server the offset and the size of the page. The server, however, has a different state than the client, resulting in wrong pages being returned. Depending on how the state diverges, pages either overlap (redundant data transfer) or some messages are ignored.

Possible countermeasures include:

• Cursor-based pagination. This is IMO the only reasonable solution for pagination here. It basically tells the server the (U)ID of the last known message, which the server can then use to build the next page. See this link for more information on that topic.

Unified folders

We do have a unified inbox at the moment, which is a killer feature I use all the time. Technically, however, we do have some problems. Implementation-wise the client doesn't know too much about this special folder, as most of the handling was baked into the server-side implementation. To make that happen, some really nasty hacks were used, e.g. the message id (which is numeric usually) is a base64 encoded json string containing the original account id, folder id and message id of a message in the unified inbox. The past few months have shown that the way it's implemented now is fragile, leading to many bugs and an instable app.

Performance wise, the server-side implementation is also counter-productive as we cannot do any kind of parallelization as PHP doesn't allow it. Thus the unified inbox scales very badly because you'll have to ask n servers sequentially, meaning the request takes n times longer to finish (n being the number of accounts).

Possible countermeasures include:

• Rewrite the unified inbox on the client-side, the server doesn't need to know anything about it.
  • I did a POC of an multi-source fetch algorithm and it seemed to work nicely. See this repo for more info. This way, we can send n requests in parallel. The server will work on them individually, resulting in an much faster and more scalable architecture. In theory, adding more accounts won't slow down checking your messages.

As always, I'm very happy about input from other devs. @nextcloud/mail & others let me know if that also makes sense to you 😉

[ChristophWurst]

For the most part the ideas in this ticket were implemented in #326, hence I'm closing this now :)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and questions.