Prime cache with message bodies through background job

[ChristophWurst]

Mail doesn't go to IMAP to fetch a list of messages, but we have to contact IMAP for the bodies. This can cause a slight delay when users browser their messages. To improve things we already have mem caching (through Horde) that is supposed to prevent the same message to be fetched twice. However, if the first fetch is triggered by the user, the user will experience the delay.

Therefore I propose that we try to prime the cache with message bodies from the cron background jobs, so that the most recent email (or all new?) are put into the cache.

Of course we might overfetch or underfetch, but that is inevitable. If it helps reduce loading times for 50% of the messages the user will browser then we've already won.

From a technical perspective we have to possibly check what type of cache is used as prefetching and caching on a cache that is not shared between CLI (cron) and web requests won't give any benefit.

[TtuxX]

This is a really great idea!

I personally badly need it as the loading time on my instance is still not as fast as Rainloop.
I get between about 6 to 10 seconds loading time per message on NC Mail VS 2 seconds via Rainloop on the same instance.

It slows me down from switching completely to the Nextcloud Mail app for the day-to-day work.

[ChristophWurst]

We'll work on this when time allows.

[dodedodo]

Memcache would be very nice to have. Like tux, I also encounter long loading times as a result of a slow IMAP server.

Should've done a search. This comment is a duplicate of #2621

Would a database cache instead of a memory cache be better suited for this?

In other words; download every imap message body locally on sync or cron. We could load the db-cached body straightaway, and send the IMAP request in the background in case the message body has changed on the server. (although I can't think of a scenario where a received message would change.)

Or would this be re-inventing pop3?

[st3iny]

I did some testing and it turns out that the builtin cache of horde does not seem to cache full message bodies. There was no measurable difference between loading bodies of cached and uncached messages. I suggest to implement a custom cache for message bodies if we really want to do this.

[ChristophWurst]

If it's not worth it then let's put this ticket on hold for now.

[atezet]

I did some testing and it turns out that the builtin cache of horde does not seem to cache full message bodies. There was no measurable difference between loading bodies of cached and uncached messages. I suggest to implement a custom cache for message bodies if we really want to do this.

@st3iny Wondering how you tested this, is there a branch I can use? I'm trying out the Mail app now and fetching mail from IMAP the first time can take from 5 up to 10 seconds on our setup. After that, it's less than a second.

Pre-fetching some emails (like the 100 most recent, or the 10 most recent of the current mailbox, or something else) could greatly improve the experience I think.

[st3iny]

I did some testing and it turns out that the builtin cache of horde does not seem to cache full message bodies. There was no measurable difference between loading bodies of cached and uncached messages. I suggest to implement a custom cache for message bodies if we really want to do this.

@st3iny Wondering how you tested this, is there a branch I can use? I'm trying out the Mail app now and fetching mail from IMAP the first time can take from 5 up to 10 seconds on our setup. After that, it's less than a second.

Pre-fetching some emails (like the 100 most recent, or the 10 most recent of the current mailbox, or something else) could greatly improve the experience I think.

IIRC, horde (imap lib we use) only caches message metadata but not the body itself. Therefore, priming the cache only benefits performance if we don't fetch the body afterwards. However, as we are fetching the body to be able to display it to the user we have to a full round trip to the imap server anyways => there won't be a measurable difference.

If we really want to improve the performance on that end, we need to implement a custom message body cache. We could for example cache the bodies of the most relevant/recent messages in the database.

[atezet]

I see. So the body is stored on the client side, and therefore way faster after the first load? What do you think about pre-fetching a bunch of emails when you open a mailbox?

A custom message body cache (or implementing it in horde?) sounds like a more durable option, but also a much bigger change.

[st3iny]

I see. So the body is stored on the client side, and therefore way faster after the first load?

Exactly. Once the email is opened on the client it is cached by the browser.

What do you think about pre-fetching a bunch of emails when you open a mailbox?

On the client-side? Sure, that would be a possibility but it would generate a lot of traffic. It could be improved by only fetching mails that are not cached on the client although I'm not sure if this is possible via JavaScript.

A custom message body cache (or implementing it in horde?) sounds like a more durable option, but also a much bigger change.

True. We discussed it internally and it might be implemented at some time.

[atezet]

I had a quick look at the behaviour and I noticed the following. When I click on a message the following happens:

1. /api/messages/{id}/thread is called - this returns the e-mail headers, which is quite fast (because it's from cache, I suppose)
2. /api/messages/{id}/body is called - this returns the e-mail body, which takes a few seconds (up to 8 seconds in my testing), as bodies always need to be fetched over IMAP
3. (if body[hasHtmlBody] then) /api/messages/{id}/html is called, which takes the same time as 2., as there's no cache

EDIT: the third step is initiated by the iframe displaying the email

[metorm]

I put my vote to this feature.
Public mail servers are usually slow while private nextcloud servers are much faster.
Pre fetching the content makes us feeling like working on local clients.