Force pre-fetch (aka mirror) complete IMAP server-content (account)

[PrivatePuffin]

Feature Request

Simply put:
Even with caching, searching extremely big mailboxes on remote IMAP servers is a chore.

Make it like a normal client like outlook and download-all-the-things.
Enable us (admins) to fetch a complete copy of the IMAP server (account) and store it locally for fast searches.

This prevents issues with big IMAP servers on slower connections and timeouts. (see below for a usecase).

Summary

it might be worthwhile to add button/feature to force download the whole imap account. Because there is a case where querying an IMAP server isn't feasable:

IMAP server is external
Pipe to the imap server isn't too big (lets say 10MB/s)
IMAP server contains GB's (lets say 4GB) of small emails
Even preloading all email from the server takes a few hours, but queries to the IMAP server might take a LONG time as well. (minutes)

If we add to this the following situation:

Using a reverse proxy or other execution time limiter
We end up with 504 A LOT, because the proxy or other limiter isn't going to accept minutes of wait time to finish such a query.

If we, as admin, could force Nextcloud to preload the whole IMAP account, this would be sped up even byond this PR and should be relatively easy to add even within scope of this PR.

Implementation proposal

Considering #2064 most of the backend to store these large(ish) amounts of mail are already there. It just needs a big fat button and some stitches it seems.

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[PrivatePuffin]

*note:
Or I am completely mistaken and the initial sync in #2064 already covers this.

[ChristophWurst]

We store all relevant headers (the envelopes) but not the structure (the body)

[PrivatePuffin]

Interesting, so the primary difference between this and what (for example) outlook does is cachine the body.

That would be more complex than I thought. None the less an interesting idea.

[ChristophWurst]

Interesting, so the primary difference between this and what (for example) outlook does is cachine the body.

Correct. And I would refrain from caching the bodies in the database. This would bloat it unnecessarily. Instead, we should try to use the memcache as much as possible and if feasible prepopulate it in a cron job so data is loaded faster when the user requests it.

[PrivatePuffin]

@ChristophWurst Wildcard queries on big portions of text in many-row tables doesn't seem the best idea indeed, mysql seems to be quite inefficient used that way.

Prepopulate the memcache is a great idea, till you hit GB's of emails.
What you could try is keeping the body it in memcache, but write the memcache chances to disk every once in a while.

That way the user can reload most of the cache from disk after (for example) a reboot, instead of IMAP.

[ChristophWurst]

The simplest strategy would just be to cache messages as they come in and when the user visits them individually. They are immutable, so caching is quite simple to do.

[PrivatePuffin]

The problem is search:
I need to find "Email mentioning John" quite often in 1 3+GB archive.

Your solution would work great, IF i would've recieved all emails mentioning John during that nextcloud/cache session. Which would obviously often not be the case.

So that would require me to prefetsh the whole 3GB every boot. Which wouldn't be an issue with the IMAP server locally, but with an off-site IMAP server even this is troublesome.

Thats why I like the way Outlook prefetches everything including body, it enables search as if I was working on the server locally.

[ChristophWurst]

Please see https://help.nextcloud.com/t/call-for-testers-mail-v1-3/75825. We'd appreciate if you could help us test the upcoming release ✌️

[kklem0]

Great work!! @ChristophWurst
This makes the mail app SO much better!

May I suggest to cache email body like iPhone does?
You (maybe admin or user) can choose to only cache for emails within a period of time.
iPhone has the option "No Limit", "1 Day", "3 Days", "1 Week", "2 Weeks", "1 Month".

For me it would make perfect sense to cache emails (or entire Thread if it has 1 new email) within a week for how I work, maybe 2 weeks when I'm on holidays. And I don't think I've ever read anything that's more than 90 days.

This way you can avoid bloated DB?
Older emails would take longer to show but I don't think that would be a big deal for most operations.

What do you guys think?

#317

[kklem0]

I also noticed that preview text is null or empty string in the db while it's set for 255 characters, is that the correct behavior?

[ChristophWurst]

I also noticed that preview text is null or empty string in the db while it's set for 255 characters, is that the correct behavior?

I worked on that for #2628 but I couldn't find an easy way to extract the subject from HTML emails, hence this is unused right now. We want to show the preview text here #2666

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.