Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a cache for IMAP message in the database #2064

Merged
merged 1 commit into from
Feb 3, 2020

Conversation

ChristophWurst
Copy link
Member

@ChristophWurst ChristophWurst commented Oct 2, 2019

This is something I've been wanting to do for a long time. So here it is. Or, well, just a few modifications towards a new architecture I would like to have with this app.

The problem

Right now this app operates on IMAP directly. Hence the performance and capabilities completely depend on the IMAP server and its implementation. This means on many accounts pagination and search is slow because SORT is not supported. We use Horde's cache driver but it doesn't really yield and noticeable performance boost.

The proposal

In addition to mailboxes #1956 we should keep a copy of message information in the database. Then we can display mailboxes (folders) very quickly, paginate through them and search for strings. We can keep the database sync'ed with Horde's sync mechanism. This means the sync would shift from browser->Nextcloud to Nextcloud->IMAP. That implies that we need a new mechanism for the front-end, but I have some ideas and it should be fairly simple to work that out.

Eventually it will make this app similar to how desktop apps work (in regards to storing data locally).

A rough list of things to do

  • Create db table for messages
  • Refactor message retrieval code to go through a new service (Refactor imap search #2058?), where we can swap our the IMAP connection with a DB query
  • Add a mechanism that syncs a mailbox the first time it's accessed
  • Add a mechanism to trigger the sync of a mailbox
  • Respect UIDVALIDITY
  • Don't sync the virtual flagged inbox to disk fine for now. can be improved later

A list of ideas for follow-up features

  • Sync mailboxes in cron as well, then you get fast page loads and an up to date representation of what's on IMAP
  • Have a virtual mailbox for outgoing mail, allowing users to compose message even when IMAP is not available
  • Store drafts locally first and only sync to IMAP later. That would eliminate the buggy, unreliable draft logic we have right now (IMAP messages are immutable, we have to replace & get a new ID every time)

Concerns

Things that might be problematic are

  • Huge mailboxes -> may take some time for the initial sync, but IMO we never supported them well and hence not a problem

cc @nextcloud/mail @rullzer for early input/feedback

@ChristophWurst
Copy link
Member Author

A feature where we can really see the benefit of this is #17. Not all servers have support for threads and then it gets very expensive to calculate them on demand.

@ChristophWurst

This comment has been minimized.

@ChristophWurst

This comment has been minimized.

@ChristophWurst

This comment has been minimized.

Co-authored-by: Roeland Jago Douma <roeland@famdouma.nl>
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
Copy link
Member

@rullzer rullzer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.
Tested the latest release and works good enough.
Lets get this is so we can have small fixes again.

@@ -60,7 +60,7 @@ public function find(string $userId, int $accountId): MailAccount {
}

/**
* @param int $id
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

opsy?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Search regressions
5 participants