[consolidated] (auto-)delete / archive message history (support continuous daily usage)

[testbird]

To avoid overgrowing amounts of old messages using all device "disk"-space, there should be ways to limit the retained history on the server and locally.

(The underlying functions also need to work when checking and hard-limiting storage usage on server and device, and work for multi-client usage.)

EDIT: For the minimal, forward compatible implementation see #120 (comment)

---

Some reasoning to arrive at a consistent solution:

• Auto-delete chat messages by default from the servers's chat folder, to maintain free space and to provide multi-client consistency. There's otherwise no easy consistent way to distinguish when the user manually deletes messages to prevent filling up the server and the user only cleaning up the chat view (which needs to be synced with other clients).

• Keeping message archives on the local devices allows for independent use of different storage sizes on the devices.

• No auto-delete of emails from the servers' INBOX by default, as that folder is considered to be in the authority of classic MUAs.

• Never auto-delete emails from the local "incoming emails" view (old "contact requests"), to provide a view that is consistent with other MUAs INBOX view. (Manual deletion of junk messages (locally and on server), and removing messages that got deleted on the server is ok.)

• Restriction: Have a local-only INBOX display limit, e.g. for users that may not even have any subfolders but large (multi GB) INBOXes and never manually delete or clean up their email INBOX. The long list of incoming emails may not bother the user, but can be too much for the mobile data or the device's storage space. So an "show only the last 100 emails, and the next 100 only upon request" (and locally purging any further messages) could make sense as a measure against message overflow, but it could interfere with email compatibility and any later showing of subfolders in the incoming emails (contact requests).

Client-Server precedence

• If a chat message gets deleted on the server, the clients locally deletes active chat messages, but keeps the message if it has the "archived" state (only local archive purging rules apply).

• If a chat message is deleted on a client and there is no "archived" message property implemented, the user would need to be able (and have to manually decide, with an annoying question) whether to only delete locally or also on the server (to not have to remove it again from all other clients).

• With the "archived" flag available but not set, deletions of messages can by default always also be carried out on on the server (if still there). When a message gets archived, it can also be deleted from the server by default, and only kept locally. These defaults limit the used server space by the device with the smallest active chat length setting. Some clients, e.g. a desktop client, may optionally archive all messages right away (set to: active chat lengh zero) and let other the clients limit the server space (set to: not deleting archived message from server). This desktop is then effectively only working with archives, and ensures that there is always a local copy kept on the desktop.

Prerequisite Conclusions

• only deleting locally -> server chat folder would never be cleaned up, no multi-client syncing => also delete on server
• always deleting locally and on server (with small local storage) -> sync problem (larger local storage on other devices not used) => device independent archives
• current "archived chat" state is rather a "inactive chat" or "hidden chat" state -> improve or add feature to have real "archived messages" so that these archived messages are not revived completely with new messages coming in.

---

ToDo things:

-> for an updated list of needed message states see: https://github.com/deltachat/deltachat-core/issues/164#issuecomment-417841129

Chat message selection

• implement pinned/sticky/starred messages (edited according to comments)
  Allows to exempt messages from getting deleted by the following (automatic) bulk archiving and deleting.
• UI for pinned/sticky/starred messages (edited according to comments)

Deletion management

• core function to "delete (local and server by default) all messages except "sticky" up to this one" in a selection
• core feature to auto-delete older messages from chats (default those exceeding a 100 messages count?)
• UI option to enable auto-delete on chats (only needed as long as there is no auto-archive)
• UI option to manually "delete (or select?) all old (non-sticky!) messages up to this one" in active chats

**message archiving ** (not to be confused with the legacy "archived (or rather hidden/inactive/backgrounded) chats" feature)

• implement archiving of individual chat messages should allow for independent storage limits on multiple-clients and the server
  To keep chatting simple, manageable and search-able, the frontend can continue to only distinguish between active chats and "archived chats" (no weekly, monthly etc. separate sub-archives), yet better rename the legacy "archived chats" view into "inactive chats" for congruent naming.
  • make "archived" a property of messages instead of chats (so individual messages can be moved into archives).
  • core function to "archive all (non-sticky) older messages up to this one" in a chat
  • For the most straight-forward frontend UI, let the legacy "archive chat" UI option add the property to all the contained messages. Local archiving then becomes an easy way to remove the chat's messages from the server and from further syncing after the chat is archived locally and removed from the server and any other clients. (Further incoming messages then simply open in a new, empty chat view again.)
  • UI option to "archive old (non-sticky) messages up to this one" in active chats
  • UI option to "delete old (non-sticky) messages up to this one" in chat archives
  • UI button in chats to jump to first message (like existing button to jump to last message)
  • Convert any UI feature to auto-delete older messages in chats views into auto-archive in chats PLUS an auto-delete-local that suggests to phase out older messages from archives (provide confirm-to-delete suggestions, required to maintain free local storage space)
  • UI feature to auto-archive stale, inactive chats (default to "no message since 6 weeks"?). An option to disable this for a chat could be called pinning / sticking / staring a chat to the active chat list.
  • archive configuration options:
    • "delete archived messages from server" (and thus from other clients in a multi-client setup) Default: yes
    • (?) "move archived messages into a separate database" (possibly encrypted if placed on external, removable storage that may have to be connected before access is possible, but would still allow to open and search the archive, and respond, if needed)

Storage Restrictions

• Enable database compression (vacuum)
  Actually free storage space when messages get deleted. Needs to be triggered with DB size < free space to ensure success.
• auto checks & deletes based on storage space limits (auto-delete-server & auto-delete-local)
  When the local filesystem, or the server space, is full, no more chat or email messages can be received.
  • On incoming messages, let DC check the available space on the server and on the device, and when the free space falls below say 6% or 110MB, notify the user that DC will soon (below 5% or 100MB) have to start automatically deleting the oldest chat messages on the server or on the device. Non-chat emails should not be deleted from the server.
  • UI to manually set the storage usage limits (percentage and absolute) for automatic deletion of old archived and current messages.

Non-chat emails ("contact requests")

• Incoming Emails (former "contact requests" must never be **auto-**deleted (neither local nor on server), but messages deleted on the server, need also be removed locally, and manually deleting messages (junk) in "contact requests" need to be deleted on the server as well Emails deleted/moved from INBOX on server, need also be removed from "contact requests" #195
• An exception for a local-only limit option: For users that may not even have any subfolders but large (multi GB) INBOXes and never manually delete or clean up their email INBOX. The long list of incoming emails may not bother the user, but can be too much for the mobile data or the device storage. So a "show only the last 100 emails" (and locally purging any further messages) make sense as a measure against email message overflow. But it should be possible to disable it, as it could interfere with email compatibility and maybe later showing subfolders in the incoming emails (contact requests).

---

Implementing the above solves at least the following issues:

• Option to limit history download Option to limit history download deltachat-android#21
• Limit retained history Limit retained history deltachat-android#22
• Contact requests are never emptied Contact requests are never emptied deltachat-android#132
• Introduce a limit for number of messages in a chat Introduce a limit for number of messages in a chat deltachat-android#244
• Request: POP protocol implementation POP protocol implementation deltachat-android#227

[csb0730]

Here my last comment from ...android's issue 22

My proposal for a handy function:

1. IMPORTANT: make messages stickable or pinable first (I don't know the best wording ;-) ).
   That means that a message can be marked manually and then it is not touched by an automatic garbage collection. This enables the user to keep important messages!
   I think for that function database has to be extended?
2. introduce an option to set max number of messages in a chat's setting (and show somewhere current number of messages in a chat).
3. trigger the garbage collection whether manually or automatic
   (with new message in a chat or once per day or even with a button?)

Comments welcome ;-)

[r10s]

for 1. do you mean sth. like "starring" messages in Whatsapp?
In fact, the core of Delta Chat already has this functionality :) see: https://deltachat.github.io/api/classmrmailbox__t.html#a8ed1c3bd08cd6515abf6046ebad9cafc

[webratte]

@testbird if you write "delete all older messages up to this one" you mean also delete from server?
I think this is important to save online storage. Some providers have strictly limited storage.

[webratte]

Ok. now as I read your first post again it seams my post is unnecessary.

[csb0730]

Hi @r10s, yes,
as far as I found this is similar to that. Because I don't have whatsapp I thougt about a pin needle icon to indicate that a certain message is fixed. Therefore I used the wording "pinnable".

[testbird]

Just added database vacuum, as it is a prerequisite and was part of deltachat/deltachat-android#22 (closed).

[testbird]

(from #164)

Concerning multi-client, I generally think old chat messages can be assumed to have been downloaded, and if several clients try to delete some old messages from the server, even different ones (until a threshhold is reached) there should not be a problem. After they are deleted, all clients will sync to the same state again (server state taking precedenc

Concerning the concurreny to classic emails, autodeleting chat messages allows icoming classic email to cause chat message deletes until only classic email fills the mailbox, and blocks all incoming messages.

But it makes sense for Email-chat to operate in a cautious mode that by default avoids blocking classic email communication:

Because of the more transient nature of chat messages, and bacause chat clients are more likely to listen to IMAP-push (part of the spec), and to download/archive before deleting, I think an autodeletion threshold for old chat messages on the server by default is ok. (Deltachat deleting classic email would not be ok. That needs to be handled by a resposible user or MUA.)

[r10s]

core function to "archive all (non-sticky) older messages up to this one"

UI option to "delete/archive old (non-sticky) messages up to this one" in active chats

UI feature to auto-archive older messages from chats (default those exceeding a 100 messages count?)

this won't work this way as "archiving" is an action taking place on chats, not on messages.

in general, imo, "archiving" won't help on the initial issue "avoid overgrowing amounts of old messages using all device "disk"-space".

[testbird]

Hm, is see, the current archiving is just an "inactived" property or flag on chats. That means auto-deleting would lead to incomplete "chat archives", that can only hold the last messages and never more than the limit.

I'm trying to avoid such unpleasant surprises by looking at the whole picture before looking on single problems (like auto-deleting messages).

I assumed "archived" was already a property of messages, because that allows consistent actions based on that distinction.

• auto limit active chat (view), without risking to delete too much
• messages first only expiring into the archive, possibly long history if storage space permitts
• delete actions on archived messages (manual and auto-limits, auto-local-storage-restrictions)

Sure archiving won't help with deleting (cutting down own storage space), but both might be designed to allow them to work nicely together.

[r10s]

That means auto-deleting would lead to incomplete "chat archives", that can only hold the last messages and never more than the limit.

sure, when the space on the device is limited, sooner or later this will lead to incomplete chats. this, however, has nothing to do with "archived chats" that are only a flag that remove them from the main chat list (as in whatsapp, signal etc.)

by looking at the whole picture before looking on single problems (like auto-deleting messages).

this is always useful :) however, in practice, with limited resources and limited skills, we have to work with "what we have" what we can realistically achieve :)

i think the intent of this issue is described in #244 so we can close this one.

[testbird]

Björn, I suspect you already know that wrong paths can easy get one lost in a mess, especially concerning basic things. "What we have" includes evaluations, analysis, and proposals made. Limited resources means of course limited pace, that is absolutely ok, but I truly think a too a high pace in dismissing and closeing issuess, isn't really solving anything, rather the contrary.
If you need, take a time off, to get back in a state that allows detecting and considerating crucial things.

The intent of this issue isn't just #244 (fixed: history <-> local deletes), but working on a concept for all the related things to form a consistent solution, that may be implemented in small steps (finally reordering checkmarks above, you have write access).

Concerning a practical implementation sequence, it may not be necessary to enhance the "archived chats" implementation at first, but why not stay on top of things by developing or having a collaborative concept.

[testbird]

Moved these points to the top, does the beginning of the concept above match you intention, now?:

• core function to "delete all (non-sticky) older messages up to this one" in a selection
• UI option to "delete old (non-sticky) messages up to this one" in active chats
• UI feature to auto-delete older messages from chats (default those exceeding a 100 messages count?)

[r10s]

i won't got for an auto-archive. and i would see #244 independent of archived or not. in fact, the "archived" state typically used on messengers is a very unstable state - one message coming in and it has a different state. i do not think the user should worry about this.
generally, for now, i see just very few options, and they are covered by #244

[testbird]

The local-only deletes of #244 won't help with server space limitations, and neither work for multi-client nor for a consistent, user understandable view of "incoming email" in the server's INBOX.

Auto-archive isn't a must at all, it could be left out in the list above. Local archives would just be a useful split to limit the space used on servers, that even works with multiple chatclients.

Mind you though, that it makes no sense for me evaluating, checking and optimizing different options for a project, if the results are just thrown away.

[testbird]

There are reasons behind the concept, that might not be obvious on the first view. Therefore please post your doubts or ask, before closing prematurely.

(For example, deleting "archived messages" from the server can work for multi-clients on a "if still exists" basis, since the messages should all be in all the local databases already.

EDIT: It can work in the same way without "archived messages", but implementing local archives won't hardly work with IMAP using a per chat property.)

[r10s]

well, the discussion can be continued anyway :) or, even better, for a high-level discussion with the wider community, being continued in the new forum (which, of course, was just not there in the past)

[testbird]

That would not follow any due diligence. I updated the first post, separating the auto-delete feature, anyway, before I forget about this.

[testbird]

Hint: Instead of fighting against an overwhelming amount of issues, you may make this one, that was consolidated from many issues, your own, and cover a lot of other issues. The plan may still be adjusted over time, if more aspects come in.

[r10s]

i personally prefer to have smaller issues (of course, more then)

[testbird]

Well, the individual issues to work on in a meta-issue like this one are as small as the checkmarks above.

See how github says "1 out of 19 are done"?: https://github.com/deltachat/deltachat-core/issues?utf8=%E2%9C%93&q=is%3Aissue+consolidated+is%3Aclosed+

EDIT: And I'd doubt we would be happier splitting it all up into separate issues.

[r10s]

i know the display "x of y" and i also use these lists sometiems.

but the 19 points here are more 19 points, to discuss. eg. it's not clear if an option as UI option to "archive old (non-sticky) messages up to this one" in active chats will ever exist. same for other points in this "todo"-list.

this can be done here (although i would suggest the forum), but, however, this is not an open issue in the meaning "there is sth. to do on the code" or "sth. is not working" and so on.

but we should definitely make this more clear in the readme and point to the forum for high-level-discussions :)

[testbird]

And, any better idea for a single option that allows bulk managing the chat history length? This is the best plan currently.

[r10s]

#244 i'm not saying this is a better idea, however, this is sth. that will probably happen.

[testbird]

#244 does not mention a UI for bulk deletion, it is the same as the first 5 points (and the vacuum) here, even if an implementation may already keep that in mind.

Even if you choose to throw my work away, and not to reopen the consolidated overview: Please properly limit #244 to chat messages that need also be deleted on the server by default, and allow the incoming emails (former "contact requests") be reduced by classic MUAs that clean up the INBOX. (Then one would only miss the "delete all old messages up to this one" to clean up the incoming emails from within deltachat in the rare case of deltachat using a separate account, but individual email deletion is possible).

[r10s]

yes, #244 is very basic, of course.

and: closing issues does not throw work away, eg. it is linked from other, open issues and, of course, they may have influenced them :)

[r10s]

Please properly limit #244 to chat messages that need also be deleted on the server by default

see comment at #244 (comment)

[testbird]

It's that closing consolidated issues and implementing and closing "limited basic issues" that have been reviewed as not solving all the known problems, disregards the work gone into devising and documenting these things.

And it further guarantees that users will create new issues, without having a relevant open issue available to point to, that provides some status (checkmarks) and overview and plan to work with.

[r10s]

closing an issue does not disregards the work gone into it imo.
it stays available, searchable, visible, and is linked from other issues.
also it can be reopened, as needed.
it's not deleted or so.

[r10s]

make "archived" a property of messages instead of chats (so messages can be moved into archives).

let the "archive chat" UI option add the property to the contained messages. Further incoming messages then simply open in an empty chat view again.

the "archive" function then would be different from the archive function that we have today and different form the one people know from other messenger. not sure if this makes sense.

[testbird]

Because old, "archived" messages would stay archived, and not just temporarily disabled or hidden, i.e. not popping up again completely with new messages arriving, I think it does something understandable that makes sense. No problem. It's perfect if the active chat length is configurable per chat, and if it is easy to extend the view to (load) the archived messages (could be default for searching/filtering). The active and archived chat lenght form intuitive ways to control the storage usage on the devices and server (smallest active chat length is kept on server).

[testbird]

I've had an idea for a very slim way to implement the basic storage limits to ensure continuous daily usage. The idea is to start only with the smallest changes but which are fully compatible for auto-archive and -delete and thus are future proof, without requiring any front-end support.

The existing message property that holds the "stared" value might be extended to also allow setting it to the "synced-new" and "synced" value. (complete list of values: https://github.com/deltachat/deltachat-core/issues/164#issuecomment-417868402)

The -core may then just

• automatically change all "synced-new" messages in a chat that are surpassing the most recent, say 3, messages to the "synced" state by default (ensuring the newest 3 messages won't ever get auto-deleted)
• check free storage space (local and server) when downloading messages, and if required start to delete messages, selecting from the oldest "synced" messages (locally or on server) until the percentage limit is met again (see todo list in this issue above). (Later, do auto-archive instead, and only force-delete messages, if they still remain on the server after force-archiving them to the local device (i.e. user has disabled the default to delete archived messages from the server).)

(The default number of most recent messages is low (e.g. 3) to work without any configuration UI and to ensures that the auto-delete can delete almost all messages, if really required, but would never delete complete chats or histories.)

This basic storage limit enforcement would then be compatible and scale to all implementation stages of this issue. And it should work regardless whether the front-ends already implements UI support for any selection of the features, or not (stared/archived messages, configuring auto-archive/delete).

For example, same storage-limit auto-delete code should work no matter if users are able to delete recent messages from all synced devices, or if a proper archiving feature deletes archived messages from the server. The limits can always be ensured by the same basic code. Multi-client should also work well, if the code doesn't error out if another client already deleted the message on the server, and it won't delete too much because the most recent messages of each chat are protected.

Even the legacy way of completely suspending and resuming "archived chats" (hidden/inactive) from the primary chat list could continue to work as is, until it is improved to support the individual "archived" state of messages.

[testbird]

also updated the above to use "synced-new" and "synced"-history now