Purge History API doesn't remove events from database

[4nd3r]

version 0.18.3-1 and postgres. events disappear in client, but stay in database.

[richvdh]

works for me.

[erikjohnston]

It won't delete local events from the db, but will remove lots of the meta data about them. So while it doesn't remove all traces of the events, it does dramatically reduce the on disk space usage.

…

On Thu, 1 Dec 2016, 13:52 Richard van der Hoff, ***@***.***> wrote: works for me. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1621 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AICaWF-b_r54prCfyYj0Vmbd3yZvaTuNks5rDtD9gaJpZM4KvAoP> .

[richvdh]

so notabug?

[4nd3r]

messages content stay in database, so that's bug.

if I delete something I don't want to be there for various reasons, I expect it will be deleted, not just hidden.

[rubo77]

@Half-Shot sais:

I want to be clear that what your asking is also against the use case of the Prune API.
It's sole purpose is to free up space on the host. It is not intended for redaction. Redaction is a different concept designed for removing events across servers.
#1730 (comment)

I will try to create a script that doesn't just prune the room, but also redacts all messages older than X days

see also #1480

[kythyria]

I will try to create a script that doesn't just prune the rume, but redacts all messages older than X days

Aka, a valid reason to patch synapse to deliberately ignore redaction events.

[rubo77]

You could redact all old messages with a script like this: #1730 (comment)

[rubo77]

@kythyria: don't you think data parsimony is a desired option for matrix?

I think we should add an option to each room, if redaction really deletes content from database or just flags it as deleted.

In most cases a deletion is not desired, but it should be an option the room admin may choose for each room.

Also I think it should be visible in the room, that users see, that what they write could be redacted, so they cannot rely on the messages to stay there forever.

On the other hand It should also be visible that you have to be careful, because messages written there are undeleteable, so you know, that, what you post there stays there forever

[kythyria]

Or just make the latter warning show all the time, because that's the best you're going to get. You can't actually force anyone to implement redaction correctly. This isn't something a room admin can choose, unless the room never leaves a server they control.

And no, it's not a desired option when it also allows people to have temper tantrums and wipe everything for no reason. Heck, it's not a desired option for matrix at all as evidenced by the fact that it's not designed to make redacting easy.

[rubo77]

So summarized: this is about " Purge History API doesn't remove events from database ", which seems true – and desired

[kythyria]

The purge history API doesn't bulk redact. That's not its job. Is this even a bug?

[4nd3r]

there are companies who are using or wish to use synapse for internal communication, so no federation. sometimes they need to wipe all data about certain project, when project ends. that's quite common in some areas. that also includes chat logs, encrypted or not. so deleting events is about compliance and audits.

also without deleting content you don't have plausible deniability.

i think it's about labeling things correctly - if you enable federation, you lose control over your messages. but if your room (and maybe server) is unfederated, then you should have complete control over your data, that includes ability to safely delete events or other infromation from database.

edit: also GDPR

[kythyria]

If and only if the room is completely unfederated, and the server honours the relevant messages, will redaction do what @rubo77 seems to think it does.

[robcolbert]

Is this resolved? Can events and their content be erased? Beyond that, can I set a policy saying, "No message should live past 30 days."?

I am currently evaluating matrix-synapse for use within a private organization that has a 30-day chat retention policy. If I can't enact something like that either through settings or by perhaps contributing the code to do it, then I cannot use matrix-synapse.

It's not even a discussion. It's just a question: Does matrix-synapse have a configurable data retention policy?

[4nd3r]

Does matrix-synapse have a configurable data retention policy?

sadly, no

[robcolbert]

@4nd3r thank you for taking the time to save me a lot of it. This process started at the documentation, then into the config files, then into some code, then here.

To devs, if I can maybe make a suggestion: Somewhere important and big in the docs, it can state that matrix does not have configurable data retention. Not even saying should, it's just a line of text you could maybe put somewhere and be explicit about the policy if you feel up to it.

I suppose this is the beauty of socket.io, MongoDB, Redis pub/sub and a tiny bit of Node.js and its crypto module, then.

I mean matrix has a bright future as a federated system of record for public conversations. Sure. But, it has no place inside most corporations and...as has been stated above...within groups that need privacy. Lazy deletes and no retention policy mean this tech is literally inappropriate in privacy scenarios. And, that's ironic given it's one of the few with e2e encryption. Shocking, actually.

[rubo77]

@robcolbert: if you follow some of the linked issues, you will see that you can really delete messages older than some days but only if your instance is not federated to any other Homeserver.

[4nd3r]

@rubo77 those are "hacks" and not really a feature and because tinkering with synapse db is slippery road, i rather not use those hacks. and in corporate env hacks aren't solutions you want to use.

[rubo77]

I wasn't referring to database hacks.

There is a purge API in the code, that deletes old messages. And if they are not federated, they are gone.

[4nd3r]

@rubo77 what do you mean by "if they are not federated"? federate = false as described here?

as far i understand and have tested it, even if the room has no external members and you use Purge History API, then yes, scrollback doesn't work, but the messages are still there, in database and are searchable. purge only removes some meta data.

as @erikjohnston said in #1621 (comment)

i think there should be big red bold text somewhere stating that.

[rubo77]

Is there a reason, why purge is not deleting the messages content?

[rubo77]

I added the command to really delete local events and messages, which is what users want to achieve mostly i think in my PR for an example script: #1034