Store synced data

[turt2live]

Checklist

• Timeline events
• Room state
• Account data
• Per-room account data
• Read receipts (+co. - ephemeral events in the room)
• Prune storage to 50 timeline events
• Prune memory timeline to 150 events
• Don't store leaves (unless they are needed)

Description

This includes a fairly major refactoring of how rooms/events are represented outside the services. It should also address issues like #64 and add in supporting framework for #54.

Much of this issue is for documentation purposes when/if the system needs to be referenced in the future.

Stats reference:

• https://gist.github.com/turt2live/ef8d501df8ee6fcd9ce7aa32b1f0d624
• https://gist.github.com/turt2live/98fafe0546bd0a518ad7af5dee612152

Information to store:

• Events in room (50 most recent, 150 in memory - a later story will have these numbers be configurable)
• Joined rooms
• Room state (except m.room.member events of leave or kick, unless they appear in the 50/150 timeline cache)
• Batch tokens for each room (for backfilling)
• Account data (including per-room)
• Read receipts in rooms (typing notifications are stored in memory only)

Estimated storage use (for a 1000 room account):

• Timeline space (non-state, ~300 bytes/event): ~15mb (~15k/room)
• Room state space (~5k events/room, ~200 bytes/event): ~1gb (~1mb/room) (it'd be really nice to trim this down)
• Account data space (including room account data): ~450 bytes
• Read receipts space: TBD

Evelium should use indexeddb to store this information. For further space-saving measures, we could filter out events which cannot be processed by the client before they leave the sync processing. This could cause problems down the road though if Evelium, for example, gains support for net.arasphere.temperature then historical events will be missing.

[turt2live]

Future code will also need to store things like groups data, crypto, etc. At that point it should be possible to do a one-time sync with a filter to get the misc bits of pre-state we need to continue.

[turt2live]

Matthew's initial work at making syncs more manageable might help make storing some of this easier. The initial implementation of storing stuff in Evelium is intentionally limited in scope pending some progress on the /sync front, although it should still be okay for daily use.

[turt2live]

Priority lowered pending work on the doc mentioned in my last comment.