Performance with large number of stored events

[jacekjagiello]

I love idea of automatic state syncing, and generating new state based on saved events. However, I've made some benchmarks and find out that the most time-consuming part of loading state is getting all stored actions from local PocuchDB instance.

I've populate PouchDB database with n number of events like this one:

{
  type: "SEND_MESSAGE", 
  message: { 
    id: "2eac4368-bc64-4ceb-82c2-ded1b1eba32e",
    content: "Lorem ipsum dolor sit ament"
  }
}

I measure time needed to load all action from PouchDB database

computeNextState.js

(...)
console.time("getAllActions")
const result = await actionsDB.allDocs({
  include_docs: true,
});
console.timeEnd("getAllActions")
console.info(`Loaded ${result.rows.length} actions`)
(...)

Results:

Stored Events	110	500	1500	8100	23000	34500
Miliseconds	~1500	~1700	~1900	~3500	~5900	~9100

The problem is that to load about ~35000 events, we need to wait almost 10 seconds. I've also measure performance of reduce that calculates next state from loaded events, but it's performance seems to be quite good - it takes around 400ms to generate state based on 35000 events.

I am aware that performance may be dependent on my computer, but i think that this test shows main bottleneck of loading all store events from PouchDB. We need to figure out some way to efficiently load inital state

[jurassix]

Great write up!

My plan to prevent this issue was to consolidate actions into a single action given a threshold. When to do this is the hard part. If we set a threshold and compress all actions into a single replace state action - with a time stamp of the newest action - then we need to handle any actions coming with a time stamp before this.

What are your thoughts on this approach?

[jurassix]

Maybe if we chunked the batching we could mitigate some of the time traveling conflicts. So instead of condensing all actions at a threshold we could just condense partitions of the actions. This would allow us to insert actions in between. However, there will need to be some merge handling which might just be simpler to list into change events on pouch. Would be interesting to prototype this.

[jurassix]

Also, I have thought about moving the reducer to the server too. In this way you would emit actions and receive the state; which was reduced using all actions in pouch. This approach fixes your 10s lag issue. But is probably better implemented in a separate library.

I can build something out if your interested?