Feature Request - Cache Failure should be Non Fatal

[ronag]

If the cache fails that should be not fatal and only result in degraded performance. Deepstream should try to reconnect to the cache.

Related to: #207

[yasserf]

How would this work in terms of data consistency? We only consider a record update to be successful when it is written to the cache so this would cause a big bottleneck and risk all the data in process to be lost since it would be in a in-memory queue.

Ideally you would use a cache connector with multiple nodes in a cluster and the cache driver should automatically switch. This is covered I believe in the 'ioredis' driver that redis is now using.

[ronag]

This is covered I believe in the 'ioredis' driver that redis is now using.

Will have to read a bit more about this:

We only consider a record update to be successful when it is written to the cache...

I believe this is a dangerous assumption in general when talking about caches. A record update should only be considered successful once written to storage? What are the implications of that?

How would this work in terms of data consistency?

Not sure...

[ronag]

@yasserf: Given redis. What if items are evicted (due to e.g. LRU) before it's written to the storage? Isn't that the same scenario, from deep streams perspective, as if redis died?

I believe this is related to: #278

We only consider a record update to be successful when it is written to the cache...

If this is the case then I believe the cache api needs to be updated so you can signal when items are allowed to be evicted by the cache... which would be strange...

[yasserf]

-> write to cache ( resets LRU )
-> if write to cache successful, write to storage

If the write to cache fails, then user gets notified that the write failed by a storage_error, meaning he should revert changes or present some other mechanism to resend updata.

[ronag]

-> write 1 to cache
-> written 1 to cache
-> write 1 to storage
-> write 2...100 to cache
-> evict 1 from cache
-> failed write 1 to storage
-> invariant broken?

Although, such a scenario is probably unlikely, not sure

[yasserf]

Write to cache fails, client recieves a RECORD_UPDATE_ERROR
Write to storage fail, it currently just logs an error.

Your scenario is perfectly valid!

This does present an issue, I thought the client received a storage write error but by the time the storage is written the handlers to the sender is already cleared and hence just the log. I'll raise this with the core team and discuss how this plans out in robustness.

Will add a new label for robustness issues.