How would you persist the data in SQL Server?

[vdurante]

Just to get an idea, how should I approach the persisting of the document?

Right now, I am using Redis, which acts like a cache and persists the state in binary format for 5mins or so.

How should I approach persisting the data in a readable/searchable format in SQL Server?

[SebastianStehle]

This is tricky. The actual state is a binary representation, e.g. see our Mongo implementation: https://github.com/y-crdt/ydotnet/blob/main/YDotNet.Server.MongoDB/DocumentEntity.cs

It is not readable and searchable. if you want that, you need another field and convert the document to JSON whenever you write it. There are a few methods for it: https://github.com/y-crdt/ydotnet/blob/main/YDotNet.Extensions/YDotNetExtensions.cs

If you have a single server I would keep the document in memory and evict it when it is not used anymore. This is what we do in the server project: https://github.com/y-crdt/ydotnet/blob/main/YDotNet.Server/Internal/DocumentCache.cs

If you have a workload with relatively few interactions (not a text editor), you can just load and store it for each request. Perhaps with optimistic concurrency and a few retries.

[vdurante]

@SebastianStehle any reason why both MongoDB and Redis have an expiration date then? If they are the final persistence layers, than shouldn't they be persisted forever? Or do you expect the client to save it somewhere else?

Also, reading MongoDB implementation it seems like you are not really deleting the expired entities? Is the expiration stored there only as an example then?

[SebastianStehle]

MongoDB is doing the deletion for you (https://github.com/y-crdt/ydotnet/blob/main/YDotNet.Server.MongoDB/MongoDocumentStorage.cs#L30).

I have a use case, where the data is not stored forever, so I added it basically for myself.