Production grade source-of-truth

[tslocke]

My question is related to #9536

Reading the docs, there seems to be an assumption that the Fluid Service, which is responsible for persistence as well as collaboration, becomes the authoritative source of truth for the application. This is a nice simple model, but it raises all sorts of questions.

For many existing apps, there will already be a database which is the source of truth. Can these be kept in sync?

For a new app, if I am to let the Fluid Service be the single source of truth, I essentially need to it to be a full database:

• Can I run arbitrary queries on the data? Using what query language? Can I configure indexes so that my particular query needs are efficient?
• Can I grant access to non-web clients? e.g. analytics tools, batch processes. Using what protocol? How do I configure access rules?
• Can I have snapshots that I can re-instate? Access logs?
• etc. etc.

Realistically, I can only imagine the Fluid Service being a viable source of truth if it is built on top of some already mature database, and that underlying database is exposed.

Alternatively, robust two-way sync with existing databases would do the job.

Maybe a server can connect to the Fluid Service just as browser clients do, but instead of rendering a UI its job is to receive changes and persist them to a regular database?

Thanks

[justinstigall]

Persistence is really the missing part of the entire documentation around Fluid Framework. As you mention from all your bullet points its very hard to trust this as a database when you can't see what's going on underneath.

My only solution right now, if I were to build a collaborative environment on Fluid is to pull a recent copy of the data I need from a database like CosmosDB, populate the container, connect the clients and then periodically save to the database using the client. Once all users have disconnected I'd have to make sure a fresh copy is pulled from the DB once a new user tries to get that content again.

Not sure this is really a valid production framework until everything you've stated is resolved. As solution like CosmosDB changefeed + azure functions + SignalR seems to be the way to go for now, but that's hard to use for collaborative text.

[AndrewCraswell]

Exactly this. I've read through all the docs multiple times. I still can't understand when you would want to use Fluid Framework. It seems great when looking at simple demos. But how would it fit into a real-world collaborative app? As far as I can perceive, all the data that ends up in the containers become a black box to the rest of my app. Other features in my app can't interact with that data.

This leaves me to feel like Fluid Framework is only meant to be used for temporal collaborative features. Like showing activity indicators, audience lists, etc. Basically, the sorts of stuff you wouldn't need to persist anyways (ex: indicators to show which text a user has highlighted in a collaborative editor).

[eduardpaul]

Totally agree and your comment is still relevant today (year after) even the Fluid Framework is almost in V2 and Azure Fluid Relay is generally available I was not able to find "production use case" as no documentation is available on how to get the state of the container to persist for long term...

From Azure Fluid Relay documentation: "If an application stores data that may need to be exported by end users, the application developer is responsible for building that export functionality into their application, using the current state of the Fluid container as represented by the Distributed Data Structures defined in the container." Hard to understand this from a paid service in Azure 💔.

[nmsimons]

The current storage solution for Azure Fluid Relay is limited in that the data is only accessible via the Fluid Framework, however, the data is secure and safe. The expectation is that products can rely on Azure Fluid Relay storage as the source of truth. That said, there is a new option available in preview where Fluid data is stored in a new M365 offering called SharePoint Embedded. In this case, the Fluid data is stored in files in a SharePoint instance that is part of a customer's M365 tenant. This is the model being used by Microsoft Loop.

Moving forward, we are always considering new options and one idea would be to offer a solution where storage is decoupled from the Fluid Relay. The advantage of such a model is that developers could retain complete control of the storage layer. However, the storage layer would need to implement an API defined as part of the Fluid Framework that allows the Fluid Relay to read and write data.

Is such an approach interesting?

[d2e]

@nmsimons
Nick How How do you persist the data in share point?
Do you run a fluid client bot which persist the data as file in share point?

[nmsimons]

@d2e, no. SharePoint has it's own Fluid Service endpoint that handles file I/O directly.

[d2e]

Is it a fluid replay that writes to sharepoint using sharepoint driver?

[nmsimons]

The driver is client code that sits between the client runtime and the service. But yes, SharePoint has a Fluid Relay that writes directly to SharePoint files.

[d2e]

The current storage solution for Azure Fluid Relay is limited in that the data is only accessible via the Fluid Framework, however, the data is secure and safe. The expectation is that products can rely on Azure Fluid Relay storage as the source of truth. That said, there is a new option available in preview where Fluid data is stored in a new M365 offering called SharePoint Embedded. In this case, the Fluid data is stored in files in a SharePoint instance that is part of a customer's M365 tenant. This is the model being used by Microsoft Loop.

Moving forward, we are always considering new options and one idea would be to offer a solution where storage is decoupled from the Fluid Relay. The advantage of such a model is that developers could retain complete control of the storage layer. However, the storage layer would need to implement an API defined as part of the Fluid Framework that allows the Fluid Relay to read and write data.

Is such an approach interesting?

Yes, this approach is highly beneficial in the long term. do you have such plan for R11 as well?

[jaime-rivas]

+1 to this approach, it is really important for developers to retain complete control of the storage layer to build coauthoring experiences with the Fluid Framework. Developers should decide if they want to use the default/convenient option like SharePoint Embedded or use their own storage layer implementing the API. To this point, I would like to emphasize that both options should have the same capabilities: SharePoint Embedded should not have additional/better capabilities than our own storage layer, both should have access to the same APIs. I mention this because I have experienced this issue with WOPI, where SharePoint has access to additional Office Online APIs that cloud storage providers do not have. The Fluid Framework seems really good and would love to use it to power several coauthoring scenarios, but we need full control to easily persist data in our own storage.

[nmsimons]

Thanks for the feedback! We definitely understand the value of symmetry between host APIs. I will add that SharePoint has features that other some storage hosts do not have (and, vice versa). In general, the goal would be to decouple the core functionality of Fluid from specific host features, however, a solution that leverages such features (say, sharing URLs) would require equivalent functionality from each host to be truly symmetrical.