Data Storage & Synchronization

[zicklag]

Handing data and state in applications is hard. This article has a good breakdown of some reasons that this is hard: Stop Building Databases. I think there are a lot of things in play here, and a lot of potential options, so I want to start by breaking down the problem a little better.

Desirables

There are some major things that we would like our data/state management system to be able to do, in order of difficulty:

• Persist data
  • This is obvious, but for completeness, we want to be able to store data without it disappearing when we turn off our computer.
  • To be more specific, we want to be able to store our data on a remote server so that it's always available over the internet.
• Optimistic updates
  • When a user clicks "edit" and starts modifying the data somehow, as soon as the user clicks save, it'd be nice to just "save" it immediately, and allow them to continue navigating the app, instead of waiting for the save to actually reach the server.
• Offline editing
  • If there is no connection to the server, we want to be able to edit our data locally. Once we connect back to the server, it should save the data we changed locally.

ℹ️ Bonus: If you get offline editing working, depending on how you do it, it's not much more work to do realtime multiplayer editing, which might not be a major use-case for weird, but could be handy in some situations.

You can do optimistic updates without doing offline editing, but if you put in the work necessary to make offline editing work, then you can kind of get optimistic updates for free, because you can just edit locally, and then sync with the server in the background.

Since we know offline editing is something we're going to want, even if we don't get it yet, lets think about what that would take.

Challenges

There are two related problems to solve to enable offline editing.

Running "Server" Code Locally

When we're offline, we don't have the server to talk to, but we still need the app to work. This means that certain things that are usually the servers responsibility, now must be handled by the local app.

But these responsibilities that are now taken on by the local app have to be done in exactly the same way that they would be done on the server, so that you don't end up with different behaviors, inconsistency or incompatibility between the offline app and the server.

The best way to do this is often, when possible, to literally run the same code on the client app and the server.

Synchronization

Even once you're able to run your app offline, you still have to be able to synchronize your offline data to your server. If there was only ever one offline device, this would be easy, but when you have a phone, and a laptop, and you make separate edits on both, you might run into conflicts, and you have to figure out how to merge the edits from both.

Possible Solutions

Running "Server" Code Locally

Service Workers

Service Workers are an incredibly useful tool for making offline-first web apps. They let you run a background JS script that acts as a middleman between your app and the server.

The service worker runs locally and can respond to network requests made by your app, as if it is the server, and this can be used to do caching, so that you don't make slow network requests to the server as often, or so that you can still access the application even when you're offline.

The service worker also seems to me like it'd be a good place to put that "server" code that we need to run locally.

WASM Components for Local "server" Code

Now that service workers allow us to pretend to be the server locally, we still need a way to locally act like the server. I think WASM components have a serious potential here. If we design our server-side to use of WASM components for server-side logic, then we can run the same WASM component locally, to respond to requests even if the server is un-accessible.

Component Isolation & Portability

WASM components have this concept that they are only allowed to talk to the outside world through specific interfaces. For example, if I run a Rust program on a server, normally that Rust program has access to the entire server, all it's networking, filesystem, etc. WASM components, on the other hand, allow you to be specific about how it can interact with the "world" around it.

This lets us create server components, for example, that are allowed to do only a few things:

• Respond to web requests
• Read and write to a database connection

By isolating the component like this, we make it more portable. On the server, it can respond to web requests that come from the internet, and it can read and write to the server database connection.

But we can run the same component in the service worker, but hook up it's outside world access to local versions. It will be able to respond to requests made by our app in the service worker, and it will be able to read and write to a local database embedded in the browser storage.

We can produce exactly the same behavior as the server locally, and our local app code doesn't hardly have to care, because the service worker can be almost "invisible" to it, pretending to be the server so that nothing else has to change in the app to work offline.

Synchronization

Now we still have to handle synchronization. When we're offline, we're writing to a local database, but we still have to be able to sync to the server database, too. This is one of the bigger questions.

SQLSync

One option is SQLSync, which is where that article linked above came from. It's something like a combination of Redux, Sqlite, and Git. It runs SQLite compiled to WASM on both the server and in the local app. I haven't looked into the details of how it works completely.

CRDTs

CRDTs ( Conflict Free Replicated Datatypes ) are a way to handle different things editing the same data at the same time without necessarily all being able to talk to each-other at the same time. This post is a great interactive explanation of how one kind of CRDT works.

A simple way to think about it is that it gives you a way to decide, for sure, how any edit will merge with another edit.

For example, if you have a list of TODO items that is edited separately on two devices. You can encode some rules like these in your CRDT:

• If you add a TODO, just add it to the list of todos.
  • This means that if your phone adds "brush teeth", and your computer adds "take out trash", then when they sync, you'll have both tasks in the list.
• If you change the title of a TODO, then pick "most recently" changed todo.
  • In this case, we don't care about how recent it is in time but how recent it is in version.
  • In other words, if you change the title twice on your phone, and once on your computer, your phone is considered the more "up-to-date" and it will use your phone's.
  • If they your computer and your phone were both edited once offline, then it will just pick one, based on the unique ID that it gives all your devices.

There's some fancy math stuff that can go into proving that your CRDT always produces consistent results, but the concept is really quite simple, and they're just built on simple rules under the hood. And we don't have to worry about the math, because somebody else has already done it for us.

Summary

So with Service Workers, WASM Components, and some method of synchronization, we're on the road to offline support, but there are still details to work out.

Databases

We need to figure out how we're going to implement the "database" piece of all this. Above, we've worked out that the database needs to be able to do the following:

• Run on a server, where it can be accessed by a WASM component.
• Run in a service worker, where it can be accessed by a WASM component.
• Have some method of synchronizing using CRDT or otherwise.

There's something else worth thinking about, related to databases, and that's migrations. As our app develops, we'll have to have a way to do things like move data around, rename it possibly, and add new kinds of data that we need to track.

All of this also needs to be handled properly when a migration happens on the server, while there are other offline edits. It's just another factor influencing the database/synchronization system.

Key Value Stores

One thing I've been leaning towards is Key-Value stores, due to their great simplicity. As SurrealDB has shown, you can build a complete database top of just a simple key-value store. In this case, we don't necessarily need a complete database with tons of features, but it proves that it's possible to achieve good performance using only a key-value store, and that it's possible to add any kind of data structures or relationships that you need on top of it.

The advantage of this is that it makes it super easy to put that database anywhere. You can run it on a server using the local filesystem and RocksDB, or you can run it in the browser using IndexedDB, or you can run it on top of any other SQL database, such as sqlite, by just creating a table that has a key and value columns.

I have a suspicion that using a key-value store could have some potentially useful features:

• Key-value presents an easy way for us to store multi-tenant data.
  • Our isolated WASM component can be given access only to keys that have a certain prefix associating the data to a specific tenant.
• Key-value might help us give isolated power to WASM plugins.
  • For example, we can reduce the security threat involved with installing plugins restricting which key prefixes a plugin is allowed to read to or write from in the database
• It might be easier to implement a CRDT on top of the key-value interface, than it would be on top of SQLite.

In other words, we can much more easily chop a key-value store up into pieces and give access to only certain pieces of it, which is only possible to do in SQLite by using multiple databases, which is not feasible in all situations.

Another thing of note is that when using a SQL database, usually people will use ORMs to make it easier to interact with databases in code. Writing SQL to do everything really isn't a great experience in code. There's no reason you can't write an ORM-type library for interacting with the key-value store either, so that you get the same kind of usability as you would with a SQL database in your code.

SQLite / libSQL

I think SQLite / libSQL is the other most promising option. I still need to investigate different options for synchronization on top of SQLite. SQLSync has one solution, but there are also others like CR-Sqlite.

Summary

There are still questions regarding synchronization regardless of whether or not we use Key-value or Sqlite storage. I feel like I want to rule a key-value store by testing it before just going with SQL. There's a chance that it will get us more "super powers" as we continue to develop, but there's also a chance that SQL support will be it's own super power. It's hard to tell, but I want to start with the simplest option, and move on to the more complicated option if it doesn't work.

Weird is a very good, simple application to test these kinds of things with. It's simple enough that it will be easy to change almost anything about it after the fact if it doesn't work.

[erlend-sh]

Illuminating ✨ This will make even more sense when we have an equivalent writeup for your thoughts on 'plugins and service integrations', but best to let that simmer for a few more days, maybe weeks.

To help contextualize this from my end, I refurbished this planning document:

• Stages of app maturation #18

At this earliest stage, my inner grugbrain is whispering: we are downstream of Rauthy. Rauthy uses SQLite. We build with SQLite, make good friends with Rauthy.

Weird is a very good, simple application to test these kinds of things with. It's simple enough that it will be easy to change almost anything about it after the fact if it doesn't work.

Exactly. The most important thing is to just pick anything and run with it. Our first integration of our linksapp is gonna be so simple that doing a database migration/rewrite for when we need more advanced capabilities ought to be very easy. If it's not, then we probably went way out of scope with our MVP.

[zicklag]

At this earliest stage, my inner grugbrain is whispering:

Hehe, that's a funny page. I've only gotten to "Testing", but it's good so far. :)

My grugbrain is thinking: "Complexity bad. Don't add SQL if no need SQL." 😁

Exactly. The most important thing is to just pick anything and run with it.

👍️ 👍️