Provider-based data persistence design

[doodlewind]

A provider-based design is proposed here to synchronize workspace content via different data persistence backends.

The basic idea is to introduce DocProvider for syncing YDoc, and BlobProvider for syncing binary contents (e.g., images, files, videos).

YDoc and DocProvider

A DocProvider is a regular YDoc provider. It doesn't have APIs to be explicitly called during the YDoc lifecycle. DocProviders may include:

• DebugDocProvider - for playground
• IndexedDBDocProvider - for web
• WebSocketDocProvider - for web and client
• FileSystemDocProvider - for client
• SQLiteDocProvider - for client

Multi DocProviders with the same type (e.g., multi WebSocketDocProvider) can be attached to one YDoc, syncing the doc to different hosts.

BlobStorage and BlobProvider

A space (or page) may add multiple images to its content. These binary contents are defined as blobs and should be stored in a collaborative KV store. This store is called BlobStorage and has similar APIs as localStorage. Generally, we should only work on these BlobStorage APIs for performing CRUD operations for blobs. A BlobStorage can connect to multi BlobProviders, where the actual persistence operation happens.

APIs for BlobStorage:

• storage.addProvider()
• storage.removeProvider()
• storage.providers
• storage.blobs
• storage.synced
• storage.onChange(event)
• storage.get(id): Promise<BlobURL>
• storage.set(file): Promise<BlobId>
• storage.delete(id): Promise<void>
• storage.clear(id): Promise<void>

APIs for underlying BlobProvider:

provider.synced
provider.status // code: 0
// host: affine.pro, sizeLimit: 1024
provider.config
provider.setConfig()
provider.disconnect() // throwable
provider.connect() // throwable
provider.getBlobInfo(id)

Different types of providers may include:

• DebugBlobProvider - for playground
• IndexedDBBlobProvider - for web
• BucketBlobProvider - for web and client
• FileSystemBlobProvider - for client
• SQLiteBlobProvider - for client

As an example, after a user opens his workspace in the Web version from Affine Cloud, if the workspace hasn't been opened locally before, these are what's going to happen:

1. IndexedDBDocProvider and WebSocketDocProvider are initialized for syncing YDoc.
2. IndexedDBBlobProvider and BucketBlobProvider are initialized (and connected to BlobStorage) for syncing images.
3. YDoc data got synchronized from WebSocket, at the same time this YDoc is also copied to IndexedDB.
4. Image blobs got synchronized from S3 buckets, at the same time these images are also copied to IndexedDB.

During this process, the onChange callback on the BlobStorage instance will be called once a new image is loaded.

Other providers

We may have AwarenessProvider and NotificationProvider in the future, but they don't have top priorities.