Skip to content

Implement sync client #70

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 38 commits into
base: main
Choose a base branch
from
Open

Implement sync client #70

wants to merge 38 commits into from

Conversation

simolus3
Copy link
Contributor

@simolus3 simolus3 commented Apr 23, 2025

This implements the sync client logic in the core extension. Client SDKs will still be responsible for opening sockets to the sync service, but the core extension is now responsible for driving that logic. The central interface here is the powersync_control SQL function, which is invoked by clients for a request to start a sync stream or lines received by the sync service. In response, the core extension responds with a list of instruction for the SDK (like e.g. updating the sync status).

We expect that implementing the stream client here will:

  1. Improve performance around handling BSON lines: Only our JavaScript SDK supports BSON at the moment. At the moment, that SDK has to deserialize the BSON line itself and re-encode it as JSON before sending it to the core extension. By implementing the deserialization in the core extension, we avoid that inefficiency.
  2. Make the protocol easier to evolve: Subsequent changes would mostly have to implemented here instead of at each client SDK.
  3. Make it easier to adopt the RSocket protocol across our SDKs.
  4. Not done here: Eventually support the BLOB column type, which requires a BSON-based transport that can now be handled here for all our SDKs.

The most important are structured like this:

  • core/src/bson contains a no-std implementation of the BSON format with serde support. The bson crate unfortunately only supports std environments and I couldn't find a decent alternative. We don't support all BSON types, but that should be fine since we only need the types supported by JSON (as well as byte arrays).
  • core/src/sync contains logic to implement the sync client:
    • line.rs defines the sync lines as structs supporting deserialization.
    • storage_adapter.rs is used as an interface for some common database operations. It's a bit simpler than the BucketStorage interface in the SDK since we can mostly just call other core functions directly without going through SQL.
    • sync_status.rs implements the SyncStatus interface from our other SDKs and logic to update it along with sending update notifications to the client.
    • interface.rs defines the powersync_control SQL function driving the sync client.
    • streaming_sync.rs contains the actual state machine for the sync client. I've implemented it as an async coroutine that is polled every time powersync_control is called, which allows an implementation that looks similar to the ones across our other SDKs.

@simolus3 simolus3 force-pushed the sync-state-machine branch from 756f514 to 2ba3209 Compare April 29, 2025 13:00
@simolus3 simolus3 changed the title WIP: Implement sync client, allow BLOB types WIP: Implement sync client Apr 29, 2025
@simolus3 simolus3 changed the title WIP: Implement sync client Implement sync client May 8, 2025
@simolus3 simolus3 marked this pull request as ready for review May 8, 2025 15:07
@simolus3 simolus3 requested a review from rkistner May 8, 2025 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants