ANNOUNCEMENT: Plans for the next major version 15.0.0

[pubkey]

I started working on the next major RxDB version 15

The biggest changes will be:

• A rewrite of the data migration plugin.
• Switch to ESM module with "type": "module" see here

Other planned changes can be found here.

Current changes in the v15 can be found here

If you have any idea or proposal on how to improve RxDB (even with breaking changes), please propose them here.

[galaxyblur]

If I may suggest a good idea for a premium plugin: Convert a Zod schema to RxSchema. Zod is becoming widely used for marrying TypeScript with runtime validation. I, and many others, already have Zod schemas defined for my data models. This would make adopting RxDB much easier.

[pubkey]

@galaxyblur I see the value in this. But there is already zod-to-json-schema. So wouldnt this be like 5 lines of code that transform the zod schema to json schema?

[gigasource]

@pubkey i paid for premium but i think i want to use surrealdb , can you support surrealdb ?

[galaxyblur]

@galaxyblur I see the value in this. But there is already zod-to-json-schema. So wouldnt this be like 5 lines of code that transform the zod schema to json schema?

Fair enough! I totally missed that RxSchema is just json-schema, which is great. Thanks.

[pubkey]

@gigasource No there is no plan for a surrealdb RxStorage. Last time I checked they where not major enough. This might change in the future.
I have plans to add a storage for the Deno Key Value store btw, but making RxDB compatible with deno was hours of pain with no end in sight. Mostly because deno is not really compatible with npm modules.

[pubkey]

Added

Switch to ESM module with "type": "module"

[gigasource]

@gigasource No there is no plan for a surrealdb RxStorage. Last time I checked they where not major enough. This might change in the future. I have plans to add a storage for the Deno Key Value store btw, but making RxDB compatible with deno was hours of pain with no end in sight. Mostly because deno is not really compatible with npm modules.

I mean replication , they supports Change feed, i think it takes not too much time because it is like couchdb . I want to change to surrealdb from couchdb now.

[patrickcze]

One thing we have come across that would be useful to be able to have a TTL on records stored locally. In our use case there are certain forms of data we only need to know about the last 14 days.

Currently after the application is initialized we pull down the last 14 days, and start to sync future records onto the device but overtime these records eventually stack up. It would be great to have the ability to remove anything older than 14 days without deleting it and having that sync to the backend.

[patrickcze]

One other use case we have is the issue of parent and child records, ie one record has a foreign key on the other. When syncing this means that until the parent is synced the child will fail. Ideal there would be a way to specify relationships in the records and the sync of collections could be coordinated. But i have also thought putting a delay on the child could also work. Not sure if anyone else has this issue.

[pubkey]

sync of collections could be coordinated

This is the big tradeoff. RxDB choose to have a simple-as-possible replication protocol, so that it can be implemented on top of any infrastructure. To make this possible, we do not have cross-document transactions and it is not possible to enforce coordinated replications. Most users do not have this problem because they pack everything that belongs together, into the same document. Your code should assume that a linked document can exist, but also might not be there (yet).

[eliias]

Instrumentation (Measure) RxDB (storage) internals with events

This could help identifying performance issues, like missing indices, bad & inefficient queries, manual re-sorts. There are two different types of use for such data points. One is obviously for immediate performance debugging (console.log), but there is also some value in sending stats like these to an APM service (Bugsnag, Datadog, Elastic, Sentry, …) to create distributed spans, etc.

Example

// When subscribed to events via console -> this is logged via console.warn in development
rxdb.storage.indexeddb.query 01HCHY44REABSK4DK22PVWQ1DN: Manual (slow) resort. Check if adding an index for sort fields could help. sortFields=[category, triage_status]

When looking at the event from a structured logging perspective, we create this string from:

event_name: rxdb.storage.indexeddb.missingSortIndex
event_id: 01HCHY44REABSK4DK22PVWQ1DN // ULID
event_message: Manual (slow) resort. Check if adding an index for sort fields could help.
event_extra: { sortFields: ["category", "triage_status"] }

rxdb.storage.indexeddb.query 01HCHY44REABSK4DK22PVWQ1DN loaded=322, visited=322, results=322, indexEfficiency=1.0, query={
  "selector": {
    "category": "article",
    "triage_status": "shortlist"
  },
  "sort": [ { "category": "asc" } ],
  "skip": 0
}

rxdb.storage.indexeddb.index 01HCHY44REABSK4DK22PVWQ1DN index=[_deleted, category, triage_status], index_id=i12, lower_bound=false,61989911,5e-324, upper_bound=false,61989911

The ULID (event_id) spans across the whole API call (e.g., indexeddb.query). Using console.time and console.timeEnd, we could then even visualize the full query span with the browser's profiler (or with any of the aforementioned APM services).

I think producing these stats can be implemented w/o having a severe effect on the overall performance. Besides instrumenting the individual storage instances, there might also be other places where eventing could be helpful. Query and Doc cache is one I can think of. Not sure how this should be implemented, there might be other popular libraries that already do that. The Rails ActiveSupport Instrumentation guide has a good summary on why this can be helpful for developers: https://guides.rubyonrails.org/active_support_instrumentation.html (the implementation will most likely not make sense though 🤣).

[ShahriarHD]

checkpoints

In order to keep track of document changes in scenarios like Undo/Redo stacks or version control in general, it would be a very pleasant DX if we could have a native checkpoints plugin. Here's a rough example API I can imagine:

const collectionCreator = {
    // schema, migrations, etc.
    checkpoints: true,
    methods: {
        someFn: function(this) {
            // incremental modify/patch, etc.
            this.setCheckpoint('someFn', {/*optional context information*/})
        }
    }
}

then when working with a document we could have methods like doc.getCheckpoints() and doc.restoreToCheckpoint(id).
Examples from similar libraries:

• YJS
• Tinybase

[pubkey]

@ShahriarHD RxDB only stores the latest known document state.
We stored all changes in the past (aka event-sourcing) when RxDB was build on pouchdb. But experience has shown that event-sourcing requires to much performance for client side databases. This is why RxDB resolves conflicts directly on replication, not afterwards so that it only has to store the latest state.
But maybe what you describe can make sense for the CRDT plugin. There we could "undo" operations by calculating the opposite CRDT operation to any given step.

[ShahriarHD]

@pubkey totally agree it's a good fit for the CRDT plugin, I think building this on top of the CRDT plugin can also help with even more complex scenarios, for example: a multiplayer undo stack where each user can undo their own actions, not affecting changes made by others.

[risavkarna]

Could we add support for JSON Type Definition, a new RFC8927 that offers a much simpler alternative to JSON Schema? It is designed to be well-aligned with type systems and has tools for both validation and type code generation for multiple languages.

Secondly, are there guidelines and API docs for developing plugins for RxDB? I would personally like to contribute my DBML - RxDB plugin, which manages server-client schema sync, partial schema sync, and generates some scaffolding in a given directory. It also can support ORMs like Prisma with a DBML generator plugin. I have only tried it with Prisma due to its flexible schema AST-as-JSON, which they internally call DMMF.

When putting this together, I had to work with JSON Schema and I personally found it too verbose and cumbersome. Now that JTD is there, adding support for it might help with RxDB's DX.

The aforementioned library (not currently an RXDB plugin) doesn't require JTD so it can come in as a PR, if welcome, regardless of JTD support.

Seems like a move to JTD or even a second class support for JTD might help with type mixing issues. Unlike JSON Schema, JTD does not allow defining values that can take one of several types. There is a workaround using nullable.

[pubkey]

@risavkarna Is it possible to define maxLength and maximum for strings and integers in JSON Type Definition?
RxDB needs these values to create efficient index-strings.

[risavkarna]

Not natively. But using metadata to extend the syntax and the library to recognize it should work. I have not tried it yet but can make a quick proof of concept. Essentially, we can add a metadata property to the JTD schema that contains the constraints we want to apply, such as minElements, maxElements, uniqueElements, or regex.

We would need a custom validator function to check if the data matches the metadata. More flexibility for other complex requirements can be had this way.

As I am not yet familiar enough with RxDB internals, perhaps you could advise if this is a worthwhile tradeoff in the context of RxDB.

[pubkey]

I do not really see the benefits here. But anyway this should be a simple functions which transfrom JDT to a valid JsonSchema. Similiar to this #4994 (comment)

[pubkey]

Thank you for your great ideas. Most of them can be implemented without any breaking change, so I will not add them to the todo list.

RxDB v15 beta has been released. Please help testing: #5155

[pubkey]

@eliias Thank you for that great proposal. I added a logging plugin to the premium plugins.
You should be able to use it with the latest beta version.
Docs can be found here: https://github.com/pubkey/rxdb/blob/master/docs-src/logger.md