Optional delta style object for update operations in changes system

[ChristopherTotty]

Currently when using the changes system an update is stored as a complete copy of the object. It would be beneficial in some cases to instead store a delta object with only the properties that have been changed from the original.

This would be most helpful when using LokiJS as a persistence layer for client data that will be synced to a central database on a server, since it would reduce the amount of traffic that is sent as part of that sync process.

I am currently working on a project where this is the case, and I am considering making this change for my own use.

Would there be any interest in a pull request for this feature, and if so, is there any opinion on it before I get started?

[obeliskos]

Sure, that would be great 👍

By default, collections do not have cloning enabled so all returned objects are the same instances as those used internally. If I do a find and update the object(s), the internal references are already updated before I call the update() method which essentially just rebuilds indices and emits events.

So if cloning is not enabled, I guess you would have to 'pre-emptively' add cloned baseline copies of all documents when initializing changes api on collections that opt in. Then I guess you would not include those in changes, but upon generating changes you would need to rebaseline 'non-dirty' baseline objects. With those copies, you can compare each updated object with highest (most recent) history object (or if empty) the baseline object. Does that make sense?

@techfort wrote changes api so if he can think of any issues maybe he can chime in.

[ChristopherTotty]

Ah, I didn't think about cloning since we added our own implementation that is run on everything as it is pulled out of the database.

Seems like doubling the memory consumption of every collection that opts in would be a turn off.

What would you think about either requiring cloning to be enabled or potentially building that clone set as items are selected from the collection? That way memory use would start at the same level as the current implementation and slowly grow to potential max of double as items are pulled from the collection.

[obeliskos]

So let's say you add an collection constructor option... something like 'deltaChangesApi' which you can turn on. If they turn that on I can see us forcing 'clones' collection option to be 'true' whether they pass it or not... possibly even override cloneMethod to ensure we use 'parse-stringify' instead of 'shallow' cloning methods which may not be effective at tracking changes to nested objects.

Since you are introducing a new option, as long as the old behavior continues to work as it did before, you can override the other options as it makes sense to do so. Does that answer your concerns?

With cloning enabled there is still (at least temporary) memory overhead and performance hit to clone internal objects as they come in and go out through find/insert/update/etc ops. I can see that being perfectly fine though as your feature would be concerned with reducing transfer size of changesets.

[ChristopherTotty]

Sounds good, a constructor option and forcing cloning should work well. I hope to look at this in the next few days.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.