Skip to content

Redis Oplog implementation to fully replace MongoDB Oplog in Meteor

License

Notifications You must be signed in to change notification settings

marsPure/redis-oplog

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

13 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Work in progress.

Not working.

Not ready for testing

Not on atmosphere yet

Thanks for understanding!

LICENSE: MIT

Install

(Temporary like this until it will be published to atmosphere)

// Inside your meteor root
mkdir -p packages
cd packages
git clone https://github.com/cult-of-coders/redis-oplog
meteor add cultofcoders:redis-oplog

// Run tests
meteor test-packages --driver-package practicalmeteor:mocha packages/redis-oplog

// Current tests unreliable, they only work the first time, because of database fixtures

Usage

Meteor.publishWithRedis('name', function (args) {
    return Collection.find(selector, options);
})
// inserting data the same way you are used to
Messages.insert(message)
Messages.update(_id, message)
Messages.remove(_id)

// Does not offer support for upsert yet. You can do upsert, but it will not trigger reactivity with Redis.
// Not hard to implement, but not the main focus right now

// inserting data without reactivity
Messages.insert(message, {pushToRedis: false})
Messages.update(_id, message, {pushToRedis: false})
Messages.remove(_id, {pushToRedis: false})

// inserting data in a certain namespace(s)
Meteor.publishWithRedis('name', function (args) {
    return Collection.find(selector, options);
}, {namespace}) // will only listen for changes in that namespace.

Messages.insert(message, {namespace: ['xxx']})
Messages.update(_id, message, {namespace: 'xxx'})
Messages.remove(_id, {namespace: ['xxx', 'yyy']})

What ?

This is a replacement for MongoDB's oplog with Pub/Sub Redis.

Why ?

Reactivity will be triggered by a mutation, something that happens in the app. This enables ability to fine-tune the level of reactivity and we can even disable it, if we want to make updates in large-batches, like migrations or something that does not require reactivity.

Other mutations outside the app can still trigger reactivity as long as they communicate with Redis. Will offer documentation and support for that.

This will also give you the ability to make any database, data-source reactive, by applying similar principles.

Data Flow:

  • Insert/Update/Remove => publish to Redis "collectionName::*"
  • Publications subscribe to "collectionName::*" and process changes for the client
  • We will also have dedicated channels for publications that filter by _id "collectionName::_id"
  • Additional support for reactivity namespacing, that can enable creating a chat app in Meteor like:
Message.insert(doc, cb, {
    namespace: 'thread-id'
})

Resources:

Merging scenarios:

https://docs.google.com/document/d/1Cx-J7xwP9IlbEa54RiT_34GK4o8M6XpPieRvNPI_aUE/edit?usp=sharing

About

Redis Oplog implementation to fully replace MongoDB Oplog in Meteor

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • JavaScript 100.0%