plugins v2 (sectioned db, with hooks)

[dominictarr]

Basically, instead of hooking new behaviour into the main database, you can create sub sections:

SubLevel(db)
var fooDb = db.sublevel('foo', '~')

fooDb is an object with the levelup api, except when you do an fooDb.put('bar') the key is prefixed with ~foo~ so that it's separated from the other keys in the main db. This is great, because if you want to build a section of the database with special behaviour in it, you can create a subsection, and extend the behaviour in anyway you like -- but it will only affect that section! So, you can monkeypatch it - whatever, you won't introduce bugs into other parts of the program.

Most of the plugins I built needed some sort of interception, where a value is inserted into one range, which triggers an action which inserts something into a different section. To get reliably consistent data this needs to be atomic.

so, you can tie set hooks on a subsection, that trigger an insert into another subsection.

when a key is inserted into the main db, index that write with a timestamp, saved into another subsection.

var SubLevel = require('level-sublevel'); SubLevel(db)
var sub = db.sublevel('SEQ')

db.pre(function (ch, add) {
  add({
    key: ''+Date.now(), 
    value: ch.key, 
    type: 'put'
  }, sub) //NOTE pass the destination db to add
          //and the value will end up in that subsection!
})

db.put('key', 'VALUE', function (err) {

  //read all the records inserted by the hook!
  sub.createReadStream()
    .on('data', console.log)
})

db.pre(function hook (ch, add) {...}) registers a function to be called whenever a key is inserted into that section. ch is a change, like a row argument to db.batch: {key: k, value: v, type: 'put' || 'del'}.

If the hook function calls add(ch) ch is added to the batch (a regular put/del is turned into a batch) but, if a subsection is passed in add(ch, sub) then that put will be added to that subsection.

Compare subsection code for queue/trigger https://github.com/dominictarr/level-sublevel/blob/e2d27cc8e8356cde6ecf4d50c980c2ba93d87b95/examples/queue.js
with the old code -
https://github.com/dominictarr/level-queue/blob/master/index.js
and https://github.com/dominictarr/level-trigger/blob/18d0a1daa21aab1cbc1d0f7ff3690b91c1e0291d/index.js

the new version is only ~ 60 lines, down from about ~ 200, also it's possible to use multiple different queue/trigger libs within the same db. And also, there is no tricky code that refers to ranges or prefixes in the subsection based code!

in summary,

• create subsections, add any features to your subsection.
• use pre(fun) to trigger atomic inserts into your subsection.

[rvagg]

I've been using sublevel and have come around to the approach for extending levelup, but it's still not quite enough, what I really want is something between hooks and the approach I outlined in #97. Perhaps hooks can be adapted though.

So, what I really want is to be able to intercept calls in the middle of the original LevelUP method. Lets say I want to intercept all put() calls and write some special entries based on the key and/or value (in a sublevel). But, there's a couple of barriers: if I monkey-patch then I have to deal with the optional arguments bit and also the encoding. Say I want to take the key and use it to insert some other key into a sublevel based on it and also based on something special in the options that may or may not have been passed to the method. First I have to do the same thing that LevelUP already does to find the options argument and decode it (e.g. it could just be a string which would be turned in to 'valueEncoding'). Then I have to encode the key, perhaps they are doing JSON encoding on keys, I'll need to do the same thing in order to have enough information to proceed. After all that I need to make sure I pass the originals back to LevelUP.

I don't believe level-hooks does enough of this to help, perhaps it can be adapted to do so? What I want is something like what the externr code was doing by giving me an injection point right in the middle of the method where I can have the encoded key and value and even the internal options object.

How can we have it all?