Need clarification on Transform behavior.

[hypercub3d]

Hi,

I'm having some trouble understanding how to correctly use the collection transforms. More specifically, I'm trying to figure out the correct way that the eqJoin and map transforms should be used.

Given this setup code:

const loki = require( 'lokijs' );
const lokiMem = new loki.LokiMemoryAdapter();
const db = new loki( 'my.db', { adapter: lokiMem } )
const coll = db.addCollection( 'mycoll', { unique: ['test'] } );

const data = [
      { test: 'a' },
      { test: 'b' },
      { test: 'c' },
      { test: 'e' },
      { test: 'd' },
      { test: 'g' },
      { test: 'f' },
    ];

const jdata = [
      { id: 'b', a: 1, b: 2 },
      { id: 'c', a: 1, b: 2 },
      { id: 'd', a: 1, b: 2 },
      { id: 'e', a: 1, b: 2 },
      { id: 'f', a: 1, b: 2 },
      { id: 'h', a: 1, b: 2 },
    ];

coll.insert(data);

I'd like to to join the data and jdata docsets. So I create a transform to allow me to do this.

coll.addTransform( 'testTrx', [
        {
          type: 'eqJoin',
          joinData: jdata,
          leftJoinKey: 'test',
          rightJoinKey: 'id'
        } 
] );

let out = coll.chain( 'testTrx' ).data();
console.log(out)

So, without providing a mapFun option to the eqJoin transform, the standard mapping function is applied yielding output that looks like this.

[
        {
            "left": {
                "test": "a",
                "meta": {
                    "revision": 0,
                    "created": 1506731589008,
                    "version": 0
                },
                "$loki": 1
            },
            "right": {}
        },
        {
            "left": {
                "test": "b",
                "meta": {
                    "revision": 0,
                    "created": 1506731589008,
                    "version": 0
                },
                "$loki": 2
            },
            "right": {
                "id": "b",
                "a": 1,
                "b": 2
            }
        },

... 
]

But I would rather have the data be joined directly into the document, so I modify the transform to look like this.

coll.addTransform( 'testTrx', [
        {
          type: 'eqJoin',
          joinData: jdata,
          leftJoinKey: 'test',
          rightJoinKey: 'id',
          mapFun: function( left, right ) {
            left = Object.assign( left, right );
            return left;
          }
        } 
] );

But when I invoke the transform, I get this error: Error: Document is already in collection, please use update()

It seems as if the data is being re-inserted into the collection, and the unique keys are clashing. So I attempted to strip the metadata from the record.

coll.addTransform( 'testTrx', [
        {
          type: 'eqJoin',
          joinData: jdata,
          leftJoinKey: 'test',
          rightJoinKey: 'id',
          mapFun: function( left, right ) {
            left = Object.assign( left, right );
            delete left.meta;
            delete left.$loki;
            return left;
          }

        } 
] );

Doing this yields the correct output:

[
        {
            "test": "a",
            "meta": {
                "revision": 0,
                "created": 1506732466194,
                "version": 0
            },
            "$loki": 1
        },
        {
            "test": "b",
            "id": "b",
            "a": 1,
            "b": 2,
            "meta": {
                "revision": 0,
                "created": 1506732466194,
                "version": 0
            },
            "$loki": 2
        },
...
]

This same behavior also occurs if I use the map transform and attempt to project the dataset. I need to strip the loki metadata, otherwise I get the same error as before.

If transforms are meant to be a way to restructure the dataset for output, does it make sense to have to manually strip the document metadata anytime the data needs to be changed?

The error advises me to use update, but as far as I understand, that modifies the original data.

Is there a better way to restructure or "project" data for use in views or as a means for output only?

[obeliskos]

eqJoin creates new wrapper object and inserts those objects into a (new) throwaway collection for future chain ops or call to data(). The derived objects to be inserted into this throwaway collections can not already have $loki or meta or those inserts will fail. Usually you would explicitly name properties in a new object literal, but to do what you are requesting required stripping meta within left and right sides.

I modified eqJoin to allow extra parameter 'dataOptions'. It will be applied to left side object always (the resultset you are calling eqJoin from). If your right side data is Collection or Resultset it will also be passed to data() call for those as well. (I just added support for Collection).

I have created a new eqJoin loki sandbox gist to demonstrate this new reworked eqJoin usage.

Note in this example i never need to save the database... passing a reference to a collection would not serialize (or deserialize) very well within a transform, so if this example was to save the transform within the database, we should probably convert the joinData -value- to be a loki transform param.

It is based on our unit tests which I will need to add a new test for this soon.

Let me know if you still have issues.

[hypercub3d]

@obeliskos,

Thank you for the quick reply, and for taking the time to implement a solution. I think the solution works very well, and it's a good approach.

Would it be possible to implement the same dataOptions configuration on the map transform? map seems like a useful method for projecting data in various ways within a transform, but the current implementation has the same issue as the eqJoin, where returning an object with its original $loki parameter will raise the Document is already in collection error.

Also, on a related note, the map transform seems to expect its callback function to be set via the value attribute, as opposed to mapFun. This seems inconsistent with the rest of the API, and so I'm assuming its a bug.

Here is the relevant code from: Resultset.prototype.transform

case "map":
          rs = rs.map(step.value);
          break;
case "eqJoin":
          rs = rs.eqJoin(step.joinData, step.leftJoinKey, step.rightJoinKey, step.mapFun, step.dataOptions);
          break;

[obeliskos]

The map transform (as well as the chained method itself) dumps mapped objects into anonymous collection which will need to add meta there. This map only accepts 1 object.

Or are you referring to eqJoin mapFun which accepts two (left and right)?

Do you expect the object you return from map of single (collection object) to have loki and or meta?

If not you can removeMeta on call to data like :

coll.chain("mytransform").data({ removeMeta:true });

Or (if chaining) :

coll.chain().map(mapFun).data({ removeMeta:true });

I tried adding to map to solve your previous problem before recognizing the issue was elsewhere. Wouldn't any

[hypercub3d]

@obeliskos,

The problem is how the data is injected into the mapping function.

Given my setup code from the top of this page, if you attempt this code:

coll.addTransform( 'testTrx', [
        {
          type: 'map',
          value: function( data ) {
            return Object.assign( { newField: 1 }, data );
          }
        }
      ] );

let out = coll.chain( 'testTrx' ).data( { removeMeta: true } );
console.log( out )

You will receive the error: Error: Document is already in collection, please use update()

So, just as in the case of eqJoin, the intermediate collection requires that the $loki metadata be stripped from the object before returning it.

That is why I'm asking if it would be possible to implement the dataOptions configuration on map, the same way as you have done on eqJoin.

[obeliskos]

Ok so for just grafting extra properties without defining new object... I've checked that in.

Word of caution, by default loki does not clone objects so arbitrarily modifying those actual object references will normally modify the documents in the collection, circumventing index logic and meta checks, etc... possibly corrupting integrity of collection.

With removeMeta, cloning is implied and if you are not already cloning we will shallow clone so you might remove or add top level properties without affecting collection.

If you wish to manipulate deeper within the objects, you should explicitly specify 'parse-stringify' cloneMethod in dataOptions to deep copy the objects... or apply that clone method to the entire collection options if you expect to be doing this in a variety of methods. You may need to balance performance with clone safety.

[hypercub3d]

I've been testing these changes for a little while now, and I can confirm that everything works as expected, so I'm closing this.

Thank you again.