Consider using relational-pouch

[sleidig]

The PouchDB plugin relational-pouch could be interesting for our implementation of PouchDatabase to simplify and optimize the management of different Entity types in the database. (also see #54).

---

Pro's using relational-pouch

• some indices may be added automatically and for other indices the syntax is easier (https://github.com/NGO-DB/ndb-core/blob/master/src/app/children/children.service.ts)
• as schemas are objects, this might help towards generating schemas from user-written configuration instead of code? (Configuration instead of Code #222)

Con's using relational-pouch

• the schema configuration is quite verbose and somewhat confusing to me
• relational-pouch would be yet another wrapper between PouchDB and out EntityMapperService, which already does similar things (we are actually using the same _id trick as they do: https://github.com/pouchdb-community/relational-pouch#how-does-it-work)
• looks like we still would have to create some indices on our own: https://github.com/pouchdb-community/relational-pouch#dbrelfindhasmanytype-belongstokey-belongstoid
• existing code (and data!) needs to be migrated

---

Anyway, we need a system to automatically create IDs and related objects (making all the duplicated code in ChildrenService obsolete. Either through relational-pouch or implemented on our own in EntityMapperService.

[sleidig]

Things like 2a1cc14 are rather boiler-plate code that relational-pouch seems to cover out of the box. Maybe it's worth using it.

[Lorsbyjm]

@TheSlimvReal @liwde Take a look at this. It seems to solve some of our problems for many to many relationships.

[liwde]

I think we talked about this with @fschoell a while ago, who was looking into this.
Coming back to it now, it seems like one main problem would be that we'd need to migrate all databases to the format the plugin uses:

db.rel.makeDocID({ "type": "author", "id": 123 });
// author_1_0000000000000123
db.rel.makeDocID({ "type": "author", "id": "onetwothree" });
// author_2_onetwothree

Another question is what to do with the MockDB, as adding map-reduce queries on the fly is possible (and might even be possible in a generic way), but reimplementing this plugin is not feasible.

[sleidig]

migrating existing data is not too bad, I think. It will be possible to bunch together some script for that.

The whole index stuff should better be hidden into the database layer anyway (not in something like our ChildrenService), maybe we have to create a MockEntityMapper for the MockDB to get around but the way we hack MockDB to handle indices is not feasible for a more complex project anyway ...

[TheSlimvReal]

Would it be possible for development to fetch a existing pouchdb into the local storage that is after that not beeing snyced with the remote db any more. This way we wouldn't need to implement the mocks our selves.

[sleidig]

Yes, I guess we could simply disable the remote database stuff. Loading the existing demo data into a PouchDB shouldn't be a problem. Although I added the MockDatabase especially to make unit tests faster (and more "unit-like", i.e. isolated). But for that maybe it's more sensible to simply mock the EntityMapperService using jasmine whenever needed in unit tests.
EDIT: or use an in-memory variant of pouch-db

I think removing MockDatabase completely (and adding a flag to PouchDatabase to disable remote sync) is a good suggestion.

[sleidig]

I have added a summary of considerations in the first post. Reading all the relational-pouch doc I am not convinced beyond a doubt. Can anybody else add their thoughts on whether relational-pouch really solves our problems?

[sleidig]

I found SlothDB, an ORM for PouchDB in TypeScript, that looks like it could be an elegant solution. Looking into it now, how we could integrate this into our database layer.

[TheSlimvReal]

It looks very promising, but it only supports one-to-many relationships but for children <-> schools we need a many to many relationship as far as I know.

[liwde]

It looks very interesting, indeed.

I think we can work around the many-to-many relationships the same way we do now, by just having an explicit Entity connecting both ends. In the ChildSchoolRelationship, we might even need such an entity as we may want to add additional properties to it.

I had a look at the issue you opened, and I understand what you mean. What causes a bit more confusion on my side is this PouchFactory and what determines the name that is used to retrieve a PouchDB from the Factory. Am I correct that this is decided by the first segment of the URL that is assigned to an Entity? Unfortunately, the example does not include the PouchFactory, so I couldn't see it there.

The Annotation-based approach seems cool and I like how this uses Reflection to define Getters and Setters for the attributes, but I'm not sure if I'd want to rely on this dependency. relational-pouch seems better maintained and I like the central modeling of the Schema in one JSON more, even if we have to still implement the OR-Mapping and cascading deletes ourselves...
But maybe I just have to play around with it a bit to be convinced...

[sleidig]

Looking further I also came across Realm. I love the simplicity of the API there but unfortunately sync seems only possible with their paid "Realm Plattform" service.

I would love to implement a similar API for us, however. A simple JavaScript object to define the schema of an entity type would be simple and elegant - and opens up easy possibilities towards creating entity types from the UI (#222).

Something like:

class Note extends Entity {
  static schema = {
    children: "Child[]",
    text: "string",
    date: { type: "string", indexed: true },
  }

  children: Child[];
  text: string;
  date: string;
}

Where EntityMapper than automatically creates database indices and provides the kind of functions to query the entities (similar to our explicit code in ChildrenService currently).

[TheSlimvReal]

I think it would be a good start to implement some generic methods to create database indexes and querys because they share most of their logic.

Another question. Is the database that much of a bottleneck at the moment or what do you think how soon we will run into limitations with the current model?

[sleidig]

Yes, I have started tinkering with this, will push an early branch so that everybody can have a look and give feedback. At the moment I favour to implement the index creation on our own without relational-pouch or other libraries - although I seem to change my mind about this every other hour ;-)

What kind of bottleneck do you mean? The biggest problem I see currently is the complexity to create new Entity classes during development. But I am no Pouch/CouchDB expert, so there may well be other unknown limitations

[TheSlimvReal]

I mean changing to relational pouch or something else would probably mean a lot of work until everything is back on track. The question now for me is, if it's really better to do this work now so we don't have to do it later when the app and data becomes even more complex or should we just hold this issue for the moment and try to get on with other more urgent problems?

[sleidig]

In theory, with a decoupled architecture it shouldn't cause too much change at other places (but we are not quite in that state (yet)). I think it's worth the effort as it can simplify all future development.

But I would probably first merge any Entity related features, like Height&Weight (#214) before putting such changes here changes into master

[sleidig]

I am closing this for now, as the move is now towards implementation of our own EntitySchema. see #260