Properties play a critical role in end to end viewing workflows. The property data pipeline is powered up by an in house developed solution called property database. This section describes requirements, design, implementation and applications of property database in the context of LMV / viewing.
- Fast single object look-up for viewing/selection.
- Easy transformation to elastic search format.
- Integration with modern NoSQL database.
- Store arbitrary objects and their properties.
- Triple store (Entity-Attribute-Value triplets).
- Content-addressable store (like e.g. git) – stuff is never duplicated.
- Flat SQLite table as physical storage, for easy transformation to desired form.
"Four tables to rule them all"
In this design, properties are classified by category/units to make search smarter. Object relationships (parent/child/instance) are also stored as properties in the same store Object IDs from the original data file are used as keys, so link back is possible. Here is a visualization of how tables are referenced and composed together to form the data model:
The practice, the payload of properties materialized in two forms: a SQLite serialization file format (with an extension of 'db'); or a set of compressed JSON streams. Viewer consumes the
Property database data sets are generated by extractors through Property DB Toolkit, which is now part of LMVTK. In LMV viewer, we have a pure Javascript implementation of Property DB JSON parser that parse the JSON streams and load them in web worker. Checkout lmvtk/propdb.js and workers/PropWorker.js for more details. Also check additional design notes here. Note: property db data is critical to viewer. Viewer built instance tree from property database (rather than from the SVF instance tree) so without property db, features like selection would not work inside viewer.