Read manifest from database

[sirex]

Currently we have a manifest called backend, which allows to read and store manifest in database, using any backend. Current implementation is too complicated and should be simplified.

Current implementation allows to use any existing backend, to store manifest, but this complicates thing and we should support only sqlalchemy backend, which should simplify things a lot. This new manifest should be called internal.

This manifest should work as a normal manifest, it should support usual things, other manifests support, for example:

spinta show
spinta copy manifest.csv -o postgresql://host/db
spinta copy postgresql://host/db -o manifest.csv
spinta run

One major difference, that sqlalchemy manifest, should support reading schema on demand, model by model. For example if someone requests data of /example/dataset/MyModel, then on request, only schema of this model should be read. This would allow to have large manifests and make it possible to have a dynamic schema updates, because schema would be stored in database and it would be read every time on request.

There is one issue, that models can have dependencies, so it should be possible to dynamically load parts of schema as they are needed.

Tasks

• Read whole manifest from database
• Read single model from database on request

[sirex]

marked this issue as related to #64