-
Notifications
You must be signed in to change notification settings - Fork 22
Datasource Plugins
The purpose of this specification is to describe the changes to apply to Opal so that it can handle the addition of datasource plugins, i.e. plugins that:
- capture user parameters for building a Magma datasource,
- build a datasource upon opal core request for (meta)data import and export.
Have an extendable system that does not require a new release of opal when there is a need for interfacing with a new (meta)data source.
| # | Who | What | How | Result |
|---|---|---|---|---|
| 1 | developer | implements a new magma datasource | based on datasource SPI | |
| 2 | administrator | extends opal system import/export capabilities | by installing datasource plugins | user can import/export data in a new format |
| 3 | user | import/export (meta)data | by selecting a datasource type and providing parameters | (meta)data are imported/exported |
opal-spi, opal-core
This feature requires some major architectural changes in Opal:
- definition a magma datasource plugin Service Provider Interface (SPI)
- magma datasources can be discovered at runtime (opal start phase) by inspecting plugins,
- these pluggable datasources specify how to collect datasource parameters (remote server URL, file path, user credentials etc.) with their validation (file extension etc.) and the associated translation,
- the dtos, web services and ui will handle these parameter specification.
These datasource plugins will not provide a datasource instance: it will provide a datasource factory that will expect optional user parameters and/or plugin static paramaters to make a new magma datasource instance. The plugin will also specify what is the expected usage of the datasource to instantiate: import or export.
The importation workflow using the GWT UI will be:
- opal ui gets from the core the list of available datasource plugin names (identifier and human friendly name) supporting the importation usage
- user selects a pluggable datasource to import from
- opal ui gets from the core the JSON schema form definition for
importusage - opal ui builds the form and capture/validate the user parameters
- user enters parameters
- opal ui creates a transient datasource based on plugin name, usage and parameters
- opal ui triggers an importation task based on the transient datasource
- opal core disposes the transient datasource after importation task is over
The exportation scenario using the GWT UI will be:
- opal ui gets from the core the list of available datasource plugin names supporting the exportation usage
- user selects a pluggable datasource to export to
- opal ui gets from the core the JSON schema form definition for
exportusage - opal ui builds the form and capture/validate the user parameters
- user enters parameters
- opal ui creates a transient datasource based on plugin name, usage and parameters
- opal ui triggers an exportation task based on the transient datasource
- opal core disposes the transient datasource after exportation task is over
Currently there are specific DTOs for each type of datasource. Some generic pluggable datasource DTOs will be defined for carrying the datasource parameters form specification (JSON schema-form based, translatable) and the user entries.
This DTO will report:
- the datasource service identifier
- the datasource service human readable name
- the (usage specific) JSON schema form (as a payload string) so that UI can capture user parameters
This DTO will be an extension of the DatasourceFactoryDto. It will carry the following information:
- the datasource plugin name (identifier) so that opal core can get the corresponding datasource service
- the user parameters as a JSON string so that the datasource service can create a corresponding datasource factory
The web services for transient datasource creation, importation and exportation tasks submissions already exists and can be used as-is.
Additional web services will provide datasource services availability to allow the user to make its source/destination choice at import/export time. These REST resources will be available through the project web services, to anticipate a project restricted setup.
| Resource | Description |
|---|---|
GET /project/<p>/datasource-services?usage=[import|export] |
Get the list of datasource service DTOs that can be used when importing or exporting project's (meta)data |
TODO
An example datasource plugin based on an existing Magma datasource could be developed for test purposes, showing the different JSON schema form possibilities.
Demonstration will be based on a REDCap datasource plugin (to be developed).
Move the LimeSurvey datasource out of Magma library and make it a pluggable datasource (opal upgrade task to be implemented).