Ingest Modularity/Improvements

[djbrooke]

Related to ingest, I'm interested in

• allowing installations to ingest larger files ( handled in Set up new server for Ingest dataverse.harvard.edu#111)
• not experiencing system instability due to aforementioned large file ingests, or many ingests ( handled in Set up new server for Ingest dataverse.harvard.edu#111)
• allowing for external developers to more easily utilize the ingest code (I'm thinking TRSAs that want to generate and deposit metadata to a Dataverse installation ex. https://github.com/akio-sone/dataverse-ingest)

[akio-sone]

Before TRSA, Tim Parsons factored out the ingest functionality from Dataverse as follows:

data-file-reader

[akio-sone]

The current ingest functionality relies on a premise that the format of a data source has been identified; now is the time to delegate this step to an established tool such as droid that is running on a different server:
droid
format list
User Guide: pdf

I suspect somebody had already developed DROID's web-service version.

[djbrooke]

• Let's set this up on a different server (this may satisfy the system load and larger files question) - let's set this up in prod and see how it works. Document in Guides.
• Once this is done, it'll give us some idea about our next steps from a code perspective