Consider having Ingest run on separate server / Modularize

[scolapasta]

No description provided.

[bencomp]

@scolapasta what? Why?

[landreev]

@bencomp: We always wanted to make the application more modular; so that individual pieces/components could run independently; and, if necessary, run on different servers - to deal with load issues and such.
This ticket is for making ingest such an independent service; so that it could run on another host, similarly to how Solr and Postgres are integrated into the db.

This doesn't mean it will have to run on a separate server.

Ingest jobs can potentially be expensive, with both memory and CPU cycles. So if this makes the rest of the app run slow, this could be a simple solution - just move it onto another server and let it chew all the CPU time and memory there.

[landreev]

@bencomp : (I meant to say "... are integrated into the application")

[pdurbin]

@landreev check out the comment by @wbuchanan about how he has built https://github.com/wbuchanan/data-transfer processing Stata files in Java: #2883 (comment)

[pdurbin]

I just went to a meeting with @jonc1438 and he mentioned that @akio-sone is working on something related to this. He's factoring out the ingest code from Dataverse to allow it to run as a standalone app, from what I understand.

[pdurbin]

As I work on #2301 I feel like it would be nice if our Stata parser were a stand alone app or library that I could use from Dataverse.

[djbrooke]

Closing this, as making the code modular will part of the work for TRSAs. We can re-evaluate after that. As far as setting it up to run on the separate server, there may be some options for ECS/Fargate for those installations running on AWS.

[pdurbin]

@djbrooke cool but heads up that @akio-sone has apparently already started working on this. The other day @donsizemore said "note that he carved out the ingest functionality so TRSA wants to talk to its corresponding fork of Dataverse: https://github.com/OdumInstitute/dataverse/tree/trsa-api " at http://irclog.iq.harvard.edu/dataverse/2019-09-06#i_104872

The plot thickens. @akio-sone let's use Quarkus for the ingest microservice! 😄 I have some notes at http://wiki.greptilian.com/java/quarkus/ 😄

[djbrooke]

Yes, that's what I was trying to express in my comment as I closed.

[qqmyers]

Also a connection with UCLA's interest in geodata processing, i.e. https://groups.google.com/g/dataverse-community/c/XKC3wwssniM/m/7lkhgnQfBAAJ .

[pdurbin]

See also:

• Ingest Modularity/Improvements Ingest Modularity/Improvements #7852