Harvester 2.0: Deploy mdTranslator on cloud.gov

[rshewitt]

User Story

In order to have a transformation service available to the harvester 2.0 pipeline, data.gov wants deploy an instance of mdTranslator on cloud.gov.

Acceptance Criteria

• GIVEN an mdTranslator app deployed on cloud.gov
  WHEN a request is made against the service
  THEN a response containing the translated metadata is provided

Background

• harvester 2.0
• mdTranslator is mature, actively worked on, we have direct contact with those that work on it, and it supports metadata standards we care about.
• example api
• mdTranslator rails API

Security Considerations (required)

none

Sketch

• Standup local instance of mdTranslator ( optional )
  • Capture local deployment instructions in README
• Determine repo to use (Jin's/Aspenwall's/ADIWG develop)
• Determine repo to store the code
• Write cloud foundry manifest using ruby buildpack
• Push and debug
• Integrate with github actions for testing and deployment
  • Use the tests which come from the repo you choose

[rshewitt]

macos has a system ruby pre-installed. if you want to install mdTranslator locally and have a mac make sure to install a version manager like RVM or rbenv.

[rshewitt]

• update from call with johnathan
  • we're going to use the science base api for the moment
    • science base has been/is currently in hot water for violating SLA.
    • we're holding off on deploying our own instance of the rails API and the translator gem on cloud.gov.
  • unless we need to add our own features sooner rather than later we may be good just sticking to their rail repo and translator repo ( i.e not using a fork )
  • dcatus reader could be ready at the beginning of the year.
  • mdtoolkit docker registry

[rshewitt]

sandbox branch with example post to mdtranslator converting fgdc to iso19110. I can probably just read the file instead of parsing it as a tree.

[rshewitt]

science base mdtranslator api returns a 200 response despite errors existing in the input fgdc dataset

• ERROR: ISO-19115-1 writer: spatial reference system identifier is missing: CONTEXT is resource spatial reference system
• ERROR: ISO-19115-1 writer: spatial reference system type is missing: CONTEXT is resource spatial reference system

There's a forceValid option which substitutes required content ( true = substitue; false = don't substitute ). Both cases includes the errors in the response content.

[rshewitt]

according to the mdtranslator api documentation "if validation errors prevent the writer from generating a return, messages will be returned as 'text/plain'". In that case, we would have to parse the response content to derive meaning from it. So far it seems as long as the response is okay ( e.g. 200 ) regardless of whether the transformation is successful the response contains what could be transformed.

[nickumia-reisys]

A tangential point... If we would be implementing our own version of the mdTranslator and the API configuration is separate from the translator logic, details on the response of the API seem to be biased and no necessarily representative on what we would be expecting in our setup.

[rshewitt]

mdtranslator rails app uses ruby 2.7.7. cf ruby buildpacks don't support 2.x. need to bump ruby to deploy.

[rshewitt]

mdtranslator is deployed here. parts of the ui aren't going to work but since we only care about the api it doesn't matter. that instance is using develop. this comment is no longer relevant.

[rshewitt]

bare bone mdtranslator app is deployed here. this route only accepts posts. here's a python implementation as an example. you can get some example data here

[rshewitt]

i'm going to remove as much content as needed from the app which derived from the rails start project utility before I commit anything to the transformation repo

[rshewitt]

sinatra could be a lighter alternative than rails

[rshewitt]

blocked until cf credentials are stored in repo

[rshewitt]

dropped instances to 0. app couldn't deploy using 64mb so bumped it to 128.