You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In order for the harvesting logic repo to only be a python module for the ETVL of dcatus catalogs, datagov wants to remove all files, functionalities, tests, and dependencies which don't involve the ETVL of a dcatus catalog.
Acceptance Criteria
GIVEN the current harvesting logic repo #4925
WHEN files, functionalities, tests, and dependencies not involving the ETVL of a dcatus catalog are removed
THEN the repo will no longer contain these things.
Background
datagov has expressed interest in replacing the current harvester with a workflow orchestration system ( e.g. airflow or windmill ) utilizing a python module for the ETVL of dcatus catalogs.
One of the first steps in this effort was creating a harvesting logic repo containing general purpose logic for the ETVL of datasets we maintain. At the moment this repo focuses only on dcatus catalogs.
Over time, the harvesting logic repo has become bloated with unneeded content ( e.g. postgres interfaces, a flask controller, an airflow implementation, s3 loading functionality, and docker services ).
Sketch
remove everything related to the unneeded content mentioned in the background.
The text was updated successfully, but these errors were encountered:
remove/commit things by component so they're easier to identify in the future and include those commits in the PR ( e.g. remove s3 help module, remove sqlachemy module, remove flask controller, etc... )
User Story
In order for the harvesting logic repo to only be a python module for the ETVL of dcatus catalogs, datagov wants to remove all files, functionalities, tests, and dependencies which don't involve the ETVL of a dcatus catalog.
Acceptance Criteria
WHEN files, functionalities, tests, and dependencies not involving the ETVL of a dcatus catalog are removed
THEN the repo will no longer contain these things.
Background
Sketch
The text was updated successfully, but these errors were encountered: