Feature/grip #36
Feature/grip #36
Conversation
* add load_flat call to create research subject index * change index name * bump versions for new release
|
|
||
| RUN git clone https://github.com/bmeg/iceberg.git && \ | ||
| cd iceberg && \ | ||
| git checkout feature/FHIR-resource-type |
There was a problem hiding this comment.
similarly, do we expect this to be a long-lived branch or just for testing purposes
There was a problem hiding this comment.
Walsh needs to approve this PR before it gets merged.
There was a problem hiding this comment.
sounds good, up to you whether you wanna wait for it or patch it later
|
|
||
| RUN git clone https://github.com/bmeg/iceberg.git && \ | ||
| cd iceberg && \ | ||
| git checkout feature/FHIR-resource-type |
There was a problem hiding this comment.
sounds good, up to you whether you wanna wait for it or patch it later
etl-job/fhir_import_export.py
Outdated
| @@ -357,21 +314,17 @@ def _empty_project(output: list[str], | |||
| """Clear out graph and flat metadata for project """ | |||
| # check permissions | |||
| try: | |||
| empty_project(program=program, project=project, dictionary_path=dictionary_path, config_path=config_path) | |||
| grip_delete(graph_name="CALIPER", project_id=f"{program}-{project}", | |||
There was a problem hiding this comment.
Should grip's URL and PORT be passed as well?
There was a problem hiding this comment.
Seems the pod has the best information (it's env) to determine grip url, port etc.
Should we pass that information to the grip submission layer?
There was a problem hiding this comment.
Yes thinking about this further, need to add this.
There was a problem hiding this comment.
|
|
||
| index_generator_dict = { | ||
| 'researchsubject': db.flattened_research_subjects, | ||
| 'specimen': db.flattened_specimens, | ||
| 'file': db.flattened_document_references | ||
| } | ||
|
|
||
| # To ensure differences in the dataframer versions do not conflict, clear the project, and reload the project. | ||
| for index in index_generator_dict.keys(): | ||
| meta_flat_delete(project_id=f"{program}-{project}", index=index) |
There was a problem hiding this comment.
One thing to note: if this FHIR job fails midway, then you're only left with the indices that were able to be loaded. Is this the expected behavior that we want for upload?
Also, does this have performance detriments / is index generation relatively fast?
There was a problem hiding this comment.
Not sure. Maybe create a Github issue on this repo and discuss with @bwalsh and others?
There was a problem hiding this comment.
okay will write as a separate issue since an incomplete / failed job in previous versions still results in a borked ES index. See #40
Adds grip etl functions to etl pod.
checkout ACED-IDP/gen3-helm#62 branch, make update, make local.
kc get services -- If local-grip doesn't appear in the list you might have to expose the service with
kc expose deployment local-grip
also containerizing fhir-server is a part of the helm PR.
The server doesn't automatically make the graph for you on a a graph that doesn't yet exist in grip. So currently you have to exec into the pod and make the graph with a
grip create [graph-name]Should this be added ?update your sower image to feature_grip , and then you should be able to load data with the util into grip.
check grip logs.
To verify that etl has worked, exec into grip pod,
and do a query like
grip query [graph-name] 'V()'to get all of the vertices in the graph.Recent commits have attached grip ETL to feed data directly into the data framer