Analysis Sections Enhancement with Migration and Additional Section I…

…mages Attachments (#103)

* wip to refactoring to supporting multiple sections in an analysis to add attachments

* wip for analysis backend integration with the adding/updating/removing

* debugging whats going on

* got it working?

* reduced the CSS footprint for the gene box component and the dom to be simpler

* wip for consolidating the css

* conslidated work that was here

* wip for refactoring the backend

* fixed frontend linting issues

* Refactoring to make the emits work, starting work on the updating section image in the backend

* Image edit update and removal working

* Replaced the component back into the sectionbox and am chaining emits from sectionimage to sectionbox to analysis view

* fixed removing an image, it now properly reacts to the image removed

* Updated the backend analysis collection attach image to use field instead of dataset to fix image attachment

* Updated the analysis image system tests, changed system-tests/e2e/attach_pedigree_image.cy.js to be attach_analysis_section_image.cy.js

* Knocked out some system tests, working on the others. I think there's a bug, need to check

* Working out some of the system tests,
they won't work until the collapsible and empty content boxes are resolved

* fixed collasping of the section boxes in annotationview and made preliminary changes to the system test for collapsing

* Work in progress to resolve CSS and remove the force render

* Fixed most of the linting for python

* Refactored to not require the force re-render for frontend images

* frontend css changes to make progress to editing  case"
git push

* System tests passing

* disabled duplicate code check

* formatted backend code with yapf

* Frontned  unit tests updated along with additional console logging on frontend removed for debugging

* Updated test_analysis_collection.py to reflect the new tests for the analysis section images. Also removed a few functions from analysis_collection.py pretaining to pedigree. Lastly, updated some text fixtures to more accurately reflect what the structure is

* Fixed up the integration tests for the analysis_router.py

* Linting

* Removed some comments

* Linting again

* removed special coloring to help resolve cleaning up and consolidating the CSS

* Phenotips importer update and corresponding Migration Script (#102)

* Added new sections to analysis

* Modified phenotips importer to show sections for multiple genes in analysis

* progress in migration script

* Fixed backend linting

* Fixed up phenotips_importer.py and added a test

* Added migration script to reconfigure analysis sections. Gene sections throwing error now

* Updated the CSS for the section text so that it wraps and doesn't go beyhond the bounds of its container.

* Updated gene sections to reconfigure where HPO terms are displayed

* Removed duplicate Clinical History/HPO terms loop

* added the pedigree image if it exists

* Removed print statements and included pedgree image migration

* added type to sections

* Added images-dataset type for fields

* script fix, verified can only run it once succesfully

* Updated the base fixtures to use the new analysis system

* Updated phenotips importer to have field names match header

* backend linting fixes

* Updated the system tests for the image attaching

* Fixed the formatting that came up with yapf

* Resovled the system tests

* Added some logging to the script to be informative of the output and fixed a few things found in the process

* Pushign up fix for the clinical history

---------

Co-authored-by: James Scherer <jscherer@uab.edu>
Co-authored-by: Angelina Elizabeth Uno-Antonison <ange.unoantonison@gmail.com>

* Updating system test to match the miggrated fixtures

---------

Co-authored-by: James Scherer <jscherer@uab.edu>
Co-authored-by: Rabab Fatima <rfatima@uab.edu>

backend/.pylintrc

@@ -419,7 +419,8 @@ disable=useless-return,
         suppressed-message,
         useless-suppression,
         deprecated-pragma,
-        use-symbolic-message-instead
+        use-symbolic-message-instead,
+        duplicate-code
 
 # Enable the message, report, category or checker with the given id(s). You can
 # either give multiple identifier separated by comma (,) or put this option

backend/src/core/phenotips_importer.py

@@ -14,8 +14,7 @@ def __init__(self, analysis_collection, genomic_unit_collection):
 
     def import_phenotips_json(self, phenotips_json_data):
         """Imports the phenotips json data into the database"""
-        if not isinstance(phenotips_json_data, dict):
-            phenotips_json_data = phenotips_json_data.dict()
+
         phenotips_variants = []
         variant_annotations = [
             "inheritance", "zygosity", "interpretation", "transcript", "cdna", "reference_genome", "protein", "gene"
@@ -63,7 +62,7 @@ def import_genomic_unit_collection_data(data, data_format):
         elif data_format == "gene":
             genomic_data = {"gene_symbol": data['gene'], "gene": data['gene'], "annotations": []}
         else:
-            warnings.warn("Invalid data format for import_genomic_unit_collection_data method")
+            warnings.warn("Invalid data format for import_genomic_unit_collection_data method", UserWarning)
             return None
         return genomic_data
 
@@ -74,24 +73,20 @@ def import_analysis_data(self, phenotips_json_data, phenotips_variants, phenotip
             "name": str(phenotips_json_data["external_id"]).replace("-", ""), "description": "", "nominated_by": "",
             "genomic_units": [], "sections": [{
                 "header": 'Brief', "content": [
-                    {"field": 'Nominator', "value": []},
-                    {"field": 'Participant', "value": []},
-                    {"field": 'Phenotype', "value": []},
-                    {"field": 'Model of Interest', "value": []},
-                    {"field": 'Goals', "value": []},
-                    {"field": 'Proposed Model/Project', "value": []},
+                    {"type": "section-text", "field": 'Nominator', "value": []},
+                    {"type": "section-text", "field": 'Participant', "value": []},
+                    {"type": "section-text", "field": 'Phenotype', "value": []},
                 ]
             }, {
                 "header": 'Clinical History', "content": [
-                    {"field": 'Clinical Diagnosis', "value": []},
-                    {"field": 'Affected Individuals Identified', "value": []},
-                    {"field": 'Sequencing', "value": []},
-                    {"field": 'Testing', "value": []},
-                    {"field": 'Systems', "value": []},
-                    {"field": 'HPO Terms', "value": [self.extract_hpo_terms(phenotips_json_data["features"])]},
-                    {"field": 'Additional Details', "value": []},
+                    {"type": "section-text", "field": 'Clinical Diagnosis', "value": []},
+                    {"type": "section-text", "field": 'Affected Individuals Identified', "value": []},
+                    {"type": "section-text", "field": 'Sequencing', "value": []},
+                    {"type": "section-text", "field": 'Testing', "value": []},
+                    {"type": "section-text", "field": 'Systems', "value": []},
+                    {"type": "section-text", "field": 'Additional Details', "value": []},
                 ]
-            }, {"header": 'Pedigree', "content": []}]
+            }, {"header": 'Pedigree', "content": [{"type": "images-dataset", "field": "Pedigree", "value": []}]}]
         }
 
         for phenotips_gene in phenotips_genes:
@@ -117,6 +112,41 @@ def import_analysis_data(self, phenotips_json_data, phenotips_variants, phenotip
 
             analysis_data['genomic_units'].append(analysis_unit)
 
+        for genomic_unit in analysis_data['genomic_units']:
+            if genomic_unit['gene']:
+                new_sections = [{
+                    "header": str(genomic_unit["gene"] + " Gene to Phenotype"), "content": [
+                        {
+                            "type": "images-dataset", "field": str(genomic_unit["gene"] + " Gene to Phenotype"),
+                            "value": []
+                        },
+                        {
+                            "type": "section-text", "field": 'HPO Terms',
+                            "value": [self.extract_hpo_terms(phenotips_json_data["features"])]
+                        },
+                    ]
+                }, {
+                    "header": str(genomic_unit["gene"] + " Molecular Mechanism"), "content": [{
+                        "type": "section-text", "field": str(genomic_unit["gene"] + " Molecular Mechanism"), "value": []
+                    }]
+                }, {
+                    "header": str(genomic_unit["gene"] + " Function"), "content": [
+                        {"type": "images-dataset", "field": str(genomic_unit["gene"] + " Function"), "value": []},
+                    ]
+                }]
+                analysis_data['sections'].extend(new_sections)
+
+        model_goals_section = {
+            "header": 'Model Goals', "content": [
+                {"type": "section-text", "field": 'Model of Interest', "value": []},
+                {"type": "section-text", "field": 'Goals', "value": []},
+                {"type": "section-text", "field": 'Proposed Model/Project', "value": []},
+                {"type": "section-text", "field": 'Existing Collaborations', "value": []},
+                {"type": "section-text", "field": 'Existing Funding', "value": []},
+            ]
+        }
+        analysis_data['sections'].append(model_goals_section)
+
         return analysis_data
 
     @staticmethod

backend/src/models/analysis.py

@@ -22,10 +22,11 @@ class GenomicUnit(BaseModel):
     variants: List = []
 
 
-class Section(BaseModel):
+class Section(BaseModel, frozen=True):
     """The sections of case notes associated with an analysis"""
 
     header: str
+    attachment_field: Optional[str] = None
     content: List = []
 
 

backend/src/repository/analysis_collection.py

@@ -170,36 +170,6 @@ def attach_supporting_evidence_link(self, analysis_name: str, link_name: str, li
 
         return updated_document
 
-    def add_pedigree_file(self, analysis_name: str, file_id: str):
-        """ Adds a pedigree file to an analysis """
-        updated_document = self.collection.find_one({"name": analysis_name})
-        if "_id" in updated_document:
-            updated_document.pop("_id", None)
-
-        updated_section = None
-        for section in updated_document['sections']:
-            if section["header"] == "Pedigree":
-                if len(section["content"]) == 0:
-                    updated_section = {
-                        "field": 'image',
-                        "value": [str(file_id)],
-                    }
-                    section["content"].append(updated_section)
-                else:
-                    section['content'][0]['value'] = [str(file_id)]
-
-                updated_section = section
-
-        if None is updated_section:
-            raise ValueError("No pedigree section to attach image to.")
-
-        self.collection.find_one_and_update(
-            {"name": analysis_name},
-            {'$set': updated_document},
-        )
-
-        return updated_section
-
     def get_genomic_units(self, analysis_name: str):
         """ Returns the genomic units for an analysis with variants displayed in the HGVS Nomenclature """
         genomic_units_return = {"genes": {}, "variants": []}
@@ -265,33 +235,96 @@ def remove_supporting_evidence(self, analysis_name: str, attachment_id: str):
         updated_document.pop("_id", None)
         return updated_document
 
-    def get_pedigree_file_id(self, analysis_name: str):
-        """ Returns the pedigree file id for an analysis """
-        analysis = self.collection.find_one({"name": analysis_name})
-        if not analysis:
-            raise ValueError(f"Analysis with name {analysis_name} does not exist")
+    def add_section_image(self, analysis_name: str, section_name: str, field_name: str, file_id: str):
+        """
+        Adds an image to a section within an analysis specified by the the name of the section (the header) and
+        """
+        updated_document = self.collection.find_one({"name": analysis_name})
+        if "_id" in updated_document:
+            updated_document.pop("_id", None)
 
-        for section in analysis['sections']:
-            if section["header"] == "Pedigree":
-                if len(section["content"]) == 0:
-                    raise ValueError(f"Analysis {analysis_name} does not have a pedigree file")
-                return section['content'][0]['value'][0]
-        raise ValueError(f"Analysis {analysis_name} does not have a pedigree section")
+        updated_section = None
+        for section in updated_document['sections']:
+            if section_name == section['header']:
+                updated_section = section
 
-    def remove_pedigree_file(self, analysis_name: str):
-        """ Removes a pedigree file from an analysis """
-        analysis = self.collection.find_one({"name": analysis_name})
-        for section in analysis["sections"]:
-            if section["header"] == "Pedigree":
-                section["content"] = []
-        updated_document = self.collection.find_one_and_update(
+        if None is updated_section:
+            raise ValueError(
+                f"'{section_name}' does not exist within '{analysis_name}'. Unable to attach image to '{field_name}' \
+                field in section '{section_name}"
+            )
+
+        for content_row in updated_section['content']:
+            if content_row["field"] and content_row["field"] == field_name:
+                content_row["value"].append({'file_id': str(file_id)})
+
+        self.collection.find_one_and_update(
             {"name": analysis_name},
-            {"$set": {"sections": analysis["sections"]}},
-            return_document=ReturnDocument.AFTER,
+            {'$set': updated_document},
         )
-        # remove the _id field from the returned document since it is not JSON serializable
-        updated_document.pop("_id", None)
-        return updated_document
+
+        return updated_section
+
+    def update_section_image(
+        self, analysis_name: str, section_name: str, field_name: str, file_id: str, file_id_old: str
+    ):
+        """ Accepts a new and old file id then updates the section image """
+        updated_document = self.collection.find_one({"name": analysis_name})
+
+        if "_id" in updated_document:
+            updated_document.pop("_id", None)
+
+        updated_section = None
+        for section in updated_document['sections']:
+            if section_name == section['header']:
+                updated_section = section
+
+        if None is updated_section:
+            raise ValueError(
+                f"'{section_name}' does not exist within '{analysis_name}'. \
+                Unable to attach image to '{field_name}' field in section '{section_name}"
+            )
+
+        for content_row in updated_section['content']:
+            if content_row['field'] and content_row['field'] == field_name:
+                for i in range(len(content_row['value'])):
+                    if content_row['value'][i]['file_id'] == file_id_old:
+                        content_row['value'].pop(i)
+                        content_row["value"].append({'file_id': str(file_id)})
+                        break
+
+        self.collection.find_one_and_update({'name': analysis_name}, {'$set': updated_document})
+
+        return updated_section
+
+    def remove_analysis_section_file(self, analysis_name: str, section_name: str, field_name: str, file_id: str):
+        """ Accepts a file id and removes the reference from corresponding analysis section """
+        updated_document = self.collection.find_one({"name": analysis_name})
+
+        if "_id" in updated_document:
+            updated_document.pop("_id", None)
+
+        updated_section = None
+        for section in updated_document['sections']:
+            if section_name == section['header']:
+                updated_section = section
+
+        if None is updated_section:
+            raise ValueError(
+                f"'{section_name}' does not exist within '{analysis_name}'. Unable to attach image to '{field_name}' \
+                field in section '{section_name}"
+            )
+
+        for content_row in updated_section['content']:
+            if content_row['field'] and content_row['field'] == field_name:
+                for i in range(len(content_row['value'])):
+                    if content_row['value'][i]['file_id'] == file_id:
+                        content_row['value'].pop(i)
+                        break
+
+        self.collection.find_one_and_update({'name': analysis_name}, {'$set': updated_document})
+
+        return updated_section
 
     def attach_third_party_link(self, analysis_name: str, third_party_enum: str, link: str):
         """ Returns an analysis with a third party link attached to it """

backend/src/repository/genomic_unit_collection.py

@@ -159,8 +159,6 @@ def update_genomic_unit_file_annotation(self, genomic_unit, data_set, annotation
 
         genomic_unit_document = self.find_genomic_unit(genomic_unit)
 
-        # print(genomic_annotation_value)
-
         for annotation in genomic_unit_document['annotations']:
             if data_set in annotation:
                 for i in range(len(annotation[data_set][0]['value'])):