Annotation Versioning Feature

[fatimarabab]

• I have performed a self-review of my code.
• My code follows the style guidelines enforced by static analysis tools.
• If it is a core feature, I have added thorough tests.
• My changes generate no new warnings.
• New and existing unit tests pass locally with my changes.
• Will this be part of a product update? If yes, please write one phrase about this update.

Pull Request Details

Wrike ticket - Get latest version of the dataset within the annotation unit

Changes made:

• [Epic] As a scientist, I want to access annotated dataset versions in order to reference versioned annotations and make accurate citations of datasources
  • [Story] As a computational-biologist, I want to retrieve the versions of the datasets & corresponding data sources so I can streamline citing the Analysis Case accurately
  • [Story] As a developer, I want to save annotations for latest versions of datasources as well as previously annotated datasets of older versions of datasets, in order to display accurate and up to date annotations for cases

To Review:

• Static Analysis by Reviewer
• Annotations are rendering correctly in local deployment of Rosalution.
  To check this run the following commands:

     docker compose down
     docker system prune -a --volumes
     docker compose up --build -d

• Navigate to https://local.rosalution.cgds/rosalution/login. Login as "developer".

• Click on any CPAM case, navigate to its Annotation view and ensure annotations are loading.

• Verify importing a new analysis annotates all of its genomic units & renders them

  • Watch logs for logged messages before importing the duplicated fixture.

        docker logs --follow rosalution-backend-1

  • Import the C-PAM0042.json into Rosalution from ./etc/fixtures/import/
  • Navigate to https://local.rosalution.cgds/rosalution/login. Login using your 'developer.
  • Confirm 'CPAM0042' annotations are rendering
  • Confirm in the Rosalution database that analysis CPAM0042 has a 'manifest'

        docker compose exec -it rosalution-db mongosh rosalution_db
        db.analyses.find({'name': 'CPAM0042'}, {'manifest': 1})

  
  

• Verify importing new analysis that has same genomic units as an existing analysis, doesn't duplicate existing annotations and creates not annotations if there is a different version calculated

  • Import the duplicated analysis 'dev-test-analysis-C-PAM0084-duplicate.jsonfrom./etc/fixtures/import/`
  • Confirm Annotations are rendering correctly and same for both CPAM0084 and CPAM0084duplicate on the Rosalution UI.
    • Confirm Logs:
    • Right after importing the duplicated analysis, see the logs notating that 'Annotations Exists'
    • Version tasks are being created and calculated.
    • Confirm existing annotations are not re-annotated, as this is a duplicated analysis fixture of an existing case. Annotations for datasets using HPO as its datasource will re-annotate as they are versioned according to the date when annotated.
    • The logs will look as follows:

  
  

  • Confirm genomic units added new annotations for different data sources and versions

        docker compose exec -it rosalution-db mongosh rosalution_db
        db.genomic_units.aggregate([ {$match: {'hgvs_variant': 'NM_001360016.2:c.563C>T'}}, {$unwind: '$transcripts'}, {$project: {_id: 0, 'hgvs_variant': 1, 'transcript': '$transcripts'}}]);

  

        docker compose exec -it rosalution-db mongosh rosalution_db
        db.genomic_units.find({'gene': 'G6PD'});

  

  • Confirm the new analysis that has existing genomic units has its own manifest

     docker compose exec -it rosalution-db mongosh rosalution_db
     db.analyses.find({'name': 'CPAM0084duplicate'}, {'manifest': 1, _id: 0})

• Verify Genomic Units and Analyses migrate successfully to use an annotation manifest for rendering annotations

  • Restore a production backed up database on your local deployment from UAB LTS. Visit Research Compute LTS Instructions for setup instructions so the restore script can use rclone to restore the latest backed up database. Use 'cgdslts' credentials shared in Keeper.
    After rclone config is finished, restore a prod backed up database from LTS.

     # Make sure you are already deployed locally
     cd etc/database
     ./restore-database-lts.sh rosalution-rosalution-db-1 

  • Run the migration script to generate an annotation manifest for each Rosalution analysis and genomic units.

    cd ./etc/
    docker exec -it rosalution-rosalution-db-1 mongosh --file /tmp/fixtures/create-annotation-manifest.js

  • Restart backend service, so logout 'developer' to prevent an infinite auth loop IMPORTANT

    docker compose restart backend

  • Navigate to https://local.rosalution.cgds/rosalution/login. Login using your BlazerID instead of 'developer.

  • Confirm many different Rosalution analyses render their annotations

  • Inspect database to confirm analyses have manifests now

    ```bash
    docker compose exec -it rosalution-db mongosh rosalution_db
    db.analyses.find({'name': 'CPAM0102 (LRRCC1)'}, {'manifest': 1, _id: 0})
    ```
    

• All Unit Tests pass.

• All Github Actions checks have passed.

[JmScherer]

Annotations are rendering correctly in local deployment of Rosalution.

---

Verify importing a new analysis annotates all of its genomic units & renders them

Confirming that the logs properly print from the annotation process

Confirm 'CPAM0042' annotations are rendering

Confirm in the Rosalution database that analysis CPAM0042 has a 'manifest'

---

Verify importing new analysis that has same genomic units as an existing analysis, doesn't duplicate existing annotations and creates not annotations if there is a different version calculated

Confirm Annotations are rendering correctly and same for both CPAM0084 and CPAM0084duplicate on the Rosalution UI.

Right after importing the duplicated analysis, see the logs notating that 'Annotations Exists'

Version tasks are being created and calculated.

---

Confirm genomic units added new annotations for different data sources and versions

db.genomic_units.aggregate([ {$match: {'hgvs_variant': 'NM_001360016.2:c.563C>T'}}, {$unwind: '$transcripts'}, {$project: {_id: 0, 'hgvs_variant': 1, 'transcript': '$transcripts'}}]);

db.genomic_units.find({'gene': 'G6PD'});

db.analyses.find({'name': 'CPAM0084duplicate'}, {'manifest': 1, _id: 0})

---

Verify Genomic Units and Analyses migrate successfully to use an annotation manifest for rendering annotations

Run the migration script to generate an annotation manifest for each Rosalution analysis and genomic units.

Confirm many different Rosalution analyses render their annotations

Inspect database to confirm analyses have manifests now

[JmScherer]

So much stuff! It all works from my end!

[SeriousHorncat]

😸