late phase 2: ingest TRAPI KP /meta_knowledge_graph edge qualifier info, and use it in sub-querying

[colleenXu]

From bullet 4 of the changelog:

In MetaKnowledgeGraph component, added MetaEdge.qualifiers as list of new MetaQualifier types https://github.com/NCATSTranslator/ReasonerAPI/pull/387/files

Currently, we don't know what qualifier-constraints a TRAPI KP can handle; when a QEdge has qualifier-constraints (and isn't an "inferred" creative-mode query), we pass along those qualifier-constraints to the TRAPI KPs in our subqueries.

However, with TRAPI 1.4, tools can specify the qualifier-constraints they can handle for each TRAPI-MetaEdge in their /meta_knowledge_graph endpoint responses (edges section).

We want to use this info: when a QEdge has qualifier-constraints, we want BTE to only send sub-queries to TRAPI KPs that advertise that they can handle those qualifier-constraints. This should make our sub-querying behavior more efficient. We probably want to change how we create BTE-operations/MetaEdges from parsing the /meta_knowledge_graph responses of the TRAPI KPs we use.

• timing: phase 2. Ideally, we will have "real" /meta_knowledge_graph responses to work with (from the TRAPI KPs we use, their TRAPI 1.4 instances or we could ask them for mock /meta_knowledge_graph output?)
• priority: low (within phase 2).
  • We should still be able to query TRAPI KPs successfully, even if we lack this functionality
  • This ingest is a prereq for this phase 3 issue

Notes:

• keep in mind the difference between x-bte annotation with qualifiers vs TRAPI KP MetaEdges with qualifiers:
  • when we parse a x-bte operation, the qualifier info is the exact set of type_ids/values for the associations/data
  • but when we parse a TRAPI KP MetaEdge, there is ambiguity and multiple applicable_values elements are possible....so this isn't an exact-matching scenario.
    • ambiguity: associations might exist with no qualifiers or a set of type_id/value pairs. That set may have 2 pairs,...up to the total number of unique type_ids, because each type_id can only appear once in a qualifier-set.
    • For each type_id, its value could be one of the elements in its applicable_values set
• maybe relevant older notes?
  • some comments on the previous behavior we implemented
  • various issues that we probably never need to worry about given how we do TRAPI KP sub-querying

[colleenXu]

More discussion of the TRAPI MetaEdge / qualifier-set point (the "keep in mind" note above) is being done in the other issue. We'll add the clarifications / answers from the TRAPI team and Translator there.

[colleenXu]

Also, based on a quick check, none of the TRAPI KPs we ingest so far seem to specify qualifiers in their MetaEdges right now (automat kps, cohd, chp).

• COHD's dev instance seems to be on TRAPI 1.4 (we can access it through the registration we currently use, but they also registered a separate yaml for TRAPI 1.4). But I didn't see any "qualifiers" specified for the MetaEdges in its /meta_knowledge_graph response

---

And This is a "technical implementation" thought: but maybe the logic can be "exclusion":

• "if this hop has qualifier_constraints AND this TRAPI KP MetaEdge specifies qualifiers"...
  • check if the qualifier_constraints are covered by the MetaEdge qualifiers
    • so iterate through each set of type_id/value pairs....are all members of the set covered
    • If all sets ARE NOT covered, then don't do this sub-query. since the TRAPI KP is basically advertising that it doesn't support those qualifier_constraints.

[colleenXu]

Restating some info from above: this issue is probably on-hold because it (1) currently don't have TRAPI KP data to work with, (2) is affected by the "qualifier-set merging" discussion, and (3) seems optional (optimizing)

[tokebe]

Just a note: above linked PRs are currently on Dev.

[colleenXu]

Appears to be working.

Testing process

TRAPI query

{
    "message": {
        "query_graph": {
            "nodes": {
                "n0": {
                    "ids":["PUBCHEM.COMPOUND:2662"],
                    "categories":["biolink:SmallMolecule"]
                },
                "n1": {
                    "categories":["biolink:Gene"]
               }
            },
            "edges": {
                "e1": {
                    "subject": "n0",
                    "object": "n1",
                    "predicates": ["biolink:affects"],
                    "qualifier_constraints": [
                        {
                            "qualifier_set": [
                                {
                                    "qualifier_type_id": "biolink:object_direction_qualifier",
                                    "qualifier_value": "decreased"
                                }
                            ]
                        }
                    ]
                }
            }
        }
    }
}

• when the query above (which has qualifier-constraints) is sent to Automat-pharos (ITRB Prod) https://automat.transltr.io/pharos/1.4/query, there are 15 edges returned.
• when the query above is sent to BTE, BTE's console logs show that 1 sub-query is generated to Automat-pharos and 15 records are retrieved (as-expected)
• Spot-checking showed that the data from the Automat-pharos response was preserved in BTE's edges.

BTE's console logs:

  bte:call-apis:query query success, transforming hits->records... +11ms
  bte:api-response-transform:index api name Automat-pharos(Trapi v1.4.0) +32ms
  bte:api-response-transform:index api tags: translator,automat,trapi,bte-trapi +0ms
  bte:call-apis:query Successful POST https://automat.transltr.io/pharos/1.4 (1 ID): SmallMolecule > affects > Gene (obtained 15 records, took 887ms) +0ms

Edge from Automat-Pharos response

                "494976": {
                    "subject": "PUBCHEM.COMPOUND:2662",
                    "object": "NCBIGene:9536",
                    "predicate": "biolink:affects",
                    "sources": [
                        {
                            "resource_id": "infores:pharos",
                            "resource_role": "primary_knowledge_source",
                            "upstream_resource_ids": null,
                            "source_record_urls": null
                        },
                        {
                            "resource_id": "infores:automat-pharos",
                            "resource_role": "aggregator_knowledge_source",
                            "upstream_resource_ids": [
                                "infores:pharos"
                            ],
                            "source_record_urls": null
                        }
                    ],
                    "qualifiers": [
                        {
                            "qualifier_type_id": "biolink:object_aspect_qualifier",
                            "qualifier_value": "activity"
                        },
                        {
                            "qualifier_type_id": "biolink:object_direction_qualifier",
                            "qualifier_value": "decreased"
                        },
                        {
                            "qualifier_type_id": "biolink:qualified_predicate",
                            "qualifier_value": "biolink:causes"
                        }
                    ],
                    "attributes": [
                        {
                            "attribute_type_id": "biolink:Attribute",
                            "value": "IC50",
                            "value_type_id": "EDAM:data_0006",
                            "original_attribute_name": "affinity_parameter",
                            "value_url": null,
                            "attribute_source": null,
                            "description": null,
                            "attributes": null
                        },
                        {
                            "attribute_type_id": "biolink:publications",
                            "value": [
                                "PMID:26653180",
                                "PMID:27554445"
                            ],
                            "value_type_id": "EDAM:data_0006",
                            "original_attribute_name": "publications",
                            "value_url": null,
                            "attribute_source": null,
                            "description": null,
                            "attributes": null
                        },
                        {
                            "attribute_type_id": "biolink:Attribute",
                            "value": 6.37,
                            "value_type_id": "EDAM:data_0006",
                            "original_attribute_name": "affinity",
                            "value_url": null,
                            "attribute_source": null,
                            "description": null,
                            "attributes": null
                        }
                    ]
                }
            }

BTE preserves sources / attributes / qualifiers from Automat-pharos response

VS corresponding Edge from BTE's response

               "46a372050469e5a98b9596619576c134": {
                    "predicate": "biolink:affects",
                    "subject": "PUBCHEM.COMPOUND:2662",
                    "object": "NCBIGene:9536",
                    "qualifiers": [
                        {
                            "qualifier_type_id": "biolink:object_aspect_qualifier",
                            "qualifier_value": "activity"
                        },
                        {
                            "qualifier_type_id": "biolink:object_direction_qualifier",
                            "qualifier_value": "decreased"
                        },
                        {
                            "qualifier_type_id": "biolink:qualified_predicate",
                            "qualifier_value": "biolink:causes"
                        }
                    ],
                    "attributes": [
                        {
                            "attribute_type_id": "biolink:Attribute",
                            "value": "IC50",
                            "value_type_id": "EDAM:data_0006",
                            "original_attribute_name": "affinity_parameter",
                            "value_url": null,
                            "attribute_source": null,
                            "description": null,
                            "attributes": null
                        },
                        {
                            "attribute_type_id": "biolink:publications",
                            "value": [
                                "PMID:26653180",
                                "PMID:27554445"
                            ],
                            "value_type_id": "EDAM:data_0006",
                            "original_attribute_name": "publications",
                            "value_url": null,
                            "attribute_source": null,
                            "description": null,
                            "attributes": null
                        },
                        {
                            "attribute_type_id": "biolink:Attribute",
                            "value": 6.37,
                            "value_type_id": "EDAM:data_0006",
                            "original_attribute_name": "affinity",
                            "value_url": null,
                            "attribute_source": null,
                            "description": null,
                            "attributes": null
                        }
                    ],
                    "sources": [
                        {
                            "resource_id": "infores:pharos",
                            "resource_role": "primary_knowledge_source",
                            "upstream_resource_ids": null,
                            "source_record_urls": null
                        },
                        {
                            "resource_id": "infores:automat-pharos",
                            "resource_role": "aggregator_knowledge_source",
                            "upstream_resource_ids": [
                                "infores:pharos"
                            ],
                            "source_record_urls": null
                        },
                        {
                            "resource_id": "infores:biothings-explorer",
                            "resource_role": "aggregator_knowledge_source",
                            "upstream_resource_ids": [
                                "infores:automat-pharos"
                            ]
                        }
                    ]
                },

This same testing process (same query) worked to review automat hetio behavior as well (4 edges in that KP's response -> 4 records in BTE's logs and 4 edges in BTE). Automat hetio's meta_knowledge_graph shows that there's a SmallMolecule - regulates - downregulated (qualifier) Gene MetaEdge. So BTE likely used that MetaEdge to generate the sub-query; this shows that qualifier-hierarchy expansion is working (since I queried for the parent qualifier decreased).

Notes:

• COHD and CHP meta_knowledge_graph endpoint responses show no qualifier-constraints, so not able to test.

Closing issue