find_trees failing

[snacktavish]

Example from the API docs is not working,
(https://github.com/OpenTreeOfLife/opentree/wiki/Open-Tree-of-Life-APIs#find_trees)

curl -X POST http://api.opentreeoflife.org/v2/studies/find_trees -H "content-type:application/json" -d '{"property":"ot:ottTaxonName","value":"Garcinia"}'

returns
{
"matched_studies" : [ ]
}

as do several other searches for tips that are in trees (using either ot:ottTaxonName or ot:originalLabel), although some searches do return results, e.g.

curl -X POST http://api.opentreeoflife.org/v2/studies/find_trees -H "content-type:application/json" -d '{"property":"ot:ottTaxonName","value":"Homo"}'

[snacktavish]

find_studies example also not working

curl -X POST http://api.opentreeoflife.org/v2/studies/find_studies -H "content-type:application/json" -d '{"property":"ot:studyId","value":"pg_719","verbose":true}'

[dwinter]

Hi all,

Just add another example, study pg_2655 exists, but doesn't seem to be discover-able using find_studies

curl http://api.opentreeoflife.org/v2/study/pg_2655/meta

Produces this record, which include ^ot:focalCladeOTTTaxonName": "Cichlidae"

{"nexml": {"^ot:messages": {"message": []}, "^ot:studyPublicationReference": "Friedman, M., B. P. Keck, A. Dornburg, R. I. Eytan, C. H. Martin, C. D. Hulsey, P. C. Wainwright, T. J. Near. 2013. Molecular and fossil evidence place the origin of cichlid fishes long after Gondwanan rifting. Proceedings of the Royal Society B: Biological Sciences 280 (1770): 20131733", "^ot:candidateTreeForSynthesis": ["tree6181"], "^ot:focalCladeOTTTaxonName": "Cichlidae", "@xmlns": {"xsd": "http://www.w3.org/2001/XMLSchema#", "ot": "http://purl.org/opentree-terms#", "$": "http://www.nexml.org/2009", "nex": "http://www.nexml.org/2009", "xsi": "http://www.w3.org/2001/XMLSchema-instance"}, "^ot:curatorName": "Joseph Brown", "@id": "study", "^ot:otusElementOrder": ["otus2655"], "^ot:treesElementOrder": ["trees2655"], "@nexml2json": "1.2.1", "treesById": {"trees2655": {"treeById": {"tree6182": null, "tree6181": null}, "^ot:treeElementOrder": ["tree6181", "tree6182"], "@otus": "otus2655"}}, "^ot:studyYear": 2013, "otusById": {"otus2655": {}}, "@nexmljson": "http://purl.org/opentree/nexson", "^ot:annotationEvents": {"annotation": []}, "^ot:studyId": "pg_2655", "^ot:studyPublication": {"@href": "http://dx.doi.org/10.1098/rspb.2013.1733"}, "^ot:agents": {"agent": [{"@description": "validator of NexSON constraints as well as constraints that would allow a study to be imported into the Open Tree of Life's phylogenetic synthesis tools", "@name": "validate_ot_nexson.py", "@version": "0.0.4a", "@url": "https://github.com/OpenTreeOfLife/peyotl", "invocation": {"otherProperty": [{"name": "pythonVersion", "value": "2.7.5+"}, {"name": "pythonImplementation", "value": "CPython"}], "commandLine": ["--embed", "--agent-only"]}, "@id": "peyotl-validator"}]}, "@generator": "Phylografter nexml-json exporter", "^ot:tag": ["ingroup added"], "^ot:focalClade": 811925, "@version": "0.9"}}

But searching for a study with this property returns nothing

curl -X POST http://api.opentreeoflife.org/v2/studies/find_studies \
    -H "content-type:application/json" \
    -d    '{"property":"ot:focalCladeOTTTaxonName","value":"Cichlidae","verbose":true}'

{
  "matched_studies" : [ ]
}

[jar398]

I think this is the same as #34

[jar398]

Fixed by OpenTreeOfLife/ot-base#17 and OpenTreeOfLife/ot-base#16 I think - please check and close if satisfied

[jar398]

dup of #34

[snacktavish]

Find_studies is working better, but I'm still getting mysterious failures in find_trees
e.g.
Searching for Limnothrix sp B15, which is mapped and is in this tree (conveniently near the top)
https://tree.opentreeoflife.org/curator/study/view/pg_2739/?tab=trees&tree=tree6601
https://tree.opentreeoflife.org/opentree/opentree3.0@3678861/Limnothrix-sp-B15

using its ottId returns no studies

curl -X POST http://api.opentreeoflife.org/v2/studies/find_trees -H "content-type:application/json" -d '{"property":"ot:ottId","value":"993343"}'

returns

"matched_studies" : [ ]

But for other ids from the same tree I am getting the correct behavior

e.g.
Leptolyngbya sp. FYG
https://tree.opentreeoflife.org/opentree/opentree3.0@1063423

 curl -X POST http://api.opentreeoflife.org/v2/studies/find_trees -H "content-type:application/json" -d '{"property":"ot:ottId","value":"1063423"}'

returns the correct

{
"matched_studies" : [ {
"ot:studyId" : "pg_2739",
"matched_trees" : [ {
"oti_tree_id" : "pg_2739_tree6601",
"nexson_id" : "tree6601"
} ]
} ]

If I get the tree using

curl http://api.opentreeoflife.org/v2/study/pg_2739.tre/?tip_label=ot:ottid > cyan.tre

They're both there.
Any ideas?

[josephwb]

Example "miss": @hdliv was looking for trees with any of the following taxa:

Helianthus annuus, Ipomoea purpurea, Solanum vescum, Jaltomata sinuosa, Fucus distichus, Halimeda Pseudanabaena limnetica, Microcystis aeruginosa, Trichodesmium hildebrandtii, Pseudanabaena tremula UTCC 471, Eragrostis plana, Cuspidothrix issatschenkoi, Baccharis trimera, Dolichospermum spiroides

but got no hits. However this tree contains a number of these taxa (definitely the first four; I have not tried them all). Here is an example call:

curl -X POST https://api.opentreeoflife.org/v3/studies/find_trees -H "content-type:application/json" -d '{"property":"ot:ottTaxonName","value":"Helianthus annuus","verbose":true}'
{
  "matched_studies" : [ ]
}

Searching for just "Helianthus" gets one hit (pg_607), but is not the tree above.

Interestingly, a search for Aves does find the "missing" tree above.

[josephwb]

Another example (a better one, since it involves Aves 😉):

curl -X POST https://api.opentreeoflife.org/v3/studies/find_trees -H "content-type:application/json" -d '{"property":"ot:ottId","value":"81461","verbose":true}' > bird_studies.txt

fails to find this study, a supertree of birds!

[josephwb]

Another bird study that is not found.

[josephwb]

And another Aves supertree not found.

[mtholder]

I'm not sure we need more examples of failure given that we know the tool is hopelessly buggy and we are replacing it rather than trying to find a specific bug.

[josephwb]

Ok. Jonathan was looking for an example.

[jar398]

Thanks @josephwb . @mtholder it's not @josephwb's fault, I just wanted a few examples since the issue had seen no action in over a year & I wanted confirmation that the problem was still present. Now we have them and can let it rest.

[bomeara]

Thanks for digging into this, folks. It came up in the context of phylotastic, esp datelife.org: since I rely on pulling chronograms from OpenTree, the issue here affects the chronogram store I have. I'm not sure how long it'll require to have a solution to this problem (not a criticism -- I just don't know the work required and resources available) -- should I look at just dealing with the tree store directly rather than through the interface, or is there some other obvious workaround?

[mtholder]

it isn't too hard to grep through the data store if you are looking for a stored property. But higher level concepts like "this study includes a representative of this taxon" are harder to deal with. I do think that @kcranston is pretty close to having the replacement tool ready.