(Question) Adding Variants/Evidences via API

[NThio]

Hi, is it possible to add variant and evidence records programmatically via API? The examples at (https://github.com/griffithlab/civic-api-client) and the API docs (http://griffithlab.org/civic-api-docs/) only describes queries only.

[susannasiebert]

We currently do not support programmatically adding variant or evidence records via the API to our production database. If you would like to add data to a local CIViC instance, we recommend a Ruby script.

[NThio]

Thank you Susanna, we are quite new on Ruby/Rails, do you have examples or suggestion of 'good practices' for performing bulk data imports?

We have in-house repository to be shared with collaborators, and currently evaluating if setting up local CIViC instance for sharing, and push our in-house repository to the local CIViC instance is a feasible option for us.
We reckon this approach give more control from our side at the initial stage, once is matured we'll look for pushing it to the CIViC production.

[susannasiebert]

I don't have any example at hand but my colleague @acoffman might. He has done data import from spreadsheets for CIViC in the past. What format is the data in currently in?

In regards to your workflow, are you talking about pushing data to the CIViC production database? We'd have to come up with a strategy for doing so. We have done bulk data imports from trusted source before but we strongly prefer data import through the web interface as bulk imports tend to be laborious and error prone.

May I ask what organization you work for?

[NThio]

For local CIViC instance, is there facility for data import through web interface?

There are a few sources from spreadsheets and mysql database; grooming the structure to fit CIViC (optimistically) should be doable.

I mentioned pushing data to CIViC production is a long term prospect, at the moment we aim to kickstart unifying collaboration efforts via local CIViC instance. Therefore for short and medium term, we need either data import via web interface, or bulk imports for our local instance. Do you mind to share more on your strategy on getting the data from trusted sources? I suppose this is the same scenario we are facing, although locally.

[susannasiebert]

There are a few different data objects in the CIViC schema the most import being genes, variants, and evidence_items. Evidence items can be added through the civic client web interface, e.g. https://civicdb.org/add/evidence/basic. Variants get created as part of evidence item creation if the variant doesn't yet exist in the database. Genes are the only object that would need to be pre-populated in your local instance. In CIViC we use Entrez as the canonical gene source so this import could happen in a Ruby script and utilizing the Entrez API. At a minimum the CIViC database schema requires the fields entrez_id, name, official_name (description in Entrez), and description (summary in Entrez). I wasn't involved in this initial import into our database but I will see if I can dig up the scripts that were used for this.

For evidence items, we recommend using the above endpoint on your local instance instead of doing a bulk import as that interface will give you the most features and will be the easier and less error-prone than a bulk import.

[NThio]

I see, thanks Susanna - if you could an example of gene import scripts that will be very helpful for us. I'm with PeterMacCallum Cancer Centre (Australia) by the way; I'll have a look on data schema and workflow in CIViC closer.

[kkrysiak]

@NThio I don't have the code at the moment but I do know we get them from the NCBI Entrez API. https://www.ncbi.nlm.nih.gov/books/NBK25501/