Importing Tools and Data

The Saturn UI provides interfaces for importing tools (e.g., WDL-based workflows) and data (e.g., samples and metadata) into a project. The purpose of these interfaces is to allow tool and data repositories to export into this system without having to build a UI flow for doing so. Instead, each repository can simply re-use the UI created explicitly for this purpose within Saturn.

Importing a Tool

Once a user has selected a tool from your repository, direct the user to a URL of this form:

https://<saturn base url>/#import-tool/[type]/[location]/[organization]/[namespace]/[name]:[version]

Example:

https://<saturn base url>/#import-tool/dockstore/github.com/DataBiosphere/topmed-workflows/Functional_Equivalence:1.7.0

Importing Data

Try it out with Terra Data Explorer

• Go to https://test-data-explorer.appspot.com
• [Optional] Select a cohort by checking some boxes
• Click Send to Terra button on bottom right
• [Optional] Terra Data Explorer writes entities to a JSON file on GCS. You can inspect this file.
  • Copy the url parameter from the import-data url.
  • Decode it.
  • Open a new Chrome tab and paste the decoded url.
  • If you selected a cohort, this file includes a sample_set entity.
  • Note this url expires after 5 minutes.
• Select a workspace (or create a workspace).
• You can now see your imported data in the Data tab.

How to import data to Terra

You can import any entity to Terra. You may follow the FireCloud Data Model, or you may import any other entity. For example, the Terra Data Explorer exports:

• samples and sample_sets, which are part of the FireCloud data model
• BigQuery tables and cohorts, which are not

Note that unlike FireCloud, Terra does not enforce that participants referenced in samples must also be imported.

Your server must:

• Create a file on GCS or AWS with JSON list of entities. Each entity is the JSON passed into https://rawls.dsde-prod.broadinstitute.org/#!/entities/create_entity You can see a sample file here.
• Create a signed URL of this file, so Terra client can fetch it.
• Navigate to https://app.terra.bio/#import-data?format=entitiesJson&url=SIGNED_URL&ad=AUTH_DOMAIN
  • url parameter is URL-encoded signed url
  • Optional ad parameter is an Authorization Domain. If this is set, only workspaces with the given authorization domain will be shown. If your dataset is private, you must use this parameter to ensure that only authorized users see the exported data.

See the Terra Data Explorer implementation here and here.