dbt-transform-example

Example of a dbt transform on Splitgraph Cloud with Github Actions

If you're reading this on Splitgraph: this is built from a GitHub repository and a dbt project at https://github.com/splitgraph/dbt-transform-example. It uses data from trase/supply-chains.

If you're reading this on GitHub: This repository contains a dbt project that builds a model from the data in https://splitgraph.com/trase/supply-chains and outputs it to https://splitgraph.com/splitgraph/dbt-transform-example.

How it works

• The dbt project defines a model that depends on a source named trase_supply_chains.
• The splitgraph.yml file maps this data source to the dataset at https://splitgraph.com/trase/supply-chains.
• The GitHub Action workflow generates a temporary clone link to this repository (using $GITHUB_TOKEN) and injects it into splitgraph.credentials.yml
• It then runs sgr cloud sync with the --wait parameter to submit the dbt model to Splitgraph Cloud and wait for it to complete.
• Splitgraph Cloud alters the model to point to trase/supply-chains and runs it against that dataset, snapshotting the output into a versioned data image.
• The result is at https://splitgraph.com/splitgraph/dbt-transform-example.

Using this in your own projects

• Fork this repository
• Get a pair of Splitgraph API credentials on the settings page and set them up as GitHub Secrets:
  • SPLITGRAPH_API_KEY
  • SPLITGRAPH_API_SECRET
• Change splitgraph/dbt-transform-example in splitgraph.yml and the GitHub Action workflow file to use your own username (e.g. someuser/dbt-transform-example)
• Run the workflow manually

Local development with the Splitgraph engine

• Get sgr from the releases page
• Set up an engine (guide)
• Install dbt
• Set up the dbt Splitgraph profile:

$ cat ~/.dbt/profiles.yml 
splitgraph:
  outputs:
    prod:
      dbname: splitgraph
      host: 127.0.0.1
      pass: password
      port: 6432
      schema: dbt_transform_example
      threads: 32
      type: postgres
      user: sgr
  target: prod

• Set up the dataset: sgr clone trase/supply-chains:latest && sgr checkout --layered trase/supply-chains:latest
• Run dbt: dbt build
• The resultant model will be in dbt_transform_example.top_exporter_metrics. You can connect to the engine with a normal PostgreSQL client by running psql $(sgr config -n).