[FEAT] option to query the Discovery API for ERD data via GraphQL

[datnguye]

Is your feature request related to a problem? Please describe.

• Support GraphQL query - built-in query and allow custom query
• New command to connect to dbt Cloud Discovery API

Related to #63

Describe the solution you'd like
The command would be something like dbterd run-metadata [--dbt-cloud-query-file-path /path/to/erd.gql]

Describe alternatives you've considered
Use the existing option with dbt Cloud Admistrative API to download files and lately using dbterd run --dbt-cloud ... command

[datnguye]

• Apollo Server: https://metadata.cloud.getdbt.com/graphql
• Querying
  See https://docs.getdbt.com/docs/dbt-cloud-apis/discovery-querying
• Schema
  See https://docs.getdbt.com/docs/dbt-cloud-apis/discovery-schema-environment
• Sample GraphQL query:

query Example(
  $environmentId: BigInt!, 
  $modelFirst: Int!, 
  $modelAfter: String, 
  $sourceFirst: Int!, 
  $sourceAfter: String, 
  $exposureFirst: Int!, 
  $exposureAfter: String, 
  $testsFirst: Int!, 
  $testAfter: String
) {
  environment(id: $environmentId){
    applied {
      tests(first: $testsFirst, after: $testAfter) {
        edges {
          node {
            uniqueId,
            parents { uniqueId, resourceType }
          }
        }
        pageInfo {
          startCursor,
          endCursor,
          hasNextPage
        }
        totalCount 
      }
      models(first: $modelFirst, after: $modelAfter){ 
        edges {
          node {
            uniqueId, name, description,
            database, schema, alias,
            tests {uniqueId, columnName, testMetadata {
              name,
              kwargs
            }},
            catalog {columns {name, description, type}},
          }
        }
        pageInfo {
          startCursor,
          endCursor,
          hasNextPage
        }
        totalCount 
      }
      sources(first: $sourceFirst, after: $sourceAfter){
        edges { 
          node {
            uniqueId, name, description,
            database, schema,
            tests {columnName, name, uniqueId, testMetadata {
              name,
              kwargs
            }},
            catalog {columns {name, description, type}},
          }
        }
        pageInfo {
          startCursor,
          endCursor,
          hasNextPage
        }
        totalCount 
      }
      exposures(first: $exposureFirst, after: $exposureAfter) {
        edges { 
          node {
            uniqueId, name, description, 
            parents { uniqueId }
          }
        }
        pageInfo {
          startCursor,
          endCursor,
          hasNextPage
        }
        totalCount 
      }
    }
  }
}

Variables:

{
  "environmentId": ???,
  "modelFirst": 500,
  "modelAfter": null,
  "sourceFirst": 500,
  "sourceAfter": null,
  "exposureFirst": 500,
  "exposureAfter": null,
  "testsFirst": 500,
  "testAfter": null,
}

Header:

{ "Authorization": "Bearer dbtc_xxx" }

[datnguye]

Introducing new command: dbterd run-metadata

Sample logs:

2024-02-03 12:14:18,453 - dbterd - INFO - Run with dbterd==1.0.0 (main.py:54)
2024-02-03 12:14:18,454 - dbterd - INFO - Looking for the query in: C:\Sources\dbterd\dbterd\adapters\dbt_cloud/include/erd_query.gql (query.py:12)
2024-02-03 12:14:18,454 - dbterd - DEBUG - Getting erd data...[URL: https://metadata.cloud.getdbt.com/graphql/graphql/, VARS: {'environment_id': '284661', 'model_first': 500, 'source_first': 500, 'exposure_first': 500, 'test_first': 500}] (graphql.py:26)
2024-02-03 12:14:19,971 - dbterd - DEBUG - Completed [status: 200] (graphql.py:34)
2024-02-03 12:14:19,973 - dbterd - INFO - Result: 5 models, 2 sources, 1 exposures, 21 tests (discovery.py:93)
Parsing - TODO
2024-02-03 12:14:20,001 - dbterd - INFO - Collected 0 table(s) and 0 relationship(s) (test_relationship.py:42)
2024-02-03 12:14:20,002 - dbterd - INFO - C:\Sources\dbterd\target (base.py:182)

With pagination dbterd run-metadata (page size = 10)

2024-02-03 15:12:23,328 - dbterd - INFO - Run with dbterd==1.0.0 (main.py:54)
2024-02-03 15:12:23,329 - dbterd - INFO - Looking for the query in: C:\Sources\dbterd\dbterd\adapters\dbt_cloud/include/erd_query.gql (query.py:12)
2024-02-03 15:12:23,330 - dbterd - DEBUG - Getting erd data...[URL: https://metadata.cloud.getdbt.com/graphql/graphql/, VARS: {'environment_id': '284661', 'model_first': 10, 'source_first': 10, 'exposure_first': 10, 'test_first': 10}] (graphql.py:26)
2024-02-03 15:12:24,898 - dbterd - DEBUG - Completed [status: 200] (graphql.py:34)
2024-02-03 15:12:24,904 - dbterd - INFO - Result: 5 models, 2 sources, 1 exposures, 10 tests (discovery.py:93)
2024-02-03 15:12:24,905 - dbterd - DEBUG - Getting erd data...[URL: https://metadata.cloud.getdbt.com/graphql/graphql/, VARS: {'environment_id': '284661', 'model_first': 10, 'source_first': 10, 'exposure_first': 10, 'test_first': 10, 'model_after': 'Y3Vyc29yOm1vZGVsLmphZmZsZV9zaG9wLnN0Z19wYXltZW50cw==', 'source_after': 'Y3Vyc29yOnNvdXJjZS5qYWZmbGVfc2hvcC5tYXJ0X2phZmZsZV9zaG9wLm9yZGVycw==', 'exposure_after': 'Y3Vyc29yOmV4cG9zdXJlLmphZmZsZV9zaG9wLmN1c3RvbWVyX29yZGVycw==', 'test_after': 'Y3Vyc29yOnRlc3QuamFmZmxlX3Nob3Aubm90X251bGxfb3JkZXJzX2dpZnRfY2FyZF9hbW91bnQuNDEzYTBkMmQ3YQ=='}] (graphql.py:26)
2024-02-03 15:12:26,282 - dbterd - DEBUG - Completed [status: 200] (graphql.py:34)
2024-02-03 15:12:26,286 - dbterd - INFO - Result: 0 models, 0 sources, 0 exposures, 10 tests (discovery.py:93)
2024-02-03 15:12:26,288 - dbterd - DEBUG - Getting erd data...[URL: https://metadata.cloud.getdbt.com/graphql/graphql/, VARS: {'environment_id': '284661', 'model_first': 10, 'source_first': 10, 'exposure_first': 10, 'test_first': 10, 'model_after': 'Y3Vyc29yOm1vZGVsLmphZmZsZV9zaG9wLnN0Z19wYXltZW50cw==', 'source_after': 'Y3Vyc29yOnNvdXJjZS5qYWZmbGVfc2hvcC5tYXJ0X2phZmZsZV9zaG9wLm9yZGVycw==', 'exposure_after': 'Y3Vyc29yOmV4cG9zdXJlLmphZmZsZV9zaG9wLmN1c3RvbWVyX29yZGVycw==', 'test_after': 'Y3Vyc29yOnRlc3QuamFmZmxlX3Nob3AudW5pcXVlX3N0Z19vcmRlcnNfb3JkZXJfaWQuZTNiODQxYzcxYQ=='}] (graphql.py:26)
2024-02-03 15:12:27,795 - dbterd - DEBUG - Completed [status: 200] (graphql.py:34)
2024-02-03 15:12:27,800 - dbterd - INFO - Result: 0 models, 0 sources, 0 exposures, 1 tests (discovery.py:93)
2024-02-03 15:12:27,819 - dbterd - INFO - Collected 7 table(s) and 2 relationship(s) (test_relationship.py:46)
2024-02-03 15:12:27,820 - dbterd - INFO - C:\Sources\dbterd\target (base.py:182)

[datnguye]

Live now in v1.10 🎉