Skip to content

Commit

Permalink
docs(ingest/dbt): update run result paths examples (#11138)
Browse files Browse the repository at this point in the history
  • Loading branch information
acrylJonny committed Sep 27, 2024
1 parent 99bfcef commit a8e6a06
Show file tree
Hide file tree
Showing 3 changed files with 28 additions and 13 deletions.
8 changes: 5 additions & 3 deletions metadata-ingestion/docs/sources/dbt/dbt.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,7 @@ To integrate with dbt tests, the `dbt` source needs access to the `run_results.j
1. Run `dbt build`
2. Copy the `target/run_results.json` file to a separate location. This is important, because otherwise subsequent `dbt` commands will overwrite the run results.
3. Run `dbt docs generate` to generate the `manifest.json` and `catalog.json` files
4. The dbt source makes use of the manifest, catalog, and run results file, and hence will need to be moved to a location accessible to the `dbt` source (e.g. s3 or local file system). In the ingestion recipe, the `test_results_path` config must be set to the location of the `run_results.json` file from the `dbt build` or `dbt test` run.
4. The dbt source makes use of the manifest, catalog, and run results file, and hence will need to be moved to a location accessible to the `dbt` source (e.g. s3 or local file system). In the ingestion recipe, the `run_results_paths` config must be set to the location of the `run_results.json` file from the `dbt build` or `dbt test` run.

The connector will produce the following things:

Expand Down Expand Up @@ -219,7 +219,8 @@ source:
config:
manifest_path: _path_to_manifest_json
catalog_path: _path_to_catalog_json
test_results_path: _path_to_run_results_json
run_results_paths:
- _path_to_run_results_json
target_platform: postgres
entities_enabled:
test_results: Only
Expand All @@ -233,7 +234,8 @@ source:
config:
manifest_path: _path_to_manifest_json
catalog_path: _path_to_catalog_json
run_results_path: _path_to_run_results_json
run_results_paths:
- _path_to_run_results_json
target_platform: postgres
entities_enabled:
test_results: No
Expand Down
3 changes: 2 additions & 1 deletion metadata-ingestion/docs/sources/dbt/dbt_recipe.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ source:
manifest_path: "${DBT_PROJECT_ROOT}/target/manifest_file.json"
catalog_path: "${DBT_PROJECT_ROOT}/target/catalog_file.json"
sources_path: "${DBT_PROJECT_ROOT}/target/sources_file.json" # optional for freshness
test_results_path: "${DBT_PROJECT_ROOT}/target/run_results.json" # optional for recording dbt test results after running dbt test
run_results_paths:
- "${DBT_PROJECT_ROOT}/target/run_results.json" # optional for recording dbt test results after running dbt test

# Options
target_platform: "my_target_platform_id" # e.g. bigquery/postgres/etc.
Expand Down
30 changes: 21 additions & 9 deletions metadata-ingestion/tests/integration/dbt/test_dbt.py
Original file line number Diff line number Diff line change
Expand Up @@ -339,9 +339,13 @@ def test_dbt_tests(test_resources_dir, pytestconfig, tmp_path, mock_time, **kwar
(test_resources_dir / "jaffle_shop_catalog.json").resolve()
),
target_platform="postgres",
test_results_path=str(
(test_resources_dir / "jaffle_shop_test_results.json").resolve()
),
run_results_paths=[
str(
(
test_resources_dir / "jaffle_shop_test_results.json"
).resolve()
)
],
),
),
sink=DynamicTypedConfig(type="file", config={"filename": str(output_file)}),
Expand Down Expand Up @@ -442,9 +446,13 @@ def test_dbt_tests_only_assertions(
(test_resources_dir / "jaffle_shop_catalog.json").resolve()
),
target_platform="postgres",
test_results_path=str(
(test_resources_dir / "jaffle_shop_test_results.json").resolve()
),
run_results_paths=[
str(
(
test_resources_dir / "jaffle_shop_test_results.json"
).resolve()
)
],
entities_enabled=DBTEntitiesEnabled(
test_results=EmitDirective.ONLY
),
Expand Down Expand Up @@ -518,9 +526,13 @@ def test_dbt_only_test_definitions_and_results(
(test_resources_dir / "jaffle_shop_catalog.json").resolve()
),
target_platform="postgres",
test_results_path=str(
(test_resources_dir / "jaffle_shop_test_results.json").resolve()
),
run_results_paths=[
str(
(
test_resources_dir / "jaffle_shop_test_results.json"
).resolve()
)
],
entities_enabled=DBTEntitiesEnabled(
sources=EmitDirective.NO,
seeds=EmitDirective.NO,
Expand Down

0 comments on commit a8e6a06

Please sign in to comment.