From f0b86608911b477157a96d73be95f3ac368b5e29 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 4 Apr 2023 11:27:13 -0400 Subject: [PATCH 1/3] Updates to env var names --- website/docs/docs/deploy/about-state.md | 17 ++++++++++++ .../versions/02-upgrading-to-v1.5.md | 11 ++++++++ website/docs/reference/global-configs.md | 25 ++++++++++++++++++ .../docs/reference/node-selection/defer.md | 10 +++++++ .../docs/reference/node-selection/methods.md | 26 +++++++++++++++++++ 5 files changed, 89 insertions(+) diff --git a/website/docs/docs/deploy/about-state.md b/website/docs/docs/deploy/about-state.md index b8d4092dcce..440f4c6a1ab 100644 --- a/website/docs/docs/deploy/about-state.md +++ b/website/docs/docs/deploy/about-state.md @@ -26,9 +26,26 @@ Together, these two features enable ["slim CI"](best-practices#run-only-modified State and defer can be set by environment variables as well as CLI flags: + + - `--state` or `DBT_ARTIFACT_STATE_PATH`: file path - `--defer` or `DBT_DEFER_TO_STATE`: boolean + + + + +- `--state` or `DBT_STATE`: file path +- `--defer` or `DBT_DEFER`: boolean + +:::warning Syntax deprecated + +The original syntax for state (`DBT_ARTIFACT_STATE_PATH`) and defer (`DBT_DEFER_TO_STATE`) have been deprecated in dbt v1.5. Backward compatibility is supported in this version but will be removed in an as-of-yet-undetermined future release. + +::: + + + If both the flag and env var are provided, the flag takes precedence. #### Notes: diff --git a/website/docs/guides/migration/versions/02-upgrading-to-v1.5.md b/website/docs/guides/migration/versions/02-upgrading-to-v1.5.md index 3cac3fd346d..de4a3acb93d 100644 --- a/website/docs/guides/migration/versions/02-upgrading-to-v1.5.md +++ b/website/docs/guides/migration/versions/02-upgrading-to-v1.5.md @@ -46,6 +46,17 @@ The manifest schema version will be updated to `v9`. Specific changes: - Addition of `access` as a top-level node config for models - Addition of `group` and `contract` as node configs +### For users of env vars + +The following env vars have been renamed: + +- `DBT_DEFER_TO_STATE` → `DBT_DEFER` +- `DBT_FAVOR_STATE_MODE` → `DBT_FAVOR_STATE` +- `DBT_NO_PRINT` → `DBT_PRINT` +- `DBT_ARTIFACT_STATE_PATH` → `DBT_STATE` + +Backward compatibility with the old syntax is still supported but will be removed in an as-of-yet-undetermined future released. + ### For maintainers of adapter plugins For more detailed information and to ask any questions, please visit [dbt-core/discussions/6624](https://github.com/dbt-labs/dbt-core/discussions/6624). diff --git a/website/docs/reference/global-configs.md b/website/docs/reference/global-configs.md index 222c89c09d0..05541efdf95 100644 --- a/website/docs/reference/global-configs.md +++ b/website/docs/reference/global-configs.md @@ -356,6 +356,8 @@ $ dbt --quiet run ### Suppress `print()` messages in stdout + + By default, dbt includes `print()` messages in standard out (stdout). You can use the `NO_PRINT` config to prevent these messages from showing up in stdout. @@ -367,6 +369,29 @@ config: + + + + +By default, dbt includes `print()` messages in standard out (stdout). You can use the `PRINT` config to prevent these messages from showing up in stdout. + + + +```yaml +config: + print: false +``` + + + +:::warning Syntax deprecation + +The original `NO_PRINT` syntax has been deprecated, starting with dbt v1.5. Backward compatibility is supported but will be removed in an as-of-yet-undetermined future release. + +::: + + + Supply `--no-print` flag to `dbt run` to suppress `print()` messages from showing in stdout. ```text diff --git a/website/docs/reference/node-selection/defer.md b/website/docs/reference/node-selection/defer.md index 59d63b6aea2..237915c48e6 100644 --- a/website/docs/reference/node-selection/defer.md +++ b/website/docs/reference/node-selection/defer.md @@ -43,8 +43,18 @@ When using defer, you may be selecting from production datasets, development dat - if you apply env-specific limits in dev but not prod, as you may end up selecting more data than you expect - when executing tests that depend on multiple parents (e.g. `relationships`), since you're testing "across" environments + + Deferral requires both `--defer` and `--state` to be set, either by passing flags explicitly or by setting environment variables (`DBT_DEFER_TO_STATE` and `DBT_ARTIFACT_STATE_PATH`). If you use dbt Cloud, read about [how to set up CI jobs](/docs/deploy/cloud-ci-job). + + + + +Deferral requires both `--defer` and `--state` to be set, either by passing flags explicitly or by setting environment variables (`DBT_DEFER` and `DBT_STATE`). If you use dbt Cloud, read about [how to set up CI jobs](/docs/deploy/cloud-ci-job). + + + #### Favor state diff --git a/website/docs/reference/node-selection/methods.md b/website/docs/reference/node-selection/methods.md index 7913e4688aa..794749f5b94 100644 --- a/website/docs/reference/node-selection/methods.md +++ b/website/docs/reference/node-selection/methods.md @@ -143,8 +143,18 @@ that defines it. For more information about how generic tests are defined, read **N.B.** State-based selection is a powerful, complex feature. Read about [known caveats and limitations](node-selection/state-comparison-caveats) to state comparison. + + The `state` method is used to select nodes by comparing them against a previous version of the same project, which is represented by a [manifest](artifacts/manifest-json). The file path of the comparison manifest _must_ be specified via the `--state` flag or `DBT_ARTIFACT_STATE_PATH` environment variable. + + + + +The `state` method is used to select nodes by comparing them against a previous version of the same project, which is represented by a [manifest](artifacts/manifest-json). The file path of the comparison manifest _must_ be specified via the `--state` flag or `DBT_STATE` environment variable. + + + `state:new`: There is no node with the same `unique_id` in the comparison manifest `state:modified`: All new nodes, plus any changes to existing nodes. @@ -222,11 +232,27 @@ The following dbt commands produce `sources.json` artifacts whose results can be After issuing one of the above commands, you can reference the source freshness results by adding a selector to a subsequent command as follows: + + ```bash # You can also set the DBT_ARTIFACT_STATE_PATH environment variable instead of the --state flag. $ dbt source freshness # must be run again to compare current to previous state $ dbt build --select source_status:fresher+ --state path/to/prod/artifacts ``` + + + + + +```bash +# You can also set the DBT_STATE environment variable instead of the --state flag. +$ dbt source freshness # must be run again to compare current to previous state +$ dbt build --select source_status:fresher+ --state path/to/prod/artifacts +``` + + + + From c960b53d872634c0cf48aa1b226b9dbb2bacad29 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 4 Apr 2023 12:51:10 -0400 Subject: [PATCH 2/3] UPdate to include dbt list changes --- website/docs/guides/migration/versions/02-upgrading-to-v1.5.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/website/docs/guides/migration/versions/02-upgrading-to-v1.5.md b/website/docs/guides/migration/versions/02-upgrading-to-v1.5.md index de4a3acb93d..efec2a958d2 100644 --- a/website/docs/guides/migration/versions/02-upgrading-to-v1.5.md +++ b/website/docs/guides/migration/versions/02-upgrading-to-v1.5.md @@ -38,6 +38,9 @@ Changes planned for v1.5: - Renaming ["global configs"](global-configs) for consistency ([dbt-core#6903](https://github.com/dbt-labs/dbt-core/issues/6903)) - Moving `log-path` and `target-path` out of `dbt_project.yml` for consistency with other global configs ([dbt-core#6882](https://github.com/dbt-labs/dbt-core/issues/6882)) - As described in [dbt-core#7169](https://github.com/dbt-labs/dbt-core/pull/7169), command-line parameters that could be silent before will no longer be silent. See [dbt-labs/dbt-core#7158](https://github.com/dbt-labs/dbt-core/issues/7158) and [dbt-labs/dbt-core#6800](https://github.com/dbt-labs/dbt-core/issues/6800) for more examples of the behavior we are fixing. +- `dbt list` will no longer suppress `INFO` level logs. To pipe the results of dbt list to a file or another process, and exclude unrelated log messages from the output, you can use either of the following flags: + - `dbt --log-level warn ls` — equivalent to previous default behavior. + - `dbt --quiet ls` — suppresses anything less than ERROR level, except for "printed" messages. ### For consumers of dbt artifacts (metadata) From 4019d253ad64dbc5c05298dbf55c7f521ac44ee4 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 4 Apr 2023 14:51:47 -0400 Subject: [PATCH 3/3] Delete about-state.md --- website/docs/docs/deploy/about-state.md | 120 ------------------------ 1 file changed, 120 deletions(-) delete mode 100644 website/docs/docs/deploy/about-state.md diff --git a/website/docs/docs/deploy/about-state.md b/website/docs/docs/deploy/about-state.md deleted file mode 100644 index 440f4c6a1ab..00000000000 --- a/website/docs/docs/deploy/about-state.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -title: "Project state" -id: "project-state" -sidebar: "Project state" ---- - - - - - The `--state` flag was introduced in dbt v0.18.0 - - The `result` selector was introduced in dbt v1.0.0 - - - -One of the greatest underlying assumptions about dbt is that its operations should be **stateless** and ****. That is, it doesn't matter how many times a model has been run before, or if it has ever been run before. It doesn't matter if you run it once or a thousand times. Given the same raw data, you can expect the same transformed result. A given run of dbt doesn't need to "know" about _any other_ run; it just needs to know about the code in the project and the objects in your database as they exist _right now_. - -That said, dbt does store "state"—a detailed, point-in-time view of project resources, database objects, and invocation results—in the form of its [artifacts](dbt-artifacts). If you choose, dbt can use these artifacts to inform certain operations. Crucially, the operations themselves are still stateless and : given the same manifest and the same raw data, dbt will produce the same transformed result. - -dbt can leverage artifacts from a prior invocation as long as their file path is passed to the `--state` flag. This is a prerequisite for: -- [The `state:` selector](methods#the-state-method), whereby dbt can identify resources that are new or modified -by comparing code in the current project against the state manifest. -- [Deferring](defer) to another environment, whereby dbt can identify upstream, unselected resources that don't exist in your current environment and instead "defer" their references to the environment provided by the state manifest. - -Together, these two features enable ["slim CI"](best-practices#run-only-modified-models-to-test-changes-slim-ci). We expect to add more features in future releases that can leverage artifacts passed to the `--state` flag. - -### Establishing state - -State and defer can be set by environment variables as well as CLI flags: - - - -- `--state` or `DBT_ARTIFACT_STATE_PATH`: file path -- `--defer` or `DBT_DEFER_TO_STATE`: boolean - - - - - -- `--state` or `DBT_STATE`: file path -- `--defer` or `DBT_DEFER`: boolean - -:::warning Syntax deprecated - -The original syntax for state (`DBT_ARTIFACT_STATE_PATH`) and defer (`DBT_DEFER_TO_STATE`) have been deprecated in dbt v1.5. Backward compatibility is supported in this version but will be removed in an as-of-yet-undetermined future release. - -::: - - - -If both the flag and env var are provided, the flag takes precedence. - -#### Notes: -- The `--state` artifacts must be of schema versions that are compatible with the currently running dbt version. -- The path to state artifacts can be set via the `--state` flag or `DBT_ARTIFACT_STATE_PATH` environment variable. If both the flag and env var are provided, the flag takes precedence. -- These are powerful, complex features. Read about [known caveats and limitations](node-selection/state-comparison-caveats) to state comparison. - -### The "result" status - -Another element of job state is the `result` of a prior dbt invocation. After executing a `dbt run`, for example, dbt creates the `run_results.json` artifact which contains execution times and success / error status for dbt models. You can read more about `run_results.json` on the ['run results'](/docs/reference/artifacts/run-results-json) page. - -The following dbt commands produce `run_results.json` artifacts whose results can be referenced in subsequent dbt invocations: -- `dbt run` -- `dbt test` -- `dbt build` (new in dbt version v0.21.0) -- `dbt seed` - -After issuing one of the above commands, you can reference the results by adding a selector to a subsequent command as follows: - -```bash -# You can also set the DBT_ARTIFACT_STATE_PATH environment variable instead of the --state flag. -$ dbt run --select result: --defer --state path/to/prod/artifacts -``` - -The available options depend on the node type: - -| | model | seed | snapshot | test | -|----------------|-------|------|------|----------| -| `result:error` | ✅ | ✅ | ✅ | ✅ | -| `result:success` | ✅ | ✅ | ✅ | | -| `result:skipped` | ✅ | | ✅ | ✅ | -| `result:fail` | | | | ✅ | -| `result:warn` | | | | ✅ | -| `result:pass` | | | | ✅ | - -### Combining `state` and `result` selectors - -The state and result selectors can also be combined in a single invocation of dbt to capture errors from a previous run OR any new or modified models. - -```bash -$ dbt run --select result:+ state:modified+ --defer --state ./ -``` - -### The "source_status" status - - -Only supported by v1.1 or newer. - - - - - -Only supported by v1.1 or newer. - -:::caution Experimental functionality -The `source_status` selection method is experimental and subject to change. During this time, ongoing improvements may limit this feature’s availability and cause breaking changes to its functionality. -::: - - -Another element of job state is the `source_status` of a prior dbt invocation. After executing `dbt source freshness`, for example, dbt creates the `sources.json` artifact which contains execution times and `max_loaded_at` dates for dbt sources. You can read more about `sources.json` on the ['sources'](/docs/reference/artifacts/sources-json) page. - -The following dbt commands produce `sources.json` artifacts whose results can be referenced in subsequent dbt invocations: -- `dbt source freshness` - -After issuing one of the above commands, you can reference the source freshness results by adding a selector to a subsequent command as follows: - -```bash -# You can also set the DBT_ARTIFACT_STATE_PATH environment variable instead of the --state flag. -$ dbt source freshness # must be run again to compare current to previous state -$ dbt build --select source_status:fresher+ --state path/to/prod/artifacts -``` -