-
Notifications
You must be signed in to change notification settings - Fork 995
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add info about + prefix and splits config page #6932
base: current
Are you sure you want to change the base?
Changes from all commits
3650ce2
f06da23
5c28238
ad2af81
918980c
e61810f
1b070fe
4ce1dd3
2c74d1e
2a1fb16
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,9 @@ | ||
--- | ||
description: "Reference guide for configuring the dbt_project.yml file." | ||
intro_text: "The dbt_project.yml file is a required file for all dbt projects. It contains important information that tells dbt how to operate your project." | ||
--- | ||
|
||
Every [dbt project](/docs/build/projects) needs a `dbt_project.yml` file — this is how dbt knows a directory is a dbt project. It also contains important information that tells dbt how to operate your project. | ||
Every [dbt project](/docs/build/projects) needs a `dbt_project.yml` file — this is how dbt knows a directory is a dbt project. It also contains important information that tells dbt how to operate your project. It works as follows: | ||
|
||
- dbt uses [YAML](https://yaml.org/) in a few different places. If you're new to YAML, it would be worth learning how arrays, dictionaries, and strings are represented. | ||
|
||
|
@@ -96,6 +97,27 @@ vars: | |
|
||
</File> | ||
|
||
## The `+` prefix | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is already a separate ref page. Every use of There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good catch, might be better to link to that page instead! Or i might turn it into a snippet so its available in multiple places |
||
|
||
dbt demarcates between a folder name and a configuration by using a `+` prefix before the configuration name. The `+` prefix is used for configs _only_ and applies to `dbt_project.yml` under the corresponding resource key. It doesn't apply to: | ||
- `config()` Jinja macro within a resource file | ||
- config property in a `.yml` file. | ||
|
||
For example, `marketing`, `paid_ads`, and `google` are folder names, whereas `+materialized` is a configuration being applied to those folder and all folders nested below them. | ||
|
||
<File name="dbt_project.yml"> | ||
|
||
```yaml | ||
models: | ||
jaffle_shop: | ||
marketing: | ||
+materialized: view | ||
paid_ads: | ||
google: | ||
+materialized: table | ||
``` | ||
</File> | ||
|
||
## Naming convention | ||
|
||
It's important to follow the correct YAML naming conventions for the configs in your `dbt_project.yml` file to ensure dbt can process them properly. This is especially true for resource types with more than one word. | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
--- | ||
title: Define configs | ||
sidebar_label: Define configs | ||
intro_text: "Learn how to define configurations for your resources in a dbt project" | ||
description: "Learn how to define configurations for your resources in a dbt project" | ||
pagination_previous: "reference/configs-and-properties" | ||
pagination_next: "reference/define-properties" | ||
--- | ||
|
||
Depending on the resource type, you can define configurations in a dbt project and also in an installed package by: | ||
|
||
<VersionBlock firstVersion="1.9"> | ||
|
||
1. Using a [`config` property](/reference/resource-properties/config) in a `.yml` file for supported resource directories like `models/`, `snapshots/`, `seeds/`, `analyses`, `tests/`, and more. | ||
2. From the [`dbt_project.yml` file](dbt_project.yml), under the corresponding resource key (`models:`, `snapshots:`, `tests:`, and so on) | ||
</VersionBlock> | ||
|
||
<VersionBlock lastVersion="1.8"> | ||
|
||
1. Using a [`config()` Jinja macro](/reference/dbt-jinja-functions/config) within a `model`, `snapshot`, or `test` SQL file | ||
2. Using a [`config` property](/reference/resource-properties/config) in a `.yml` file for supported resource directories like `models/`, `snapshots/`, `seeds/`, `analyses/`, or `tests/` directory. | ||
3. From the [`dbt_project.yml` file](dbt_project.yml), under the corresponding resource key (`models:`, `snapshots:`, `tests:`, and so on) | ||
</VersionBlock> | ||
|
||
### Config inheritance | ||
|
||
The most specific config always takes precedence. This generally follows the order above: an in-file `config()` block --> properties defined in a `.yml` file --> config defined in the project file. | ||
|
||
Note - Generic data tests work a little differently when it comes to specificity. See [test configs](/reference/data-test-configs). | ||
|
||
Within the project file, configurations are also applied hierarchically. The most specific config always takes precedence. In the project file, for example, configurations applied to a `marketing` subdirectory will take precedence over configurations applied to the entire `jaffle_shop` project. To apply a configuration to a model or directory of models, define the [resource path](/reference/resource-configs/resource-path) as nested dictionary keys. | ||
|
||
Configurations in your root dbt project have _higher_ precedence than configurations in installed packages. This enables you to override the configurations of installed packages, providing more control over your dbt runs. | ||
|
||
|
||
### Combining configs | ||
|
||
Most configurations are "clobbered" when applied hierarchically. Whenever a more specific value is available, it will completely replace the less specific value. Note that a few configs have different merge behavior: | ||
- [`tags`](/tags) are additive. If a model has some tags configured in `dbt_project.yml`, and more tags applied in its `.sql` file, the final set of tags will include all of them. | ||
- [`meta`](/reference/resource-configs/meta) dictionaries are merged (a more specific key-value pair replaces a less specific value with the same key) | ||
- [`pre-hook` and `post-hook`](/reference/resource-configs/pre-hook-post-hook) are also additive. | ||
|
||
import Example from '/snippets/_configs-properties.md' ; | ||
|
||
<Example /> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
--- | ||
title: Define properties | ||
sidebar_label: Define properties | ||
intro_text: "Learn how to define properties for your resources in a properties.yml file" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same question as above for best practices for intro text - Should our intro text be more descriptive than the descriptions? |
||
description: "Learn how to define properties for your resources in a properties.yml file" | ||
pagination_previous: "reference/define-configs" | ||
--- | ||
|
||
In dbt, you can use `properties.yml` files to define properties for resources. You can declare properties in `.yml` files, in the same directory as your resources. You can name these files `whatever_you_want.yml` and nest them arbitrarily in sub-folders within each directory. | ||
|
||
We highly recommend that you define properties in dedicated paths alongside the resources they're describing. | ||
|
||
:::info | ||
|
||
#### schema.yml files | ||
mirnawong1 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Previous versions of the docs referred to these as `schema.yml` files — we've moved away from that terminology since the word `schema` is used to mean other things when talking about databases, and people often thought that you _had_ to name these files `schema.yml`. | ||
|
||
Instead, we now refer to these files as `properties.yml` files. (Of course, you're still free to name your files `schema.yml`) | ||
|
||
::: | ||
|
||
### Which properties are _not_ also configs? | ||
|
||
In dbt, you can define node configs in `properties.yml` files, in addition to `config()` blocks and `dbt_project.yml`. However, some special properties can only be defined in the `.yml` file and you cannot configure them using `config()` blocks or the `dbt_project.yml` file: | ||
|
||
Certain properties are special, because: | ||
|
||
- They have a unique Jinja rendering context | ||
- They create new project resources | ||
- They don't make sense as hierarchical configuration | ||
- They're older properties that haven't yet been redefined as configs | ||
|
||
These properties are: | ||
|
||
- [`description`](/reference/resource-properties/description) | ||
- [`tests`](/reference/resource-properties/data-tests) | ||
- [`docs`](/reference/resource-configs/docs) | ||
- [`columns`](/reference/resource-properties/columns) | ||
- [`quote`](/reference/resource-properties/quote) | ||
- [`source` properties](/reference/source-properties) (for example, `loaded_at_field`, `freshness`) | ||
- [`exposure` properties](/reference/exposure-properties) (for example, `type`, `maturity`) | ||
- [`macro` properties](/reference/macro-properties) (for example, `arguments`) | ||
|
||
import Example from '/snippets/_configs-properties.md' ; | ||
|
||
<Example /> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's our best practice for intro text? Should it be describing the feature or describing what's on the page (for example "This is the reference page for configuring the dbt_project.yml, a required file for all dbt projects that contains important information that tells dbt how to operate your project."
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a little bit of both I'd say, intro text is probably a little longer than description