Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add info about + prefix and splits config page #6932

Open
wants to merge 10 commits into
base: current
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ is_featured: false

We’re announcing that [dbt Server](https://github.com/dbt-labs/dbt-server) is officially deprecated and will no longer be maintained by dbt Labs going forward. You can continue to use the repository and fork it for your needs. We’re also looking for a maintainer of the repository from our community! If you’re interested, please reach out by opening an issue in the [repository](https://github.com/dbt-labs/dbt-server/issues).


<!-- truncate -->
## Why are we deprecating dbt Server?

At dbt Labs, we are continually working to build rich experiences that help our users scale collaboration around data. To achieve this vision, we need to take moments to think about which products are contributing to this goal, and sometimes make hard decisions about the ones that are not, so that we can focus on enhancing the most impactful ones.
Expand Down
199 changes: 4 additions & 195 deletions website/docs/reference/configs-and-properties.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
---
title: Configs, properties, what are they?
sidebar_label: Configs and properties
intro_text: "Clarify the difference between properties and configs in dbt: properties describe resources, while configs control how dbt builds them in the warehouse."
description: "Clarify the difference between properties and configs in dbt: properties describe resources, while configs control how dbt builds them in the warehouse."
pagination_next: "reference/define-configs"
---

Resources in your project—models, snapshots, seeds, tests, and the rest—can have a number of declared **properties**. Resources can also define **configurations**, which are a special kind of property that bring extra abilities. What's the distinction?
Expand All @@ -21,198 +25,3 @@ Whereas you can use **configurations** to:
* Declare where a seed will be created in the database (`<database>.<schema>.<alias>`)
* Declare whether a resource should persist its descriptions as comments in the database
* Apply tags and "meta" properties

## Where can I define configs?

Depending on the resource type, configurations can be defined in the dbt project and also in an installed package by:

<VersionBlock firstVersion="1.9">

1. Using a [`config` property](/reference/resource-properties/config) in a `.yml` file for supported resource directories like `models/`, `snapshots/`, `seeds/`, `analyses`, `tests/`, and more.
2. From the [`dbt_project.yml` file](dbt_project.yml), under the corresponding resource key (`models:`, `snapshots:`, `tests:`, etc)
</VersionBlock>

<VersionBlock lastVersion="1.8">

1. Using a [`config()` Jinja macro](/reference/dbt-jinja-functions/config) within a `model`, `snapshot`, or `test` SQL file
2. Using a [`config` property](/reference/resource-properties/config) in a `.yml` file for supported resource directories like `models/`, `snapshots/`, `seeds/`, `analyses/`, or `tests/` directory.
3. From the [`dbt_project.yml` file](dbt_project.yml), under the corresponding resource key (`models:`, `snapshots:`, `tests:`, etc)
</VersionBlock>

### Config inheritance

The most specific config always takes precedence. This generally follows the order above: an in-file `config()` block --> properties defined in a `.yml` file --> config defined in the project file.

Note - Generic data tests work a little differently when it comes to specificity. See [test configs](/reference/data-test-configs).

Within the project file, configurations are also applied hierarchically. The most specific config always takes precedence. In the project file, for example, configurations applied to a `marketing` subdirectory will take precedence over configurations applied to the entire `jaffle_shop` project. To apply a configuration to a model or directory of models, define the [resource path](/reference/resource-configs/resource-path) as nested dictionary keys.

Configurations in your root dbt project have _higher_ precedence than configurations in installed packages. This enables you to override the configurations of installed packages, providing more control over your dbt runs.


### Combining configs

Most configurations are "clobbered" when applied hierarchically. Whenever a more specific value is available, it will completely replace the less specific value. Note that a few configs have different merge behavior:
- [`tags`](/tags) are additive. If a model has some tags configured in `dbt_project.yml`, and more tags applied in its `.sql` file, the final set of tags will include all of them.
- [`meta`](/reference/resource-configs/meta) dictionaries are merged (a more specific key-value pair replaces a less specific value with the same key)
- [`pre-hook` and `post-hook`](/reference/resource-configs/pre-hook-post-hook) are also additive.

## Where can I define properties?

In dbt, you can use `properties.yml` files to define properties for resources. You can declare properties in `.yml` files, in the same directory as your resources. You can name these files `whatever_you_want.yml` and nest them arbitrarily in sub-folders within each directory.

We highly recommend that you define properties in dedicated paths alongside the resources they're describing.

:::info

#### schema.yml files

Previous versions of the docs referred to these as `schema.yml` files — we've moved away from that terminology since the word `schema` is used to mean other things when talking about databases, and people often thought that you _had_ to name these files `schema.yml`.

Instead, we now refer to these files as `properties.yml` files. (Of course, you're still free to name your files `schema.yml`)

:::

### Which properties are _not_ also configs?

In dbt, you can define node configs in `properties.yml` files, in addition to `config()` blocks and `dbt_project.yml`. However, some special properties can only be defined in the `.yml` file and you cannot configure them using `config()` blocks or the `dbt_project.yml` file:

Certain properties are special, because:

- They have a unique Jinja rendering context
- They create new project resources
- They don't make sense as hierarchical configuration
- They're older properties that haven't yet been redefined as configs

These properties are:

- [`description`](/reference/resource-properties/description)
- [`tests`](/reference/resource-properties/data-tests)
- [`docs`](/reference/resource-configs/docs)
- [`columns`](/reference/resource-properties/columns)
- [`quote`](/reference/resource-properties/quote)
- [`source` properties](/reference/source-properties) (e.g. `loaded_at_field`, `freshness`)
- [`exposure` properties](/reference/exposure-properties) (e.g. `type`, `maturity`)
- [`macro` properties](/reference/macro-properties) (e.g. `arguments`)

## Example
Here's an example that defines both `sources` and `models` for a project:

<File name='models/jaffle_shop.yml'>

```yml
version: 2

sources:
- name: raw_jaffle_shop
description: A replica of the postgres database used to power the jaffle_shop app.
tables:
- name: customers
columns:
- name: id
description: Primary key of the table
tests:
- unique
- not_null

- name: orders
columns:
- name: id
description: Primary key of the table
tests:
- unique
- not_null

- name: user_id
description: Foreign key to customers

- name: status
tests:
- accepted_values:
values: ['placed', 'shipped', 'completed', 'return_pending', 'returned']


models:
- name: stg_jaffle_shop__customers
config:
tags: ['pii']
columns:
- name: customer_id
tests:
- unique
- not_null

- name: stg_jaffle_shop__orders
config:
materialized: view
columns:
- name: order_id
tests:
- unique
- not_null
- name: status
tests:
- accepted_values:
values: ['placed', 'shipped', 'completed', 'return_pending', 'returned']
config:
severity: warn


```

</File>


## Related documentation
You can find an exhaustive list of each supported property and config, broken down by resource type:
* Model [properties](/reference/model-properties) and [configs](/reference/model-configs)
* Source [properties](/reference/source-properties) and [configs](source-configs)
* Seed [properties](/reference/seed-properties) and [configs](/reference/seed-configs)
* Snapshot [properties](snapshot-properties)
* Analysis [properties](analysis-properties)
* Macro [properties](/reference/macro-properties)
* Exposure [properties](/reference/exposure-properties)

## FAQs
<FAQ path="Project/schema-yml-name" />
<FAQ path="Project/resource-yml-name" />
<FAQ path="Project/multiple-resource-yml-files" />
<FAQ path="Project/properties-not-in-config" />
<FAQ path="Project/why-version-2" />
<FAQ path="Project/yaml-file-extension" />

## Troubleshooting common errors

### Invalid test config given in [model name]

This error occurs when your `.yml` file does not conform to the structure expected by dbt. A full error message might look like:
```
* Invalid test config given in models/schema.yml near {'namee': 'event', ...}
Invalid arguments passed to "UnparsedNodeUpdate" instance: 'name' is a required property, Additional properties are not allowed ('namee' was unexpected)
```

While verbose, an error like this should help you track down the issue. Here, the `name` field was provided as `namee` by accident. To fix this error, ensure that your `.yml` conforms to the expected structure described in this guide.

### Invalid syntax in your schema.yml file

If your `.yml` file is not valid yaml, then dbt will show you an error like this:

```
Runtime Error
Syntax error near line 6
------------------------------
5 | - name: events
6 | description; "A table containing clickstream events from the marketing website"
7 |

Raw Error:
------------------------------
while scanning a simple key
in "<unicode string>", line 6, column 5:
description; "A table containing clickstream events from the marketing website"
^

```

This error occurred because a semicolon (`;`) was accidentally used instead of a colon (`:`) after the `description` field. To resolve issues like this, find the `.yml` file referenced in the error message and fix any syntax errors present in the file. There are online YAML validators that can be helpful here, but please be mindful of submitting sensitive information to third-party applications!

24 changes: 23 additions & 1 deletion website/docs/reference/dbt_project.yml.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
---
description: "Reference guide for configuring the dbt_project.yml file."
intro_text: "The dbt_project.yml file is a required file for all dbt projects. It contains important information that tells dbt how to operate your project."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's our best practice for intro text? Should it be describing the feature or describing what's on the page (for example "This is the reference page for configuring the dbt_project.yml, a required file for all dbt projects that contains important information that tells dbt how to operate your project."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little bit of both I'd say, intro text is probably a little longer than description

---

Every [dbt project](/docs/build/projects) needs a `dbt_project.yml` file — this is how dbt knows a directory is a dbt project. It also contains important information that tells dbt how to operate your project.
Every [dbt project](/docs/build/projects) needs a `dbt_project.yml` file — this is how dbt knows a directory is a dbt project. It also contains important information that tells dbt how to operate your project. It works as follows:

- dbt uses [YAML](https://yaml.org/) in a few different places. If you're new to YAML, it would be worth learning how arrays, dictionaries, and strings are represented.

Expand Down Expand Up @@ -96,6 +97,27 @@ vars:

</File>

## The `+` prefix
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is already a separate ref page. Every use of + in the docs links to this page. Will this section cause confusion or hurt the search?

Copy link
Contributor Author

@mirnawong1 mirnawong1 Feb 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, might be better to link to that page instead! Or i might turn it into a snippet so its available in multiple places


dbt demarcates between a folder name and a configuration by using a `+` prefix before the configuration name. The `+` prefix is used for configs _only_ and applies to `dbt_project.yml` under the corresponding resource key. It doesn't apply to:
- `config()` Jinja macro within a resource file
- config property in a `.yml` file.

For example, `marketing`, `paid_ads`, and `google` are folder names, whereas `+materialized` is a configuration being applied to those folder and all folders nested below them.

<File name="dbt_project.yml">

```yaml
models:
jaffle_shop:
marketing:
+materialized: view
paid_ads:
google:
+materialized: table
```
</File>

## Naming convention

It's important to follow the correct YAML naming conventions for the configs in your `dbt_project.yml` file to ensure dbt can process them properly. This is especially true for resource types with more than one word.
Expand Down
45 changes: 45 additions & 0 deletions website/docs/reference/define-configs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
title: Define configs
sidebar_label: Define configs
intro_text: "Learn how to define configurations for your resources in a dbt project"
description: "Learn how to define configurations for your resources in a dbt project"
pagination_previous: "reference/configs-and-properties"
pagination_next: "reference/define-properties"
---

Depending on the resource type, you can define configurations in a dbt project and also in an installed package by:

<VersionBlock firstVersion="1.9">

1. Using a [`config` property](/reference/resource-properties/config) in a `.yml` file for supported resource directories like `models/`, `snapshots/`, `seeds/`, `analyses`, `tests/`, and more.
2. From the [`dbt_project.yml` file](dbt_project.yml), under the corresponding resource key (`models:`, `snapshots:`, `tests:`, and so on)
</VersionBlock>

<VersionBlock lastVersion="1.8">

1. Using a [`config()` Jinja macro](/reference/dbt-jinja-functions/config) within a `model`, `snapshot`, or `test` SQL file
2. Using a [`config` property](/reference/resource-properties/config) in a `.yml` file for supported resource directories like `models/`, `snapshots/`, `seeds/`, `analyses/`, or `tests/` directory.
3. From the [`dbt_project.yml` file](dbt_project.yml), under the corresponding resource key (`models:`, `snapshots:`, `tests:`, and so on)
</VersionBlock>

### Config inheritance

The most specific config always takes precedence. This generally follows the order above: an in-file `config()` block --> properties defined in a `.yml` file --> config defined in the project file.

Note - Generic data tests work a little differently when it comes to specificity. See [test configs](/reference/data-test-configs).

Within the project file, configurations are also applied hierarchically. The most specific config always takes precedence. In the project file, for example, configurations applied to a `marketing` subdirectory will take precedence over configurations applied to the entire `jaffle_shop` project. To apply a configuration to a model or directory of models, define the [resource path](/reference/resource-configs/resource-path) as nested dictionary keys.

Configurations in your root dbt project have _higher_ precedence than configurations in installed packages. This enables you to override the configurations of installed packages, providing more control over your dbt runs.


### Combining configs

Most configurations are "clobbered" when applied hierarchically. Whenever a more specific value is available, it will completely replace the less specific value. Note that a few configs have different merge behavior:
- [`tags`](/tags) are additive. If a model has some tags configured in `dbt_project.yml`, and more tags applied in its `.sql` file, the final set of tags will include all of them.
- [`meta`](/reference/resource-configs/meta) dictionaries are merged (a more specific key-value pair replaces a less specific value with the same key)
- [`pre-hook` and `post-hook`](/reference/resource-configs/pre-hook-post-hook) are also additive.

import Example from '/snippets/_configs-properties.md' ;

<Example />
47 changes: 47 additions & 0 deletions website/docs/reference/define-properties.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
---
title: Define properties
sidebar_label: Define properties
intro_text: "Learn how to define properties for your resources in a properties.yml file"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question as above for best practices for intro text - Should our intro text be more descriptive than the descriptions?

description: "Learn how to define properties for your resources in a properties.yml file"
pagination_previous: "reference/define-configs"
---

In dbt, you can use `properties.yml` files to define properties for resources. You can declare properties in `.yml` files, in the same directory as your resources. You can name these files `whatever_you_want.yml` and nest them arbitrarily in sub-folders within each directory.

We highly recommend that you define properties in dedicated paths alongside the resources they're describing.

:::info

#### schema.yml files

Previous versions of the docs referred to these as `schema.yml` files — we've moved away from that terminology since the word `schema` is used to mean other things when talking about databases, and people often thought that you _had_ to name these files `schema.yml`.

Instead, we now refer to these files as `properties.yml` files. (Of course, you're still free to name your files `schema.yml`)

:::

### Which properties are _not_ also configs?

In dbt, you can define node configs in `properties.yml` files, in addition to `config()` blocks and `dbt_project.yml`. However, some special properties can only be defined in the `.yml` file and you cannot configure them using `config()` blocks or the `dbt_project.yml` file:

Certain properties are special, because:

- They have a unique Jinja rendering context
- They create new project resources
- They don't make sense as hierarchical configuration
- They're older properties that haven't yet been redefined as configs

These properties are:

- [`description`](/reference/resource-properties/description)
- [`tests`](/reference/resource-properties/data-tests)
- [`docs`](/reference/resource-configs/docs)
- [`columns`](/reference/resource-properties/columns)
- [`quote`](/reference/resource-properties/quote)
- [`source` properties](/reference/source-properties) (for example, `loaded_at_field`, `freshness`)
- [`exposure` properties](/reference/exposure-properties) (for example, `type`, `maturity`)
- [`macro` properties](/reference/macro-properties) (for example, `arguments`)

import Example from '/snippets/_configs-properties.md' ;

<Example />
2 changes: 1 addition & 1 deletion website/docs/reference/project-configs/version.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ A version tag in a `.yml` property file provides the control tag, which informs

Starting from version 1.5, dbt will no longer require this configuration in your resource `.yml` files. If you want to know more about why this tag was previously required, you can refer to the [FAQs](#faqs). For users on dbt version 1.4 or lower, this tag is required,

For more on property files, see their general [documentation](/reference/configs-and-properties#where-can-i-define-properties) on the same page.
For more on property files, see their general [documentation](/reference/define-properties) on the same page.

<Tabs
groupId="resource-version-configs"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,4 +73,4 @@ Additional resources:
- [Data test configurations](/reference/data-test-configs#related-documentation)
- [Data test-specific configurations](/reference/data-test-configs#test-data-specific-configurations)
- [Configuring directories of models in dbt_project.yml](/reference/model-configs#configuring-directories-of-models-in-dbt_projectyml)
- [Config inheritance](/reference/configs-and-properties#config-inheritance)
- [Config inheritance](/reference/define-configs#config-inheritance)
14 changes: 13 additions & 1 deletion website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -938,8 +938,20 @@ const sidebarSettings = {
type: "category",
label: "Resource configs and properties",
items: [
"reference/configs-and-properties",
"reference/resource-configs/resource-path",
{
type: "category",
label: "Configs and properties",
link: {
type: "doc",
id: "reference/configs-and-properties",
},
items: [
"reference/configs-and-properties",
"reference/define-configs",
"reference/define-properties",
],
},
{
type: "category",
label: "General properties",
Expand Down
Loading
Loading