Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source freshness #6744

Open
wants to merge 9 commits into
base: current
Choose a base branch
from
8 changes: 8 additions & 0 deletions website/docs/docs/build/sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,14 @@ The results of this query are used to determine whether the source is fresh or n

<Lightbox src="/img/docs/building-a-dbt-project/snapshot-freshness.png" title="Uh oh! Not everything is as fresh as we'd like!"/>

### Build models based on source freshness

dbt has a feature to [check the freshness](https://docs.getdbt.com/docs/build/sources#declaring-source-freshness) (how up to date) of a specific source. This functionality is specifically designed for sources and does not apply to models.

Neither `dbt source freshness` nor `dbt test` will perform any freshness checking of models. This is intentional because, in a typical dbt project, raw data usually comes in as sources. Your models then transform this data by selecting from those sources. If your source data is fresh, there's generally no need to perform freshness checks on your models as well.

However, dbt does however does provide the flexibility for users to create their own [generic tests](https://docs.getdbt.com/docs/build/tests#generic-tests) that they can then apply to models.

### Filter

Some databases can have tables where a filter over certain columns are required, in order prevent a full scan of the table, which could be costly. In order to do a freshness check on such tables a `filter` argument can be added to the configuration, e.g. `filter: _etl_loaded_at >= date_sub(current_date(), interval 1 day)`. For the example above, the resulting query would look like
Expand Down
22 changes: 22 additions & 0 deletions website/docs/reference/commands/source.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,16 @@ The `dbt source` command provides subcommands that are useful when working with

If your dbt project is [configured with sources](/docs/build/sources), then the `dbt source freshness` command will query all of your defined source tables, determining the "freshness" of these tables. If the tables are stale (based on the `freshness` config specified for your sources) then dbt will report a warning or error accordingly. If a source <Term id="table" /> is in a stale state, then dbt will exit with a nonzero exit code.

You can also use [source freshness](/docs/deploy/source-freshness) commands help make sure the data you get is new and not old or outdated.

#### Configure source freshness

You can configure source freshness in the execution settings within your job in dbt Cloud. For more information, refer to [enabling source freshness snapshots](/docs/deploy/source-freshness#enabling-source-freshness-snapshots).

import SourcefreshnessTable from '/snippets/_selecting_source_freshness.md';

<SourcefreshnessTable />

### Specifying sources to snapshot

By default, `dbt source freshness` will calculate freshness information for all of the sources in your project. To snapshot freshness for a subset of these sources, use the `--select` flag.
Expand Down Expand Up @@ -76,3 +86,15 @@ Snapshots of source freshness can be used to understand:
This command can be run manually to determine the state of your source data freshness at any time. It is also recommended that you run this command on a schedule, storing the results of the freshness snapshot at regular intervals. These longitudinal snapshots will make it possible to be alerted when source data freshness SLAs are violated, as well as understand the trend of freshness over time.

dbt Cloud makes it easy to snapshot source freshness on a schedule, and provides a dashboard out of the box indicating the state of freshness for all of the sources defined in your project. For more information on snapshotting freshness in dbt Cloud, check out the [docs](/docs/build/sources#source-data-freshness).

### Source freshness commands

Source freshness commands ensure you're receiving the most up-to-date, relevant, and accurate information.

Some of the typical commands you can use are:

| **Command** | **Description** |
| ----------------------------------------------------------------------------| ---------------------------------|
|[`source_status`](/reference/node-selection/methods#the-source_status-method)|dbt generates the [`sources.json`](/reference/artifacts/sources-json) artifact, which includes execution times and [`max_loaded_at`](/reference/artifacts/sources-json#top-level-keys) timestamps for dbt sources.|
|[`state:modified`](/reference/node-selection/methods#state) |Used to select nodes by comparing them to a previous version of the same project, represented by a [manifest](/reference/artifacts/manifest-json).|
|[`dbt source freshness`](/reference/commands/source#dbt-source-freshness) |If your dbt project includes configured sources, the dbt source freshness command will query all your defined source tables to determine their "freshness."|
1 change: 1 addition & 0 deletions website/snippets/_selecting_source_freshness.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<Lightbox src="/img/docs/dbt-cloud/select-source-freshness.png" title="Selecting source freshness"/>
Loading