add dbt_valid_to_current #6308
add dbt_valid_to_current #6308
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
github-actions
bot
added
content
|
@mirnawong1 grouping your outstanding questions into a handful of categories: Hardcoded SQL expressionsQuestion:
👍 Hardcoded expressions that work within the data platform is the only option that is available currently. The precise syntax will vary per warehouse, so Jinja within
|
dbeatty10
added
the
blocked_by_dev
|
@mirnawong1 I added the |
website/docs/reference/resource-configs/dbt_valid_to_current.md
Outdated
Show resolved
Hide resolved
thank you @dbeatty10 ! it looks like this is resolved now right? |
dbeatty10
removed
the
blocked_by_dev
website/docs/reference/resource-configs/dbt_valid_to_current.md
Outdated
Show resolved
Hide resolved
website/docs/reference/resource-configs/dbt_valid_to_current.md
Outdated
Show resolved
Hide resolved
website/docs/reference/resource-configs/dbt_valid_to_current.md
Outdated
Show resolved
Hide resolved
|
|
||
| By default, `dbt_valid_to` is `NULL` for current records. However, if you set the [`dbt_valid_to_current` configuration](/reference/resource-configs/dbt_valid_to_current) (available in Versionless and 1.9 and higher), `dbt_valid_to` will be set to your specified value (such as `9999-12-31`) for current records. | ||
|
|
||
| This simplifies your SQL queries by avoiding `NULL` checks and allowing for straightforward date range filtering. |
There was a problem hiding this comment.
What does "simplifies your SQL queries by avoiding NULL checks" mean?
There was a problem hiding this comment.
i meant users not needing to include checks or add'l logic to check for null records.
i can remove if it's confusing or clarify further!
| Note, these column names can be customized to your team or organizational conventions using the [snapshot_meta_column_names](#snapshot-meta-fields) config. | ||
| #### Note | ||
| - These column names can be customized to your team or organizational conventions using the [snapshot_meta_column_names](#snapshot-meta-fields) config. | ||
| - If you have set the `dbt_valid_to_current` configuration option, then instead of `NULL`, the `dbt_valid_to` field in future records will be set to your specified value (such as `9999-12-31`). |
There was a problem hiding this comment.
I think we're missing some clarity that this is set to NULL (or whatever you have set dbt_valid_to_current to for current records)
| default_value: {NULL} | ||
| id: "dbt_valid_to_current" | ||
| --- | ||
|
|
There was a problem hiding this comment.
I don't know where we want this, but we should probably have a migration callout/warning like what we have for the other new configs cc: @dbeatty10
| Starting in 1.9 or with [dbt Cloud Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless), these column names can be customized to your team or organizational conventions via the [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) config. | ||
| Starting in 1.9 or with [dbt Cloud Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless): | ||
| - These column names can be customized to your team or organizational conventions using the [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) config. | ||
| - Use the [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current) config to set a custom future date for `dbt_valid_to` in new snapshot columns. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. |
There was a problem hiding this comment.
again, i would rephrase this as suggested above
|
|
||
| ## Description | ||
|
|
||
| Use the `dbt_valid_to_current` config to set a custom future date for `dbt_valid_to` in new snapshot columns. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. |
There was a problem hiding this comment.
again, rephrase as suggested above
thinking about this more, i do like that we're calling out that the main use case for this is "a future date"
|
|
||
| Use the `dbt_valid_to_current` config to set a custom future date for `dbt_valid_to` in new snapshot columns. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` in the snapshot table. | ||
|
|
||
| This approach makes it easier to assign a custom date date, work in a join, or perform range-based filtering that require an end date. |
| The value assigned to `dbt_valid_to_current` should be a string representing a valid date or timestamp, depending on your database's requirements. Use expressions that work within the data platform. | ||
|
|
||
| ### Managing records | ||
| - **For existing records** — To avoid any unintentional data modification, dbt will _not_ automatically adjust the current value in the existing `dbt_valid_to` column. Existing current records will still have `dbt_valid_to` set to `NULL`. |
There was a problem hiding this comment.
Ah here's the migration callout - I think we just want to make sure this warning exists elsewhere as well!
|
|
||
| The value assigned to `dbt_valid_to_current` should be a string representing a valid date or timestamp, depending on your database's requirements. Use expressions that work within the data platform. | ||
|
|
||
| ### Managing records |
There was a problem hiding this comment.
Is there a more specific descriptor here for the heading?
|
|
||
| - **For new records** — Any new records inserted after applying the `dbt_valid_to_current` configuration will have `dbt_valid_to` set to the specified value (for example, '9999-12-31'), instead of `NULL`. | ||
|
|
||
| This means your snapshot table will have current records with `dbt_valid_to` values of both `NULL` (from existing data) and the new specified value (from new data). If you'd rather have consistent `dbt_valid_to` values for current records, you can either manually update existing records in your snapshot table where `dbt_valid_to` is `NULL` to match your `dbt_valid_to_current` value or rebuild your snapshot table. |
There was a problem hiding this comment.
"rebuild your snapshot table" would be very risky (lose all of your historical data!)
website/docs/reference/resource-configs/dbt_valid_to_current.md
Outdated
Show resolved
Hide resolved
website/docs/docs/build/snapshots.md
Outdated
| @@ -87,13 +81,14 @@ The following table outlines the configurations available for snapshots: | |||
| | [check_cols](/reference/resource-configs/check_cols) | If using the `check` strategy, then the columns to check | Only if using the `check` strategy | ["status"] | | |||
| | [updated_at](/reference/resource-configs/updated_at) | If using the `timestamp` strategy, the timestamp column to compare | Only if using the `timestamp` strategy | updated_at | | |||
| | [invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) | Find hard deleted records in source and set `dbt_valid_to` to current time if the record no longer exists | No | True | | |||
| | [dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current) | Set a custom indicator for the value of `dbt_valid_to` in current snapshot records. By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` in the snapshot table.| No | string | | |||
There was a problem hiding this comment.
@graciegoheen , it looks like gerda says the config applies to new snapshot columns, should i add that here?
| | [dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current) | Set a custom indicator for the value of `dbt_valid_to` in current snapshot records. By default, this value is `NULL`. When configured, dbt will use the specified value instead of `NULL` for `dbt_valid_to` in the snapshot table.| No | string | | |
| | [dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current) | Set a custom indicator for the value of `dbt_valid_to` in current snapshot records. By default, this value is `NULL`. When configured, dbt will use the specified value for `dbt_valid_to` in new snapshot columns, but it will not automatically adjust the value in existing columns. Existing columns that use `NULL` can be updated manually if needed. | No | string | |
adding new config for
dbt_valid_to_current, which allows users to set a future date fordbt_valid_tousing the dbt_valid_to_current config in yaml/config/project.Resolves #6275
Outstanding questions:
✅ - Can date be hardcoded using this syntax? "to_date('2024-05-10')" or can they use "to_date('2024, 05, 10')" ?
✅ - is the syntax just a string (e.g.
dbt_valid_to_current: '9999, 12, 31') or must they useto_date?✅ - can user use
varto return a SQL statement '{{ var('my_future_date') }}'✅ - can they use a macro that returns a SQL statement '{{ dbt.date(9999, 12, 31) }}'?
✅ - how does this new config work with deferral/state:modified? will we warn users that the config has been updated and they need to manually update their snapshot?
🚀 Deployment available! Here are the direct links to the updated files: