Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add custom granularities to mf timespine #6145

Merged
merged 27 commits into from
Oct 3, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
d33aa6b
add custom granularities
mirnawong1 Sep 25, 2024
cf6500a
fix space
mirnawong1 Sep 25, 2024
07816ee
fix order
mirnawong1 Sep 25, 2024
15e86c2
add headers
mirnawong1 Sep 25, 2024
f1686d4
Merge branch 'current' into mwong-custom-calendar
mirnawong1 Sep 25, 2024
4b9ebc4
Merge branch 'current' into mwong-custom-calendar
mirnawong1 Sep 26, 2024
e7c5156
fold feedback
mirnawong1 Sep 26, 2024
3c0bd33
fix
mirnawong1 Sep 26, 2024
4a5b15c
Merge branch 'current' into mwong-custom-calendar
mirnawong1 Sep 26, 2024
0d60378
Merge branch 'current' into mwong-custom-calendar
mirnawong1 Sep 30, 2024
09e3a3a
Update metricflow-time-spine.md
mirnawong1 Sep 30, 2024
1f1e177
Update metricflow-time-spine.md
mirnawong1 Sep 30, 2024
89d3d1d
Merge branch 'current' into mwong-custom-calendar
mirnawong1 Oct 2, 2024
42132b9
Merge branch 'current' into mwong-custom-calendar
mirnawong1 Oct 2, 2024
405130f
Update release-notes.md
mirnawong1 Oct 2, 2024
98243aa
Merge branch 'current' into mwong-custom-calendar
mirnawong1 Oct 2, 2024
f5bd3d4
Merge branch 'current' into mwong-custom-calendar
mirnawong1 Oct 2, 2024
2921f0e
Merge branch 'current' into mwong-custom-calendar
mirnawong1 Oct 2, 2024
f626ab4
Update website/docs/docs/build/metricflow-time-spine.md
mirnawong1 Oct 2, 2024
60eef89
Update website/docs/docs/dbt-versions/release-notes.md
mirnawong1 Oct 2, 2024
46aabda
Merge branch 'current' into mwong-custom-calendar
mirnawong1 Oct 3, 2024
e023b96
Update website/docs/docs/dbt-versions/release-notes.md
mirnawong1 Oct 3, 2024
8ebdccc
Update release-notes.md
mirnawong1 Oct 3, 2024
beefe57
Merge branch 'current' into mwong-custom-calendar
mirnawong1 Oct 3, 2024
5e6a7a7
Update release-notes.md
mirnawong1 Oct 3, 2024
dd8547e
Merge branch 'current' into mwong-custom-calendar
mirnawong1 Oct 3, 2024
1ad6362
Merge branch 'current' into mwong-custom-calendar
mirnawong1 Oct 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 80 additions & 3 deletions website/docs/docs/build/metricflow-time-spine.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,13 @@ MetricFlow requires you to define a time-spine table as a model-level configurat
- [Metric offsets](/docs/build/derived#derived-metric-offset)
- [Conversion metrics](/docs/build/conversion)
- [Slowly Changing Dimensions](/docs/build/dimensions#scd-type-ii)
- [Metrics](/docs/build/metrics-overview) with the `join_to_timespine` configuration set to true
To see the generated SQL for the metric and dimension types that use time-spine joins, refer to the respective documentation or add the `compile=True` flag when querying the Semantic Layer to return the compiled SQL.
- [Metrics](/docs/build/metrics-overview) with the `join_to_timespine` configuration set to `true`. To see the generated SQL for the metric and dimension types that use time-spine joins, refer to the respective documentation or add the `compile=True` flag when querying the Semantic Layer to return the compiled SQL.

#### Configuring time-spine
### Configuring time-spine
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason we use "time-spine" instead of just "time spine"? Curious if @Jstein77 has an opinion on this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i was originally using time spine but then saw it used with a dash and thought/assumed i had a wrong. happy with time spine too! @Jstein77 are we good with time spine?

- You only need to configure time-spine models that the Semantic Layer should recognize.
- At a minimum, define a time-spine table for a daily grain.
- You can optionally define a time-spine table for a different granularity, like hourly.
- If your organization uses [custom calendars](#custom-calendar) (like fiscal years) or custom granularities (like `retail_month`), you can configure those as well. Learn how to [add custom calendars and granularities](#add-custom-granularities) further down this page to help support specific business needs.
- Note that if you don’t have a date or calendar model in your project, you'll need to create one.
- If you're looking to specify the grain of a time dimension so that MetricFlow can transform the underlying column to the required granularity, refer to the [Time granularity documentation](/docs/build/dimensions?dimension=time_gran)

Expand All @@ -36,13 +36,18 @@ If you don’t have a date dimension table, you can still create one by using th

<Lightbox src="/img/time_spines.png" title="Time spine directory structure" />

<VersionBlock firstVersion="1.9">
<File name="models/_models.yml">

```yaml
models:
- name: time_spine_hourly
description: "my favorite time spine"
time_spine:
standard_granularity_column: date_hour # column for the standard grain of your table
custom_granularities:
- name: fiscal_year
column_name: fiscal_year_column
columns:
- name: date_hour
granularity: hour # set granularity at column-level for standard_granularity_column
Expand All @@ -54,6 +59,29 @@ models:
granularity: day # set granularity at column-level for standard_granularity_column
```
</File>
</VersionBlock>

<VersionBlock lastVersion="1.8">
<File name="models/_models.yml">

```yaml
models:
- name: time_spine_hourly
description: "my favorite time spine"
time_spine:
standard_granularity_column: date_hour # column for the standard grain of your table
columns:
- name: date_hour
granularity: hour # set granularity at column-level for standard_granularity_column
- name: time_spine_daily
time_spine:
standard_granularity_column: date_day # column for the standard grain of your table
columns:
- name: date_day
granularity: day # set granularity at column-level for standard_granularity_column
```
</File>
</VersionBlock>

For an example project, refer to our [Jaffle shop](https://github.com/dbt-labs/jaffle-sl-template/blob/main/models/marts/_models.yml) example.

Expand Down Expand Up @@ -230,3 +258,52 @@ where date_day > dateadd(year, -4, current_timestamp())
and date_hour < dateadd(day, 30, current_timestamp())
```
</File>


## Custom calendar
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

<VersionBlock lastVersion="1.8">

Being able to configure custom calendars, such as like a fiscal calendar, is available in [dbt Cloud Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) or dbt Core [v1.9 and above](/docs/dbt-versions/core).

To access this feature, [upgrade to Versionless](/docs/dbt-versions/versionless-cloud) or dbt Core v1.9 and above.
</VersionBlock>

<VersionBlock firstVersion="1.9">

If you use a custom calendar in your organization, such as a fiscal calendar, you can configure it in MetricFlow using its date and time operations.

This is useful for calculating metrics based on a custom calendar, such as fiscal quarters or weeks. Use the `custom_granularities` key to define a non-standard time period for querying data, such as a `retail_month` or `fiscal_week`, instead of standard options like `day`, `month`, or `year`. This feature provides more control over how time-based metrics are calculated.


### Why use a custom calendar model?
Custom date transformations can be complex, and organizations often have unique needs that can’t be easily generalized. Creating a custom calendar model allows you to define these transformations in SQL, offering more flexibility than native transformations in MetricFlow. This approach lets you map custom columns back to MetricFlow granularities, ensuring consistency while giving you control over the transformations.

### Calendar model requirements

To use a custom calendar model, ensure it meets the following requirements:

- The model must be at a daily grain and joinable to other date columns using `date_day`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is a requirement, you could use a dim date table at an hourly grain and define custom granularities for that model.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++ you can use any of our standard granularities!

- Columns must align with expected data types for a given granularity (for example, `quarter` should be a date).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we update this to be a little more specific:
"The standard_granularity_column must be a date time type."
(Or something similar - datetime/date/timestamp I think are all fine but not sure if I'm missing options from any specific engines)
Also could move that note to the section about standard_granularity_column.
The other columns can be any type so I don't think we need to add any detail about that.

- Support period-over-period calculations.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't a requirement of the model, it's a requirement of for the functionality we support. We can remove it.

- Support cumulative metric windows using custom granularities (for example, cumulative fiscal year-to-date bookings).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as above, this is a requirement for us, not the underlying model


### Add custom granularities

To add custom granularities, the Semantic Layer supports custom calendar configurations that allow users to query data using non-standard time periods like `fiscal_year` or `retail_month`. You can define these custom granularities by modifying your model's YAML configuration like this:

<File name="models/_models.yml">

```yaml
models:
- name: my_time_spine
description: "my favorite time spine"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to wrap this in quotes

time_spine:
standard_granularity_column: date_day
custom_granularities:
- name: fiscal_year
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
column_name: fiscal_year_column
```
</File>

</VersionBlock>
1 change: 1 addition & 0 deletions website/docs/docs/dbt-versions/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo

## September 2024

- **New**: The dbt Semantic Layer supports custom calendar configurations in MetricFlow that allow you to query data using non-standard time periods like `fiscal_year` or `retail_month`. Refer to [custom calendar](/docs/build/metricflow-time-spine#custom-calendar) to learn how to define these custom granularities in your MetricFlow timespine YAML configuration.
- **New:** There are two new [environment variable defaults](/docs/build/environment-variables#dbt-cloud-context) &mdash; `DBT_CLOUD_ENVIRONMENT_NAME` and `DBT_CLOUD_ENVIRONMENT_TYPE`.
- **New:** The [Amazon Athena warehouse connection](/docs/cloud/connect-data-platform/connect-amazon-athena) is available as a public preview for dbt Cloud accounts that have upgraded to [`versionless`](/docs/dbt-versions/versionless-cloud).

Expand Down
Loading