Feature: Add optional lookback_interval param to insert_by_period materialization

[GClunies]

This is a:

• bug fix PR with no breaking changes — please ensure the base branch is master
• new functionality — please ensure the base branch is the latest dev/ branch
• a breaking change — please ensure the base branch is the latest dev/ branch

Description & motivation

The insert_by_period materialization does not work with sessionization models like segment_web_page_views__sessionized from the dbt Segment package due to some unusually complex logic. Since these sessionization models are often very large and complex, I would like to be able to use the insert_by_period materialization with them.

This is my first PR to a dbt repo, let alone a dev branch, so please let me know if any additional work is needed for this PR.

Also, huge thanks to Scott Barber on the dbt-labs customer support engineering team to get me started on refactoring the macro! His input and guidance were a massive help and this was a great learning experience for me.

Describe your changes, and why you're making them.

This PR adds an optional lookback_interval parameter to the insert_by_period materialization, allowing it to be used with sessionization models like segment_web_page_views__sessionized from the dbt Segment package.

This is achieved by allowing the end user to include __PERIOD_FILTER_WITH_LOOKBACK__ in their model SQL, which the materialization replaces at runtime of each period using this modified macro logic in the materialization code.

{% macro default__get_period_sql(target_cols_csv, sql, timestamp_field, period, start_timestamp, stop_timestamp, offset, lookback_interval) -%}

  {%- set period_filter -%}
    ("{{timestamp_field}}" >  '{{start_timestamp}}'::timestamp + interval '{{offset}} {{period}}' and
     "{{timestamp_field}}" <= '{{start_timestamp}}'::timestamp + interval '{{offset}} {{period}}' + interval '1 {{period}}' and
     "{{timestamp_field}}" <  '{{stop_timestamp}}'::timestamp)
  {%- endset -%}

  {%- set period_filter_with_lookback -%}
    ("{{timestamp_field}}" >  '{{start_timestamp}}'::timestamp + interval '{{offset}} {{period}}' - interval '{{lookback_interval}}' and
     "{{timestamp_field}}" <= '{{start_timestamp}}'::timestamp + interval '{{offset}} {{period}}' + interval '1 {{period}}' and
     "{{timestamp_field}}" <  '{{stop_timestamp}}'::timestamp)
  {%- endset -%}

  {%- set filtered_sql = sql | replace("__PERIOD_FILTER__", period_filter) | replace("__PERIOD_FILTER_WITH_LOOKBACK__", period_filter_with_lookback) -%}

  select
    {{target_cols_csv}}
  from (
    {{filtered_sql}}
  )

{%- endmacro %}

In addition to the integration tests I have added in this PR, I have tested the application of these changes to the materialization on my sessionization models @Surfline to ensure idempotency of session_ids is still maintained. I have added documentation on usage to the README.

The materialization can also be used on any model where loading by period is desired, but a lookback interval is also required to filter the upstream data (which is typically accomplished using incremental logic).

Checklist

• I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)
  • BigQuery
  • Postgres
  • Redshift
  • Snowflake
• I have "dispatched" any new macro(s) so non-core adapters can also use them (e.g. the star() source)
• I have updated the README.md (if applicable)
• I have added tests & descriptions to my models (and macros if applicable)
• I have added an entry to CHANGELOG.md

[GClunies]

I also see issue dbt-labs/dbt-labs-experimental-features#32 talking about modernizing this materialization. But, it looks like all of the changes discussed there have happened in the dbtvault repo and are largely Snowflake focused.

Would be great to modernize the original redshift version as well (or a universal materialization). But that work seems like a separate PR.

[barberscott]

@jtcohen6 since you have done a fair amount of work with Segment in the past, thought you might have the context to review this -- it's an application of insert_by_period that allows for a window to use for sessionization-type queries.

[jasnonaz]

Hi @GClunies! This is a really fantastic addition to the codebase and clearly has a ton of thought and care put into it. We are actually in the process of determining what the future of the insert_by_period materialization is. It's less battle tested and cross compatible and much more experimental than the level of assurance you would normally expect in dbt-utils and it'd quite likely we will be moving it into a repo for experimental functionality in the near future. We'd love this addition and other things like cross database functionality to get added when we do that.

In the meantime I'm going to close this as it isn't a good fit for utils right now but we really, really do appreciate the contribution and look forward to merging this in when we ultimately determine where insert_by_period is going to live.

[GClunies]

@jasnonaz thanks for the kind words and update! Agree that a separate repo makes a lot of sense. Feel free to ping me in the new repo when you get to it. For now I have this as a macro in our dbt project so no risk of a breaking change for me.