[CT-1734] [Feature] Make the partitions_to_replace part in the macro bq_dynamic_insert_overwrite_sql into it's own macro

[StoryFabian]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt-bigquery functionality, rather than a Big Idea better suited to a discussion

Describe the feature

To be able to make it possible for dbt users to change the behaviour on how to get partitions to replace in insert overwrite incremental strategy it would be nice to move that part into it's own macro that could be overwritten to change the behaviour.

I'm talking about this part:

      -- 2. define partitions to update
      set (dbt_partitions_for_replacement) = (
          select as struct
              array_agg(distinct {{ partition_by.render_wrapped() }})
          from {{ tmp_relation }}
      );

in the file dbt/include/bigquery/macros/materializations/incremental_strategy/insert_overwrite.sql

Describe alternatives you've considered

Move the part above into its own macro can can be overwritten by dbt users to fulfill their needs.

Who will this benefit?

Anyone that want to change the behaviour of the insert overwrite incremental strategy

Are you interested in contributing this feature?

I believe I could write it myself if needed.

Anything else?

No response

[Fleid]

Hi @StoryFabian, would you mind having a look at what we're thinking of doing there and confirm it would fit your original requirements?

What I would like then, is an additional flag in the partition_by bag (like done for copy_partitions in 1.4) that signals the difference in behavior (default is current behavior) - below with replace_partitions:

{{ config(
    materialized='incremental',
    incremental_strategy='insert_overwrite',
    partition_by={
      "field": "user_id",
      "data_type": "int64",
      "range": {
        "start": 0,
        "end": 100,
        "interval": 10
      },
      "replace_partitions": "min_max"
    }
)}}

select
  user_id,
  event_name,
  created_at

from {{ ref('events') }}

I'm making it a enumeration rather than a boolean, as I'm expecting more strategies to appear (current would be list_values, an alternative could be distinct_partitions).

The idea would be to make your original concern a top level configuration.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.