Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework _dbt_max_partition #41

Merged
merged 2 commits into from
Oct 21, 2021
Merged

Rework _dbt_max_partition #41

merged 2 commits into from
Oct 21, 2021

Conversation

jtcohen6
Copy link
Contributor

@jtcohen6 jtcohen6 commented Oct 14, 2021

resolves #17
resolves #39

Summary of changes

  • Rather than hard-coding the declaration of _dbt_max_partition in the dynamic insert-overwrite script, shell out to a macro declare_dbt_max_partition() that users can override with custom logic
  • Keep the default logic simple (+ expensive) for now, but only declare _dbt_max_partition if _dbt_max_partition is present in the compiled SQL
  • That custom logic could include:
    • disabling this variable entirely
    • using get_partitions_metadata (i.e. querying $__PARTITIONS_SUMMARY__ with legacySQL)
    • using INFORMATION_SCHEMA.PARTITIONS (standardSQL)
  • Include _dbt_max_partition declaration alongside temp table creation, as needed for on_schema_change. Add an integration test for this. (Should we try to be fancier about this, and actually append _dbt_max_partition to the sql_header config? I'd prefer to keep this simple + janky for now.)

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-bigquery next" section.

@cla-bot cla-bot bot added the cla:yes label Oct 14, 2021
Copy link

@alexras alexras left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jtcohen6 jtcohen6 marked this pull request as ready for review October 21, 2021 10:48
@jtcohen6 jtcohen6 merged commit 24cfa97 into main Oct 21, 2021
@jtcohen6 jtcohen6 deleted the refactor/dbt-max-partition branch October 21, 2021 13:00
siephen pushed a commit to AgencyPMG/dbt-bigquery that referenced this pull request May 16, 2022
* Rework _dbt_max_partition

* Add changelog entry
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
3 participants