Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dbt perf tuning #581

Draft
wants to merge 15 commits into
base: main
Choose a base branch
from
Draft
Prev Previous commit
Next Next commit
misc updates
  • Loading branch information
aaronsteers committed Feb 28, 2023
commit b5540723af48b8a4d22ecc20ca893ce5780e6fe3
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{{
config(
materialized='incremental'
materialized='incremental',
cluster_by=['event_id']
)
}}

Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
{{
# TODO: Debug performance. As of Feb 28, build time is approximately 40 minutes
config(
materialized='table'
)
@@ -14,8 +15,10 @@ WITH source AS (
ORDER BY event_created_at::TIMESTAMP DESC
) AS dedupe_rank
FROM {{ ref('stg_snowplow__events_union_all') }}

)
-- {% if is_incremental() %}
-- -- TODO: Is this safe or would we lose records?:
-- WHERE uploaded_at >= (SELECT max(UPLOADED_AT) FROM {{ this }})
-- {% endif %}

SELECT *
FROM source