Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

T 15666 fix dbt source snapshot freshness warn instead of error #1483

Merged

Conversation

soispoke
Copy link
Contributor

@soispoke soispoke commented Sep 2, 2022

Brief comments on the purpose of your changes:
This PR aims to use the source freshness utility better. Since the DX team doesn't ultimately errors/delays on base and decoded tables, I propose to have the source freshness command attached to our daily job to check if everything looks good, but instead of throwing errors (that ultimately end up in failed jobs), throwing warnings so we can alert the data platform team without having to worry about failed jobs on our end

For Dune Engine V2
I've checked that:
General checks:

  • I tested the query on dune.com after compiling the model with dbt compile (compiled queries are written to the target directory)
  • I used "refs" to reference other models in this repo and "sources" to reference raw or decoded tables
  • if adding a new model, I added a test
  • the filename is unique and ends with .sql
  • each sql file is a select statement and has only one view, table or function defined
  • column names are lowercase_snake_cased
  • if adding a new model, I edited the dbt project YAML file with new directory path for both models and seeds (if applicable)
  • if adding a new model, I edited the alter table macro to display new database object (table or view) in UI explorer
  • if adding a new materialized table, I edited the optimize table macro

Join logic:

  • if joining to base table (i.e. ethereum transactions or traces), I looked to make it an inner join if possible

Incremental logic:

  • I used is_incremental & not is_incremental jinja block filters on both base tables and decoded tables
    • where block_time >= date_trunc("day", now() - interval '1 week')
  • if joining to base table (i.e. ethereum transactions or traces), I applied join condition where block_time >= date_trunc("day", now() - interval '1 week')
  • if joining to prices view, I applied join condition where minute >= date_trunc("day", now() - interval '1 week')

@height
Copy link

height bot commented Sep 2, 2022

This pull request has been linked to 1 task:

💡Tip: Add "Close T-15666" to the pull request title or description, to a commit message, or in a comment to mark this task as "Done" when the pull request is merged.

@@ -3,7 +3,7 @@ version: 2
sources:
- name: tornado_cash_bnb
freshness:
warn_after: { count: 12, period: hour }
warn_after: { count: 7, period: day }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why 7 days here and 12h below? I think these should also be 12h if it's not too much

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's just because noone uses tornado cash anymore haha so I didn't want it to warn us all the time

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh lol okay then it's fine I guess

@soispoke
Copy link
Contributor Author

soispoke commented Sep 5, 2022

thanks @aalan3

@soispoke soispoke merged commit 56e0df2 into main Sep 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants