Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Truncate timestamp to shift windowing in trade agg #87

Merged
merged 1 commit into from
Sep 6, 2024

Conversation

sydneynotthecity
Copy link
Contributor

PR Checklist

PR Structure

  • This PR has reasonably narrow scope (if not, break it down into smaller PRs).
  • This PR avoids mixing refactoring changes with feature changes (split into two PRs
    otherwise).
  • This PR's title starts with the jira ticket associated with the PR.

Thoroughness

  • This PR adds tests for the most critical parts of the new functionality or fixes.
  • I've updated the docs and README with the added features, breaking changes, new instructions on how to use the repository.

Release planning

  • I've decided if this PR requires a new major/minor/patch version accordingly to
    semver, and I've changed the name of the BRANCH to release/* , feature/* or patch/* .

What

Truncate the timestamp on date filter to ensure that the full day of data is considered. The current code will only pull partial date for the current date, dependent on runtime

Why

Jenkins build pipeline is failing tests for trade_agg and int_trade_agg_day because no data is pulled for the 1 day aggregate. This is because the current filter expects data newer than current timestamp:

where ledger_closed_at < timestamp_add('{{ dbt_airflow_macros.ts() }}',interval 1 day )
and ledger_closed_at >= '{{ dbt_airflow_macros.ts() }}'

Added logic to truncate the timestamp to the beginning of the date, so that all data for the current run is pulled.

Known limitations

This is less of a problem in production because we pass Airflow variables. The jenkins build job relies on the default, current timestamp, instead of an env variable being passed in.

To make this fix permanent, we also need to pass current_date - 1 in via env var in the jenkins_lint_and_test build script.

@sydneynotthecity sydneynotthecity requested a review from a team as a code owner September 6, 2024 20:20
@@ -120,7 +120,7 @@ with
/* obtain aggregate function metrics for the asset pair */
, trade_day_agg_group as (
select
date('{{ dbt_airflow_macros.ts(timezone=none) }}') as day_agg
date('{{ dbt_airflow_macros.ts() }}') as day_agg
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering. Is there an issue with using timezones=none? I'd assume that's the same as the default UTC

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there isn't, but it is the default. My thought was why specify the default

@sydneynotthecity sydneynotthecity merged commit a1fd6e4 into master Sep 6, 2024
3 checks passed
@sydneynotthecity sydneynotthecity deleted the patch/fix-trade-agg-dates branch November 14, 2024 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants