Truncate timestamp to shift windowing in trade agg #87
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Checklist
PR Structure
otherwise).
Thoroughness
Release planning
semver, and I've changed the name of the BRANCH to release/* , feature/* or patch/* .
What
Truncate the timestamp on date filter to ensure that the full day of data is considered. The current code will only pull partial date for the current date, dependent on runtime
Why
Jenkins build pipeline is failing tests for
trade_agg
andint_trade_agg_day
because no data is pulled for the 1 day aggregate. This is because the current filter expects data newer than current timestamp:Added logic to truncate the timestamp to the beginning of the date, so that all data for the current run is pulled.
Known limitations
This is less of a problem in production because we pass Airflow variables. The jenkins build job relies on the default, current timestamp, instead of an env variable being passed in.
To make this fix permanent, we also need to pass current_date - 1 in via env var in the
jenkins_lint_and_test
build script.