Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix!: Handle lookback correctly on incremental models with an end date defined #2962

Merged
merged 1 commit into from
Jul 30, 2024

Conversation

erindru
Copy link
Collaborator

@erindru erindru commented Jul 30, 2024

Slack context: https://tobiko-data.slack.com/archives/C044BRE5W4S/p1721917406454879

This PR allows lookback to return the correct missing intervals if the execution date is beyond the model's end date. Prior this, sqlmesh run would always show a phantom interval for these models even if the current date was well beyond the model end date + lookback period.

Assuming an incremental model with an end date of 2023-01-05, @daily intervals and a lookback of 2 days:

  • The user does not want data from the source system beyond 2023-01-05 so they don't want new intervals after 2023-01-05
  • However, they do want already run intervals to re-run after 2023-01-05 if they are within the lookback period
  • So the following intervals would run on the following days:
    • 2023-01-05 -> 2023-01-05, 2023-01-04, 2023-01-03
    • 2023-01-06 -> 2023-01-04, 2023-01-05
    • 2023-01-07 -> 2023-01-05
    • 2023-01-08 -> (none, we are done)

@erindru erindru force-pushed the erin/respect-model-end-date-lookback branch from 2339eef to 34c8e4b Compare July 30, 2024 01:55
@erindru erindru force-pushed the erin/respect-model-end-date-lookback branch from 34c8e4b to 18d8f1b Compare July 30, 2024 04:41
@erindru erindru merged commit 877787e into main Jul 30, 2024
20 of 21 checks passed
@erindru erindru deleted the erin/respect-model-end-date-lookback branch July 30, 2024 20:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants