Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: add gateway name to model cache identifier #2987

Merged
merged 2 commits into from
Aug 7, 2024

Conversation

treysp
Copy link
Contributor

@treysp treysp commented Aug 7, 2024

Model definitions are cached by the SQLMesh loader. Currently, the cached model ID consists of the latest time any project files were modified, a fingerprint of the context config object, and the context's default catalog.

If a model uses the @gateway macro variable, its value will be stored in the model's python environment and included in the cache. If the project is run with a different gateway, a cache hit can occur with the wrong gateway value being rendered into the model.

This PR fixes this by adding the gateway name to the model cache ID.

Implications:

  • All project models will have a cache miss when the gateway changes. The miss will be unnecessary for all models that do not use the @gateway macro variable.
  • An alternative approach is to leave the cache ID as-is, and after cache read check whether the gateway has changed and the @gateway macro variable is used. That approach will result in unnecessary cache hits when the gateway has changed and is used in the model.

@treysp treysp requested a review from eakmanrq August 7, 2024 20:37
sqlmesh/core/loader.py Outdated Show resolved Hide resolved
@treysp treysp merged commit 9c575d3 into main Aug 7, 2024
21 checks passed
@treysp treysp deleted the trey/model-cache-gateway branch August 7, 2024 21:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants