Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework dispatch, with backwards compatibility for current use #3363

Merged
merged 2 commits into from
May 27, 2021

Conversation

jtcohen6
Copy link
Contributor

@jtcohen6 jtcohen6 commented May 17, 2021

resolves #3362
resolves #3383

This just adds a reproduction case for now. I've confirmed that, when I run pytest locally, it raises the same error we're seeing in the wild:

E       dbt.exceptions.CompilationException: Compilation Error in test pkg_and_dispatch_model_c_ (test-context-models/schema.yml)
E         'dict object' has no attribute 'current_timestamp'
E
E         > in macro test_pkg_and_dispatch (macros/custom_test.sql)
E         > called by macro default__test_pkg_and_dispatch (macros/custom_test.sql)
E         > called by test pkg_and_dispatch_model_c_ (test-context-models/schema.yml)

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt next" section.

Copy link
Contributor Author

@jtcohen6 jtcohen6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I confirmed that this offers backwards compatibility for the dbt-utils package. Hooray! I moved a little of the logic around, to account for when packages is a positional arg and also a Call, and I touched up the error message. (Looks like I messed up the flake8 linting, sorry about that.)

I'm so happy to be adding this logic and ensure projects have a smooth upgrade path to v0.19.2 + v0.20.0; I also want to get "the right way" to do this in place, and immediately recommend it to everyone who's upgrading. I laid out my sketch of a longer-term sustainable solution over in #3383.

Copy link
Contributor Author

@jtcohen6 jtcohen6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes are looking really good! I confirmed they worked with dbt-utils, both with:

  • The setup as it exists today
  • The changes we'll want to make to support the new dispatch project config

I left a few comments, nothing major.

Next steps from here:

  • Merge this PR into develop (or a new one, it doesn't have to be the one I stubbed out last week! haha)
  • Cherry-pick this PR onto 0.19.latest
  • Create a new PR, for inclusion in develop (v0.20) only, that raises a deprecation warning if the user either provides the packages kwarg or passes a list to the second positional arg

"dispatch": [
{
"macro_namespace": "test_utils",
"search_order": ['test_utils'],
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we adjust this test, or add another one, to confirm that when a user sets this to

"search_order": ['another_package', 'test_utils']

dbt will indeed prefer the version of the macro defined in another_package? Ultimately, that's the dispatch functionality that we want to be safeguarding here.

core/dbt/clients/jinja_static.py Show resolved Hide resolved
core/dbt/clients/jinja_static.py Outdated Show resolved Hide resolved
@jtcohen6 jtcohen6 changed the title Fix #3362 Rework dispatch, with backwards compatibility for current use May 24, 2021
@jtcohen6 jtcohen6 marked this pull request as ready for review May 24, 2021 21:53
used in parsing schema tests by looking at the arguments to
adapter.dispatch. Includes providing an alternative way of specifying
macro search order in project config.
Collaboratively developed with Jeremy Cohen.
@gshank gshank requested a review from kwigley May 25, 2021 15:42
@jtcohen6 jtcohen6 merged commit 98c015b into develop May 27, 2021
@jtcohen6 jtcohen6 deleted the fix/bug-3362 branch May 27, 2021 21:13
@jtcohen6 jtcohen6 mentioned this pull request May 27, 2021
2 tasks
gshank added a commit that referenced this pull request May 28, 2021
…be (#3363)

used in parsing schema tests by looking at the arguments to
adapter.dispatch. Includes providing an alternative way of specifying
macro search order in project config.
Collaboratively developed with Jeremy Cohen.

Co-authored-by: Gerda Shank <gerda@fishtownanalytics.com>
gshank added a commit that referenced this pull request May 28, 2021
…be (#3363)

used in parsing schema tests by looking at the arguments to
adapter.dispatch. Includes providing an alternative way of specifying
macro search order in project config.
Collaboratively developed with Jeremy Cohen.

Co-authored-by: Gerda Shank <gerda@fishtownanalytics.com>
gshank added a commit that referenced this pull request May 28, 2021
…be (#3363)

used in parsing schema tests by looking at the arguments to
adapter.dispatch. Includes providing an alternative way of specifying
macro search order in project config.
Collaboratively developed with Jeremy Cohen.

Co-authored-by: Gerda Shank <gerda@fishtownanalytics.com>
gshank added a commit that referenced this pull request May 29, 2021
…be (#3363)

used in parsing schema tests by looking at the arguments to
adapter.dispatch. Includes providing an alternative way of specifying
macro search order in project config.
Collaboratively developed with Jeremy Cohen.

Co-authored-by: Gerda Shank <gerda@fishtownanalytics.com>
gshank added a commit that referenced this pull request May 31, 2021
…be (#3363)

used in parsing schema tests by looking at the arguments to
adapter.dispatch. Includes providing an alternative way of specifying
macro search order in project config.
Collaboratively developed with Jeremy Cohen.

Co-authored-by: Gerda Shank <gerda@fishtownanalytics.com>
iknox-fa pushed a commit that referenced this pull request Feb 8, 2022
…be (#3363)

used in parsing schema tests by looking at the arguments to
adapter.dispatch. Includes providing an alternative way of specifying
macro search order in project config.
Collaboratively developed with Jeremy Cohen.

Co-authored-by: Gerda Shank <gerda@fishtownanalytics.com>

automatic commit by git-black, original commits:
  98c015b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Rework adapter.dispatch(), remove packages arg Test parsing context: package + macro deps + dispatch, oh my
3 participants