-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add selected_resources
to the Jinja context
#5001
Merged
ChenyuLInx
merged 7 commits into
main
from
feature/3471-add-selected_resources-to-context
Apr 12, 2022
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
989e97d
Add selected_resources in the Jinja context
b-per 78848b3
Add tests for the Jinja variable selected_resources
b-per 2df169f
Add Changie entry for the addition of selected_resources
b-per 015b3a9
Move variable to the ProviderContext
b-per d30ce14
Move selected_resources from ModelContext to ProviderContext
b-per b7b5ecd
Update unit tests for context to cater for the new selected_resources…
b-per f18a171
Move tests to a Class where tests are run after a dbt build
b-per File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
kind: Features | ||
body: Add a variable called selected_resources in the Jinja context containing a list | ||
of all the resources matching the nodes for the --select, --exclude and/or --selector | ||
parameters. | ||
time: 2022-04-04T19:04:39.347479+02:00 | ||
custom: | ||
Author: b-per | ||
Issue: "3471" | ||
PR: "5001" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
SELECTED_RESOURCES = [] | ||
|
||
|
||
def set_selected_resources(selected_resources): | ||
global SELECTED_RESOURCES | ||
SELECTED_RESOURCES = list(selected_resources) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
on_run_start_macro_assert_selected_models_expected_list = """ | ||
{% macro assert_selected_models_expected_list(expected_list) %} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Really like the way you use this macro to test the functionality! |
||
|
||
{% if execute and (expected_list is not none) %} | ||
|
||
{% set sorted_selected_resources = selected_resources | sort %} | ||
{% set sorted_expected_list = expected_list | sort %} | ||
|
||
{% if sorted_selected_resources != sorted_expected_list %} | ||
{{ exceptions.raise_compiler_error("FAIL: sorted_selected_resources" ~ sorted_selected_resources ~ " is different from " ~ sorted_expected_list) }} | ||
{% endif %} | ||
|
||
{% endif %} | ||
|
||
{% endmacro %} | ||
""" | ||
|
||
|
||
my_model1 = """ | ||
select 1 as id | ||
""" | ||
|
||
my_model2 = """ | ||
select * from {{ ref('model1') }} | ||
""" | ||
|
||
my_snapshot = """ | ||
{% snapshot cc_all_snapshot %} | ||
{{ config( | ||
check_cols='all', unique_key='id', strategy='check', | ||
target_database=database, target_schema=schema | ||
) }} | ||
select * from {{ ref('model2') }} | ||
{% endsnapshot %} | ||
""" |
104 changes: 104 additions & 0 deletions
104
tests/functional/selected_resources/test_selected_resources.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,104 @@ | ||
import pytest | ||
from dbt.tests.util import run_dbt | ||
from tests.functional.selected_resources.fixtures import ( | ||
on_run_start_macro_assert_selected_models_expected_list, | ||
my_model1, | ||
my_model2, | ||
my_snapshot, | ||
) | ||
|
||
|
||
@pytest.fixture(scope="class") | ||
def macros(): | ||
return { | ||
"assert_selected_models_expected_list.sql": on_run_start_macro_assert_selected_models_expected_list, | ||
} | ||
|
||
|
||
@pytest.fixture(scope="class") | ||
def models(): | ||
return {"model1.sql": my_model1, "model2.sql": my_model2} | ||
|
||
|
||
@pytest.fixture(scope="class") | ||
def snapshots(): | ||
return { | ||
"my_snapshot.sql": my_snapshot, | ||
} | ||
|
||
|
||
@pytest.fixture(scope="class") | ||
def project_config_update(): | ||
return { | ||
"on-run-start": "{{ assert_selected_models_expected_list(var('expected_list',None)) }}", | ||
} | ||
|
||
|
||
@pytest.fixture | ||
def build_all(project): | ||
run_dbt(["build"]) | ||
|
||
|
||
@pytest.mark.usefixtures("build_all") | ||
class TestSelectedResources: | ||
def test_selected_resources_build_selector(self, project): | ||
results = run_dbt( | ||
[ | ||
"build", | ||
"--select", | ||
"model1+", | ||
"--vars", | ||
'{"expected_list": ["model.test.model1", "model.test.model2", "snapshot.test.cc_all_snapshot"]}', | ||
] | ||
) | ||
assert results[0].status == "success" | ||
|
||
def test_selected_resources_build_selector_subgraph(self, project): | ||
results = run_dbt( | ||
[ | ||
"build", | ||
"--select", | ||
"model2+", | ||
"--vars", | ||
'{"expected_list": ["model.test.model2", "snapshot.test.cc_all_snapshot"]}', | ||
] | ||
) | ||
assert results[0].status == "success" | ||
|
||
def test_selected_resources_run(self, project): | ||
results = run_dbt( | ||
[ | ||
"run", | ||
"--select", | ||
"model1+", | ||
"--vars", | ||
'{"expected_list": ["model.test.model2", "model.test.model1"]}', | ||
] | ||
) | ||
assert results[0].status == "success" | ||
|
||
def test_selected_resources_build_no_selector(self, project): | ||
results = run_dbt( | ||
[ | ||
"build", | ||
"--vars", | ||
'{"expected_list": ["model.test.model1", "model.test.model2", "snapshot.test.cc_all_snapshot"]}', | ||
] | ||
) | ||
assert results[0].status == "success" | ||
|
||
def test_selected_resources_build_no_model(self, project): | ||
results = run_dbt( | ||
[ | ||
"build", | ||
"--select", | ||
"model_that_does_not_exist", | ||
"--vars", | ||
'{"expected_list": []}', | ||
] | ||
) | ||
assert not results | ||
|
||
def test_selected_resources_test_no_model(self, project): | ||
results = run_dbt(["test", "--select", "model1+", "--vars", '{"expected_list": []}']) | ||
assert not results |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gshank could there be situations where
selected_resources
is referenced before it got set byget_graph_queue
? I think there isn't but want to double-check.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the selected_resources will be empty unless it's execution time, which should be okay. I suppose the list could be really big for a run command that executes the whole project, but I think that should be okay too; it doesn't do much except reference the list of nodes.