Add `selected_resources` to the Jinja context #5001

b-per · 2022-04-06T12:04:06Z

resolves #3471

Description

I don't know if I followed the correct approach but this code is adding a variable called selected_resources to the Jinja context. This is a list of all models/tests/snapshots etc... captured by the selectors for the dbt run. I checked how flags were added to the context and did the same for selected_resources, hence why it is under the core/dbt folder.

The issue #3471 also mentions making this available for run-operation but this will be taken care of in another issue.

My testing approach is to use a on-run-start hook to compare the value of selected_resources with the expected one.

If this is merged the docs will need to be updated to mention the new variable and highlight that similar to graph is will only contain some information at execute time.

Checklist

I have signed the CLA
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have added information about my change to be included in the CHANGELOG.

github-actions · 2022-04-06T12:04:21Z

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

gshank · 2022-04-06T14:06:50Z

core/dbt/context/base.py

@@ -556,6 +557,15 @@ def flags(self) -> Any:
        """
        return flags

+    @contextproperty
+    def selected_resources(self) -> Any:


I'm pretty sure that the base context is the wrong context for this functionality. We don't need this in yaml rendering and other non-sql rendering. It probably ought to go in a provider context. Is there some reason that you put it in base?

I put it in Base because I looked at howflags were configured and replicated some of the logic for selected_resources, but I get your point and it makes much more sense to be in another context. I will update the code.

I have just moved it to the ProviderContext

gshank

Looks like you need to update the tests. In test_context.py you need to add 'selected_resources' to REQUIRED_MACRO_KEYS.

ChenyuLInx

LGTM with one nitpicking item for tests. I feel like this is ready to go once other tests are fixed.

ChenyuLInx · 2022-04-06T23:26:07Z

tests/functional/selected_resources/fixtures.py

@@ -0,0 +1,35 @@
+on_run_start_macro_assert_selected_models_expected_list = """
+{% macro assert_selected_models_expected_list(expected_list) %}


Really like the way you use this macro to test the functionality!

ChenyuLInx · 2022-04-06T23:28:21Z

tests/functional/selected_resources/test_selected_resources.py

+        [
+            "build",
+            "--select",
+            "model1+",


Should we do a model2+ as one of the test cases? since the result of this selection is the same as the no select specified test below.

I just modified a test to be model2+

It was failing at first (because model1 was not built) so I moved all the tests to a Class that uses the fixture build_all, meaning that a full dbt build is run before each test (which allows us to select nodes that are depending on other ones)

… variable

ChenyuLInx

LGTM! Left one question but not a blocker

ChenyuLInx · 2022-04-12T16:04:55Z

.changes/unreleased/Features-20220404-190439.yaml

@@ -0,0 +1,9 @@
+kind: Features
+body: Add a variable called selected_resources in the Jinja context containing a list


@gshank could there be situations where selected_resources is referenced before it got set by get_graph_queue ? I think there isn't but want to double-check.

I think the selected_resources will be empty unless it's execution time, which should be okay. I suppose the list could be really big for a run command that executes the whole project, but I think that should be okay too; it doesn't do much except reference the list of nodes.

the test has been updated

* Bumping version to 1.1.0rc1 * Updating requirements to latest release branch * Bumping manifest schema in tests * Fix unit test per dbt-labs/dbt-core#5001 Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com> Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com> Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>

* Bumping version to 1.2.0a1 * Bumping schema version in tests * Try skipping test-build if alpha version * try this * i meant this * Fix unit test per dbt-labs/dbt-core#5001 * One less change than needed Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com> Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>

* Add selected_resources in the Jinja context * Add tests for the Jinja variable selected_resources * Add Changie entry for the addition of selected_resources * Move variable to the ProviderContext * Move selected_resources from ModelContext to ProviderContext * Update unit tests for context to cater for the new selected_resources variable * Move tests to a Class where tests are run after a dbt build

* Bumping version to 1.2.0a1 * Bumping schema version in tests * Try skipping test-build if alpha version * try this * i meant this * Fix unit test per dbt-labs/dbt-core#5001 * One less change than needed Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com> Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>

b-per added 2 commits April 6, 2022 13:43

Add selected_resources in the Jinja context

989e97d

Add tests for the Jinja variable selected_resources

78848b3

b-per requested a review from a team April 6, 2022 12:04

b-per requested review from a team as code owners April 6, 2022 12:04

b-per requested review from ChenyuLInx and gshank April 6, 2022 12:04

cla-bot bot added the cla:yes label Apr 6, 2022

Add Changie entry for the addition of selected_resources

2df169f

jtcohen6 added Team:Execution labels Apr 6, 2022

gshank requested changes Apr 6, 2022

View reviewed changes

Move variable to the ProviderContext

015b3a9

gshank previously requested changes Apr 6, 2022

View reviewed changes

ChenyuLInx reviewed Apr 6, 2022

View reviewed changes

b-per added 3 commits April 7, 2022 10:32

Move selected_resources from ModelContext to ProviderContext

d30ce14

Update unit tests for context to cater for the new selected_resources…

b7b5ecd

… variable

Move tests to a Class where tests are run after a dbt build

f18a171

b-per mentioned this pull request Apr 7, 2022

[CT-466] [Feature] Make run-operation accept selectors to be able to use the selected_resources Jinja variable #5005

Open

1 task

b-per requested a review from gshank April 11, 2022 15:46

ChenyuLInx approved these changes Apr 12, 2022

View reviewed changes

ChenyuLInx self-requested a review April 12, 2022 16:23

ChenyuLInx approved these changes Apr 12, 2022

View reviewed changes

ChenyuLInx merged commit 15ad34e into main Apr 12, 2022

ChenyuLInx deleted the feature/3471-add-selected_resources-to-context branch April 12, 2022 16:25

b-per mentioned this pull request Apr 13, 2022

Add selected_resources Jinja context to docs - v1.1 dbt-labs/docs.getdbt.com#1333

Merged

7 tasks

jtcohen6 added a commit to dbt-labs/dbt-redshift that referenced this pull request Apr 13, 2022

Fix unit test per dbt-labs/dbt-core#5001

801d79d

jtcohen6 added a commit to dbt-labs/dbt-redshift that referenced this pull request Apr 13, 2022

Fix unit test per dbt-labs/dbt-core#5001

3221523

gshank mentioned this pull request Apr 18, 2022

[CT-504] Intermittent test failure in 'selected_resource' test. #5088

Closed

vergenzt mentioned this pull request May 14, 2022

Jinja context variable for selected resources #3471

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `selected_resources` to the Jinja context #5001

Add `selected_resources` to the Jinja context #5001

b-per commented Apr 6, 2022

github-actions bot commented Apr 6, 2022

gshank Apr 6, 2022

b-per Apr 6, 2022

b-per Apr 6, 2022

gshank left a comment

ChenyuLInx left a comment

ChenyuLInx Apr 6, 2022

ChenyuLInx Apr 6, 2022

b-per Apr 7, 2022

ChenyuLInx left a comment

ChenyuLInx Apr 12, 2022

gshank Apr 12, 2022

		@@ -0,0 +1,35 @@
		on_run_start_macro_assert_selected_models_expected_list = """
		{% macro assert_selected_models_expected_list(expected_list) %}

		@@ -0,0 +1,9 @@
		kind: Features
		body: Add a variable called selected_resources in the Jinja context containing a list

Add selected_resources to the Jinja context #5001

Add selected_resources to the Jinja context #5001

Conversation

b-per commented Apr 6, 2022

Description

Checklist

github-actions bot commented Apr 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gshank left a comment

Choose a reason for hiding this comment

ChenyuLInx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChenyuLInx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Add `selected_resources` to the Jinja context #5001

Add `selected_resources` to the Jinja context #5001