[CT-2536] [Feature] Enable indirect_selection as a global config

[otttt]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

By default indirect-selection is eager. Aim is to define in on project level as cautious or buildable to reduce code in selectors.

Describe alternatives you've considered

Right now my solution is to define it separately in every selector as cautious.

Who will this benefit?

No response

Are you interested in contributing this feature?

No response

Anything else?

No response

[dbeatty10]

Thanks for reaching out @otttt !

Did you already try setting the DBT_INDIRECT_SELECTION environment variable?

[dbeatty10]

If you are using YAML selectors, you can also do something like this:

selectors:
  - name: nodes_to_joy
    definition:
      method: tag
      value: nightly
      parents: true
      indirect_selection: cautious

[otttt]

Looking into DBT_INDIRECT_SELECTION environment variable.

I am already using indirect_selection: cautious in selectors but the issue is that I have to define it for every selector and I have like 20 of them.

[dbeatty10]

Looking into DBT_INDIRECT_SELECTION environment variable.

Awesome! Let us know how it goes.

[josh-acuitymd]

I would actually also appreciate this being a global-config! We use Dagster to orchestrate our dbt jobs which makes it difficult to supply the CLI argument. We could try and define some YAML selectors but being able to have it as a global config would certainly make everything so much simpler.

It was suggested in this slack thread to raise an issue as it seems making the parameter a global config had already been discussed.

[samkorn-acuitymd]

Upvoting this! We've had a number of issues with tests being run out of order, and failing because the downstream asset is not even materialized when a test referencing it is run. And because of the orchestration environment we use, we can't really use YAML selectors.

[otttt]

Looking into DBT_INDIRECT_SELECTION environment variable.

Awesome! Let us know how it goes.

Tried global environment variable by defining DBT_INDIRECT_SELECTION="cautious" but it was not picked up. A test which is not supposed to run is still picked up.
Also tried CLI flag --indirect-selection=cautious but same result, cautious is not picked up and the test I don't want in scope is still there.

Only option right now working for me is defining indirect_selection: cautious in selectors.yml

[josh-acuitymd]

Yeah I even tried to define it in the profiles.yml file setting the variable as indirect_select: cautious similar to how it's done here and still having no luck.

[dbeatty10]

Thanks for checking that @otttt I opened up #7673 as a result since I expected --indirect-selection and DBT_INDIRECT_SELECTION to apply to selectors in selectors.yml as well.

[dbeatty10]

@josh-acuitymd and @samkorn-acuitymd which of the following are you able to use with Dagster and which are you not able?

1. indirect_selection in selectors.yml
2. --indirect-selection CLI argument along with --select
3. DBT_INDIRECT_SELECTION environment variable along with --select

Note that the first one uses --selector and the last two use --select.

[josh-acuitymd]

@dbeatty10 I can try and reach out to the Dagster team to see if this can be modified but as far as I know Dagster only runs dbt run --select {whatever assets you request} or dbt build --select {whatever assets you request. So that may exclude selectors and the CLI argument options though I can try and experiment with hacking the CLI argument into it.

As @otttt mentioned, I'm not sure dbt is picking up the DBT_INDIRECT_SELECTION environment variable as that was the first thing I tried upon seeing this issue. I've been testing by defining a seed, creating a downstream model that relies upon the seed for a test, and building the seed (dbt build --select seed_model) without ever building the downstream asset. The tests always run and fail.

[dbeatty10]

@josh-acuitymd Sounds good -- please let us know what you hear back from the Dagster team.

In terms of DBT_INDIRECT_SELECTION, the following worked correctly for me using dbt-core==1.5.0 and zsh on macOS:

export DBT_INDIRECT_SELECTION=buildable
echo $DBT_INDIRECT_SELECTION
dbt ls --select model_b

[josh-acuitymd]

🤦‍♂️. After upgrading to 1.5.0 the DBT_INDIRECT_SELECTION environment variable worked but did not work for 1.4.6. It'd still be nice to be able to set this as a config but at least have an idea for a workaround for now. Thanks @dbeatty10 !!

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.