Support deferred tests

[jtcohen6]

Describe the feature

In general: It may be desirable to test changes to a test without needing to first run the models it selects from:

dbt test -m my_new_test --defer --state path/to/artifacts

The resources that my_new_test depend on are not selected, and as such, references to those resources will be deferred to the supplied manifest's namespace.

For "Slim CI": In order to execute new tests that reference old unchanged models, we'll need to be able to:

dbt test -m state:modified+ --defer --state path/to/artifacts

If a model has been modified, this should run tests against the new model. If a test is new/modified, but the model it selects from is not, this should run tests against the deferred model.

This is intuitively tricky for relationships tests or data tests which depend on multiple models. If one model is included in the node selection criteria, and another model is not included, the test should defer only the latter's reference.

The only open question for me is: What should full test do?

dbt test --defer --state path/to/artifacts

Are all model nodes selected here?

• If so, this is identical to dbt test. The question is, how would you write "run all my tests and defer all their references"? Excluding models in this syntax would also exclude their dependent tests.

• If not, all references should be deferred. This feels like the right approach, but it's different from the dbt run analogue. dbt run --defer --state path/to/artifacts has identical behavior to dbt run (no deferral). dbt run selects all models and so defers nothing, whereas dbt test selects all tests but no models and so defers everything.

Describe alternatives you've considered

• This adds complexity to deferral. The alternative is to not do it for a while.

[jtcohen6]

The gap we planned to fill with dbt test --defer is better addressed with selectors. In particular, see discussion in #2704. Closing this issue and its corresponding PR (#2706) in the meantime.