Make it possible to run multiple Tasks in a single dbt process

[cmcarthur]

Subtask of #1274

In their current state, Tasks are tightly coupled to command line invocation:

• args is an argparse result
• config is built up during dbt invocation and passed to Tasks on instantiation

This makes it practically impossible to invoke Tasks when using as a library without going through the dbt.main codepaths. It is also impossible to invoke multiple tasks in a single dbt process.

Two things need to change to enable invoking Tasks when including dbt as a library:

1. Pass in the argparse result as a set of kwargs to the Task, instead of as args. This should be relatively simple.

2. Either: make it possible to instantiate Tasks without passing in a config (meaning all config generation would move into Tasks), and/or factor out the config-generating parts of the code from dbt.main so that it can be called separately before instantiating Tasks. The key config-generation code (RuntimeConfig.from_args for example) also requires the argparse result. It's not exactly clear to me how to better factor all this, but it seems possible given that the config can be generated from just the project + kwargs

With those changes in place, we should be able to (inefficiently) invoke multiple dbt Tasks inside a single Python process.

[dlawrences]

Hi @cmcarthur @drewbanin - sorry for reaching out to you guys on a closed task. I was wondering whether this behaviour is still supported [running multiple tasks (compile/run/test/whatever) in a single dbt process]?

The background to this is that my team and I are seeing some issues with respect to multi-task concurrency while running dbt-core via its Python API. We believe some state/context is shared between these events and leads to errors. This is not reproducible if we interact with dbt-core via the CLI (and typically under separate sub-processes).