[CT-1581] [Epic] dbt-core as a library: first steps

[ChenyuLInx]

The scope of this epic is adding new functionality to dbt-core, on top of the foundations being laid in #5527, that will enable its top-level commands to be invoked via Python—and to make that a delightful experience.

In addition, we'll be tracking lower-level changes that become relevant in the context of programmatic invocations, including (though not exclusively) integration with dbt-server + dbt Cloud.

This represents a first big step on a much longer journey, toward dbt-core as a mature & reliable software library. The initial set of functional APIs that we're aiming to include in v1.5 will be, by intention and necessity, a small subset of everything dbt-core is capable of doing today.

Ready To implement

New commands

These represent long-awaited functionality that we should expose as top-level commands, both in the CLI + API. They should be highly useful during programmatic invocations, and they provide parity with functionality that's been historically available in dbt-rpc.

• [CT-1584] [Feature] New top level commands: interactive compile #6358
• [CT-1585] [Feature] New top level commands: interactive preview #6359

Follow-up:

• [CT-1751] Config to optionally skip population of relation cache #6526
• [CT-2295] Refactor expected_duplicate_params #7156

Changes to parsing

There is one fairly clear point of separation in dbt-core today: the end of parsing, and the beginning of execution. That remains an imperfect interface; this represents the work required to refactor & solidify it.

• [CT-1582] [Feature] Refactor tasks to breakout parsing the manifest into a separate piece #6357

UX for programmatic invocations

We're talking about a totally new way to invoke dbt-core (at least as the official ones go). Let's make sure that we have the requisite constructs, entry points, syntactic sugar, and documentation to support intermediate/advanced community members who want to start using it.

This is an area where we're especially eager for feedback from the wider community!

• [CT-1602] [Feature] a thin layer for calling dbt top level commands programatically #6381
• [CT-1770] [Feature] [Spike] Review programmatic entry point for top-level commands #6550

Integration with dbt-server

Track ongoing dbt-server work items that are relevant to or enabled by changes in this epic.

• [CT-1583][Feature] Utilize new way of providing manifest for tasks dbt-server#127
• Support partial update(similar to partial parsing) of a state via API?  dbt-server#128
• [CT-1928] Support passing event callbacks to EventManager via dbtRunner #6763
• [CT-2049] Fire an event at the end of every command #6878
• [CT-2205] [Feature] Support supply params in dbtRunner.invoke to overwrite flags #7070

Current in refinement:

• [CT-2236] [Feature] Remove lib.py [as much as possible] #7093
• [CT-1767] [Feature] dbt parse should return a manifest #6547
• [CT-2254] dbt debug for connection testing #7104

Need to create implementation ticket

• [CT-1763] [Feature] Configurable behavior for print #6543

To reprioritize later on

Spikes & investigations

• Triggering re-parses based on changes to env vars, --vars, credentials: Support partial update(similar to partial parsing) of a state via API?  dbt-server#128 (comment)
• De-globalizing adapters: [CT-909] Adapter Management works with Click #5533 (comment)

Lower-level changes

There are simplifying assumptions we can make about most dbt-core invocations: they have direct access to a terminal & file system, they're running one process at a time (zero concurrency), all steps take place in sequence, and it all happens in one OS/environment.

We don't anticipate these changes being relevant for most users who are interacting with dbt-core, whether via the CLI, or via scripting around its programmatic API. The current behavior will remain the sensible default. We still encourage any community members who are interested to weigh into any of these issues.

• [CT-1621] [Feature] de-globalize things through contextvar #6399
• [CT-1763] [Feature] Configurable behavior for print #6543
• [CT-1764] [Feature] write_file not to disk #6544
• [CT-1806] [Spike] Measure usage of certain functions(os.environ, os.path) #6587 → [CT-1765] [Feature] Provide env vars as data during runtime #6545
• [CT-1847] Move MP_CONTEXT somewhere else. #6647

[jtcohen6]

Closing in favor of #7162, which defines narrower scope of work to ship in v1.5