Document dbtRunner (programmatic invocation) #3118

jtcohen6 · 2023-03-31T11:41:57Z

What are you changing in this pull request and why?

I've scaled back my original ambitions from #2674 / #2693, given that we de-prioritized some docs improvements, in favor of bugs/regressions before the v1.5 release (more important!).

I think some human-written docs for now are more than sufficient. I need to add examples for the "advanced" usage patterns, and also add more caveats about what users should/shouldn't expect to remain stable in future versions.

Checklist

Other acceptance criteria

remove this example script (completed in #7320)

Adding new pages (delete if not applicable):

Add page to website/sidebars.js
Provide a unique filename for the new page

netlify · 2023-03-31T11:42:05Z

✅ Deploy Preview for docs-getdbt-com ready!

Name	Link
🔨 Latest commit	`9ece56d`
🔍 Latest deploy log	https://app.netlify.com/sites/docs-getdbt-com/deploys/64383da02616f9000871b80a
😎 Deploy Preview	https://deploy-preview-3118--docs-getdbt-com.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

website/docs/reference/programmatic-invocations.md

aranke · 2023-04-12T18:31:36Z

website/docs/reference/programmatic-invocations.md

+title: "Programmatic invocations"
+---
+
+In v1.5, dbt-core added support for programmatic invocations. The intent of this entry point is provide **exact parity** with CLI functionality, callable from within a Python script or application.


Something like this?

The intent of this entry point is to expose the existing dbt CLI via a Python entrypoint, so it is callable from a Python script or application.

aranke · 2023-04-12T18:33:46Z

website/docs/reference/programmatic-invocations.md

+
+In v1.5, dbt-core added support for programmatic invocations. The intent of this entry point is provide **exact parity** with CLI functionality, callable from within a Python script or application.
+
+The main entry point is a `dbtRunner` class that wraps around `dbt-core`'s CLI, and allows you to "invoke" CLI commands as Python methods. Each command returns a `dbtRunnerResult` object, which has three attributes:


This should be the opposite? The CLI technically wraps the Click runner that dbtRunner depends on.

aranke · 2023-04-12T18:34:11Z

website/docs/reference/programmatic-invocations.md

+
+### `dbtRunnerResult` and exit codes
+
+There is a 1:1 correspondence between [CLI exit codes](reference/exit-codes) and the `dbtRunnerResult` returned by a programmatic invocation:


Let's put this in a table?

@aranke I think the table will be covered by the link.

i.e., [CLI exit codes](reference/exit-codes) will link to this page which has the following table:

I like the idea of tabling it up here as well!

Let's copy the same table here; most people aren't going to click on links.

Wait, it's a different table – here the columns should be: scenario, success, exit_code, result, exception.

aranke · 2023-04-12T18:37:01Z

website/docs/reference/programmatic-invocations.md

+2. At least one handled error (e.g. test failure, model build error): `success=False`, exit code `1`, `result` populated (based on the command), `exception` empty.
+3. Unhandled error: `success=False`, exit code `2`, `exception` populated with top-level error, `result` empty because the invocation did not gracefully complete and return results.
+
+### Caveats


Should this be SLA or similar?

As in - our SLA (set of guarantees) to end users, in terms of stability of the interface across future versions?

Yes, correct.

aranke · 2023-04-12T20:53:01Z

website/docs/reference/events-logging.md

@@ -190,4 +190,10 @@ The `EVENT_HISTORY` object has been deprecated and removed in dbt Core v1.4+

 Older versions of `dbt-core` made available a full history of events fired during an invocation, in the form of an `EVENT_HISTORY` object.

-The Python interface into events is significantly less mature than the structured logging interface. For all use cases, we recommend parsing JSON-formatted logs.
+<VersionBlock firstVersion="1.5">


aranke · 2023-04-12T20:56:30Z

website/docs/reference/programmatic-invocations.md


-### Caveats
+| Scenario                                                                                    | CLI Exit Code | `success` | `result`         | `exception` |


Nit: numbers should be right-aligned using the |---:| header.

aranke · 2023-04-12T20:57:17Z

website/docs/reference/programmatic-invocations.md


- The `results` objects returned by each command are liable to change. Some of these objects are already contracted, in part, because they overlap with [dbt artifacts](dbt-artifacts).
+From dbt Core v1.5 onward, we making an ongoing commitment to providing a Python entry point at functional parity with dbt-core's CLI. We reserve the right to change the underlying implementation used to achieve that goal. We expect that the current implementation will unlock real use cases, in the short & medium term, while we work on a set of stable, long-term interfaces that will ultimately replace it.


I like this language, ty.

aranke · 2023-04-12T20:57:51Z

website/docs/reference/programmatic-invocations.md

-Pass in params as `**kwargs`, instead of passing in a list of CLI-style strings. At present, dbt will not do any validation or type coercion on your inputs. The subcommand must be specified, in a list, as the first positional argument.
+### Overriding parameters
+
+Pass in parameters as Pythonic `**kwargs`, instead of a list of CLI-style strings. At present, dbt will not do any validation or type coercion on your inputs. The subcommand must be specified, in a list, as the first positional argument.


Nit: this is not Pythonic style, Python lets you do this as an escape hatch.

Heh, good point. I'm just going to change this to Pass in parameters as keyword arguments

aranke

Mostly LGTM, couple nits.

jtcohen6 · 2023-04-12T22:26:33Z

@aranke Thanks for the re-review! Good nits, quickly fixed

dbeatty10

Most Python examples confirmed to work!

Suggestion with code fix for the lone exception.

website/docs/reference/programmatic-invocations.md

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

website/docs/reference/programmatic-invocations.md

dbeatty10

Love having these example Python scripts!

Draft docs for programmatic invocations

107d517

jtcohen6 added the dbt-core v1.5 Docs impact for the v1.5 release (Apr 2023) label Mar 31, 2023

jtcohen6 requested review from aranke and matthewshaver March 31, 2023 11:41

github-actions bot added content Improvements or additions to content guides Knowledge best suited for Guides size: medium This change will take up to a week to address labels Mar 31, 2023

jtcohen6 mentioned this pull request Apr 4, 2023

Integrate auto-generated CLI/API documentation for dbt-core #2674

Closed

5 tasks

matthewshaver added this to the dbt Core v1.5 Documentation milestone Apr 4, 2023

matthewshaver reviewed Apr 5, 2023

View reviewed changes

website/docs/reference/programmatic-invocations.md Outdated Show resolved Hide resolved

Update website/docs/reference/programmatic-invocations.md

1012a4c

matthewshaver reviewed Apr 5, 2023

View reviewed changes

website/docs/reference/programmatic-invocations.md Outdated Show resolved Hide resolved

Update website/docs/reference/programmatic-invocations.md

95c8647

matthewshaver reviewed Apr 5, 2023

View reviewed changes

website/docs/reference/programmatic-invocations.md Outdated Show resolved Hide resolved

Update website/docs/reference/programmatic-invocations.md

0716f02

aranke reviewed Apr 5, 2023

View reviewed changes

website/docs/reference/programmatic-invocations.md Outdated Show resolved Hide resolved

aranke reviewed Apr 5, 2023

View reviewed changes

website/docs/reference/programmatic-invocations.md Outdated Show resolved Hide resolved

jtcohen6 mentioned this pull request Apr 7, 2023

Handle internal exceptions in postflight dbt-labs/dbt-core#7242

Merged

6 tasks

jtcohen6 changed the title ~~Document programmatic invocation API~~ Document dbtRunner (programmatic invocation) Apr 10, 2023

jtcohen6 mentioned this pull request Apr 10, 2023

[CT-1767] [Feature] dbt parse should return a manifest dbt-labs/dbt-core#6547

Closed

dbeatty10 mentioned this pull request Apr 10, 2023

[CT-2391] [Bug] Following the Python API example.py example raises AttributeError dbt-labs/dbt-core#7312

Closed

2 tasks

dbeatty10 reviewed Apr 10, 2023

View reviewed changes

website/docs/reference/programmatic-invocations.md Outdated Show resolved Hide resolved

Revise, update with examples

c52d0aa

jtcohen6 marked this pull request as ready for review April 11, 2023 11:56

jtcohen6 requested a review from a team as a code owner April 11, 2023 11:56

jtcohen6 mentioned this pull request Apr 11, 2023

Update CLI/API docs example, header dbt-labs/dbt-core#7320

Merged

2 tasks

jtcohen6 self-assigned this Apr 12, 2023

aranke reviewed Apr 12, 2023

View reviewed changes

PR feedback

f200d2e

aranke reviewed Apr 12, 2023

View reviewed changes

aranke previously approved these changes Apr 12, 2023

View reviewed changes

PR feedback

4015c09

jtcohen6 dismissed aranke’s stale review via 4015c09 April 12, 2023 22:26

Merge branch 'current' into jerco/programmatic-invocations-api

0deed83

dbeatty10 reviewed Apr 12, 2023

View reviewed changes

Update website/docs/reference/programmatic-invocations.md

a178c06

Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>

dbeatty10 reviewed Apr 12, 2023

View reviewed changes

website/docs/reference/programmatic-invocations.md Outdated Show resolved Hide resolved

Helpful error message for failed assertions

44107d9

matthewshaver requested review from dbeatty10, matthewshaver and aranke April 13, 2023 15:00

dbeatty10 approved these changes Apr 13, 2023

View reviewed changes

Merge branch 'current' into jerco/programmatic-invocations-api

9ece56d

jtcohen6 merged commit a3b26f3 into current Apr 13, 2023

jtcohen6 deleted the jerco/programmatic-invocations-api branch April 13, 2023 17:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document dbtRunner (programmatic invocation) #3118

Document dbtRunner (programmatic invocation) #3118

jtcohen6 commented Mar 31, 2023 •

edited by dbeatty10

Loading

netlify bot commented Mar 31, 2023 •

edited

Loading

aranke Apr 12, 2023

aranke Apr 12, 2023

aranke Apr 12, 2023

dbeatty10 Apr 12, 2023

jtcohen6 Apr 12, 2023

aranke Apr 12, 2023

aranke Apr 12, 2023

aranke Apr 12, 2023

jtcohen6 Apr 12, 2023

aranke Apr 12, 2023

aranke Apr 12, 2023

aranke Apr 12, 2023

jtcohen6 Apr 12, 2023

aranke Apr 12, 2023

aranke Apr 12, 2023

jtcohen6 Apr 12, 2023

aranke left a comment

jtcohen6 commented Apr 12, 2023

dbeatty10 left a comment

dbeatty10 left a comment


		In v1.5, dbt-core added support for programmatic invocations. The intent of this entry point is provide exact parity with CLI functionality, callable from within a Python script or application.

		The main entry point is a `dbtRunner` class that wraps around `dbt-core`'s CLI, and allows you to "invoke" CLI commands as Python methods. Each command returns a `dbtRunnerResult` object, which has three attributes:


		### `dbtRunnerResult` and exit codes

		There is a 1:1 correspondence between [CLI exit codes](reference/exit-codes) and the `dbtRunnerResult` returned by a programmatic invocation:


		### Caveats
		\| Scenario \| CLI Exit Code \| `success` \| `result` \| `exception` \|


		- The `results` objects returned by each command are liable to change. Some of these objects are already contracted, in part, because they overlap with [dbt artifacts](dbt-artifacts).
		From dbt Core v1.5 onward, we making an ongoing commitment to providing a Python entry point at functional parity with dbt-core's CLI. We reserve the right to change the underlying implementation used to achieve that goal. We expect that the current implementation will unlock real use cases, in the short & medium term, while we work on a set of stable, long-term interfaces that will ultimately replace it.

Document dbtRunner (programmatic invocation) #3118

Document dbtRunner (programmatic invocation) #3118

Conversation

jtcohen6 commented Mar 31, 2023 • edited by dbeatty10 Loading

What are you changing in this pull request and why?

Checklist

netlify bot commented Mar 31, 2023 • edited Loading

✅ Deploy Preview for docs-getdbt-com ready!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aranke left a comment

Choose a reason for hiding this comment

jtcohen6 commented Apr 12, 2023

dbeatty10 left a comment

Choose a reason for hiding this comment

dbeatty10 left a comment

Choose a reason for hiding this comment

jtcohen6 commented Mar 31, 2023 •

edited by dbeatty10

Loading

netlify bot commented Mar 31, 2023 •

edited

Loading