Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dbt clone command #3742

Merged
merged 19 commits into from
Jul 18, 2023
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion website/dbt-versions.js
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@ exports.versions = [
]

exports.versionedPages = [
{
"page": "reference/commands/clone",
"firstVersion": "1.6",
},
{
"page": "docs/collaborate/govern/project-dependencies",
"firstVersion": "1.6",
Expand All @@ -51,7 +55,7 @@ exports.versionedPages = [
"page": "docs/collaborate/govern/model-contracts",
"firstVersion": "1.5",
},
{
{
"page": "reference/commands/show",
"firstVersion": "1.5",
},
Expand Down
30 changes: 30 additions & 0 deletions website/docs/reference/commands/clone.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
---
title: "About dbt clone command"
sidebar_label: "clone"
id: "clone"
---

The `dbt clone` command clones selected nodes from the [specified state](/reference/node-selection/syntax#establishing-state) to the target schema(s). This command makes use of the `clone` materialization:
- If your data platform supports zero-copy cloning of tables, and this model exists as a table in the other environment, dbt will create it in your environment as a clone
- Otherwise, dbt will create a simple pointer view (`select * from` the other object)
- By default, `dbt clone` will not recreate pre-existing relations in the current target. To override this, use the `--full-refresh` flag.
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
- You may want to specify a higher number of [threads](/docs/running-a-dbt-project/using-threads) to decrease execution time since individual clone statements are independent of one another.

The `clone` command is useful for:
- blue/green continuous deployment
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it be helpful to link to this so there's context? is that the right link?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, good call! that's a fairly old post at this point, but until we have a newer one, it's good to provide some definition/context

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we feel about this being a snowflake-specific article? I think blue/green continuous blue/green deployment will be different depending on the warehouse being used (zero copy cloning vs. pointer views)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even better point. Blue/green is really only enabled by table clones. There's also hackery cleverness in that post around overriding ref so that when the schemas get swapped, the views stay pointed to the same database.

Maybe better to skip it for now, and just say something like:

  • blue/green continuous deployment (on data warehouses that support cloning tables)

Or remove the bullet entirely, until we have a more up-to-date and thorough dev blog article that we could link to

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just updated - let me know what you think

- cloning current production state into development schema(s)
- handling incremental models in Slim CI dbt Cloud jobs

```bash
# clone all of my models from specified state to my target schema(s)
dbt clone --state path/to/artifacts

# clone one_specific_model of my models from specified state to my target schema(s)
dbt clone --select one_specific_model --state path/to/artifacts

# clone all of my models from specified state to my target schema(s) and recreate all pre-exisiting relations in the current target
dbt clone --state path/to/artifacts --full-refresh

# clone all of my models from specified state to my target schema(s), running up to 50 clone statements in parallel
dbt clone --state path/to/artifacts --threads 50
```
2 changes: 2 additions & 0 deletions website/docs/reference/dbt-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Select the tabs that are relevant to the your development workflow. For example,
Use the following dbt commands in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) and use the `dbt` prefix. For example, to run the `test` command, type `dbt test`.

- [build](/reference/commands/build): build and test all selected resources (models, seeds, snapshots, tests)
- [clone](/reference/commands/clone): clone selected nodes from specified state (requires dbt 1.6 or higher)
- [compile](/reference/commands/compile): compiles (but does not run) the models in a project
- [deps](/reference/commands/deps): downloads dependencies for a project
- [docs](/reference/commands/cmd-docs) : generates documentation for a project
Expand All @@ -40,6 +41,7 @@ Use the following dbt commands in the [CLI](/docs/core/about-the-cli) and use th

- [build](/reference/commands/build): build and test all selected resources (models, seeds, snapshots, tests)
- [clean](/reference/commands/clean): deletes artifacts present in the dbt project
- [clone](/reference/commands/clone): clone selected models from specified state (requires dbt 1.6 or higher)
- [compile](/reference/commands/compile): compiles (but does not run) the models in a project
- [debug](/reference/commands/debug): debugs dbt connections and projects
- [deps](/reference/commands/deps): downloads dependencies for a project
Expand Down
3 changes: 2 additions & 1 deletion website/docs/reference/node-selection/syntax.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,9 @@ dbt can leverage artifacts from a prior invocation as long as their file path is
- [The `state:` selector](/reference/node-selection/methods#the-state-method), whereby dbt can identify resources that are new or modified
by comparing code in the current project against the state manifest.
- [Deferring](/reference/node-selection/defer) to another environment, whereby dbt can identify upstream, unselected resources that don't exist in your current environment and instead "defer" their references to the environment provided by the state manifest.
- The [`dbt clone` command](/reference/commands/clone), whereby dbt can clone nodes based on their location in the manifest provided to the `--state` flag.

Together, these two features enable ["slim CI"](/guides/legacy/best-practices#run-only-modified-models-to-test-changes-slim-ci). We expect to add more features in future releases that can leverage artifacts passed to the `--state` flag.
Together, the `state:` selector and deferral enable ["slim CI"](/guides/legacy/best-practices#run-only-modified-models-to-test-changes-slim-ci). We expect to add more features in future releases that can leverage artifacts passed to the `--state` flag.
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved

### Establishing state

Expand Down
1 change: 1 addition & 0 deletions website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -735,6 +735,7 @@ const sidebarSettings = {
items: [
"reference/commands/build",
"reference/commands/clean",
"reference/commands/clone",
"reference/commands/cmd-docs",
"reference/commands/compile",
"reference/commands/debug",
Expand Down