Skip to content

Commit

Permalink
Options for retaining previous behavior of dbt deps for v1.7+ (#5659)
Browse files Browse the repository at this point in the history
### Previews
- [Upgrading to v1.7: Behavior changes](
https://docs-getdbt-com-git-dbeatty10-patch-1-dbt-labs.vercel.app/docs/dbt-versions/core-upgrade/upgrading-to-v1.7#behavior-changes)
- [Advanced package
configuration](https://docs-getdbt-com-git-dbeatty10-patch-1-dbt-labs.vercel.app/docs/build/packages#advanced-package-configuration)
- [Predictable package
installs](https://docs-getdbt-com-git-dbeatty10-patch-1-dbt-labs.vercel.app/reference/commands/deps#forcing-upgrades)

## What are you changing in this pull request and why?
Resolves #5140

### Problem

The default behavior of `dbt deps` changed in v1.7+, which can be
surprising to long-time users of `dbt deps`.

### Solution

Document this in both the [migration guide for
v1.7](https://docs.getdbt.com/docs/dbt-versions/core-upgrade/upgrading-to-v1.7)
and anywhere else it is applicable (like
[here](https://docs.getdbt.com/docs/build/packages) and/or
[here](https://docs.getdbt.com/reference/commands/deps)).

## Checklist
- [x] Review the [Content style
guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md)
so my content adheres to these guidelines.
- [ ] For [docs
versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#about-versioning),
review how to [version a whole
page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version)
and [version a block of
content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content).

---------

Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com>
  • Loading branch information
dbeatty10 and mirnawong1 authored Jun 18, 2024
1 parent 89c76ef commit e2b532a
Show file tree
Hide file tree
Showing 3 changed files with 52 additions and 8 deletions.
43 changes: 39 additions & 4 deletions website/docs/docs/build/packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,12 @@ packages:
version: [">=0.7.0", "<0.8.0"]
```

<VersionBlock firstVersion="1.7">

Beginning in v1.7, `dbt deps` "pins" each package by default. See ["Pinning packages"](#pinning-packages) for details.

</VersionBlock>

Where possible, we recommend installing packages via dbt Hub, since this allows dbt to handle duplicate dependencies. This is helpful in situations such as:
* Your project uses both the dbt-utils and Snowplow packages; and the Snowplow package _also_ uses the dbt-utils package.
* Your project uses both the Snowplow and Stripe packages, both of which use the dbt-utils package.
Expand Down Expand Up @@ -137,13 +143,17 @@ packages:
revision: 4e28d6da126e2940d17f697de783a717f2503188
```

We **strongly recommend** "pinning" your package to a specific release by specifying a release name.
<VersionBlock lastVersion="1.6">

If you do not provide a revision, or if you use `master`, then any updates to the package will be incorporated into your project the next time you run `dbt deps`. While we generally try to avoid making breaking changes to these packages, they are sometimes unavoidable. Pinning a package revision helps prevent your code from changing without your explicit approval.
We **strongly recommend** ["pinning" your packages](#pinning-packages) to a specific release by specifying a release name.

To find the latest release for a package, navigate to the `Releases` tab in the relevant GitHub repository. For example, you can find all of the releases for the dbt-utils package [here](https://github.com/dbt-labs/dbt-utils/releases).
</VersionBlock>

<VersionBlock firstVersion="1.7">

As of v0.14.0, dbt will warn you if you install a package using the `git` syntax without specifying a version (see below).
By default, `dbt deps` "pins" each package. See ["Pinning packages"](#pinning-packages) for details.

</VersionBlock>

### Internally hosted tarball URL

Expand Down Expand Up @@ -304,6 +314,31 @@ When you remove a package from your `packages.yml` file, it isn't automatically
* delete the package directory in `dbt_packages/`; or
* run `dbt clean` to delete _all_ packages (and any compiled models), followed by `dbt deps`.

### Pinning packages

<VersionBlock lastVersion="1.6">

We **strongly recommend** "pinning" your package to a specific release by specifying a tagged release name or a specific commit hash.

If you do not provide a revision, or if you use the main branch, then any updates to the package will be incorporated into your project the next time you run `dbt deps`. While we generally try to avoid making breaking changes to these packages, they are sometimes unavoidable. Pinning a package revision helps prevent your code from changing without your explicit approval.

To find the latest release for a package, navigate to the `Releases` tab in the relevant GitHub repository. For example, you can find all of the releases for the dbt-utils package [here](https://github.com/dbt-labs/dbt-utils/releases).

</VersionBlock>

<VersionBlock firstVersion="1.7">

Beginning with v1.7, running [`dbt deps`](/reference/commands/deps) "pins" each package by creating or updating the `package-lock.yml` file in the _project_root_ where `packages.yml` is recorded.

- The `package-lock.yml` file contains a record of all packages installed.
- If subsequent `dbt deps` runs contain no changes to `dependencies.yml` or `packages.yml`, dbt-core installs from `package-lock.yml`.

For example, if you use a branch name, the `package-lock.yml` file pins to the head commit. If you use a version range, it pins to the latest release. In either case, subsequent commits or versions will **not** be installed. To get new commits or versions, run `dbt deps --upgrade` or add `package-lock.yml` to your .gitignore file.

</VersionBlock>

As of v0.14.0, dbt will warn you if you install a package using the `git` syntax without specifying a revision (see below).

### Configuring packages
You can configure the models and seeds in a package from the `dbt_project.yml` file, like so:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,14 @@ As part of this change, the `loaded_at_field` is no longer required to generate
- If a `loaded_at_field` is _not_ provided, dbt will calculate freshness via warehouse metadata tables when possible (new behavior).

This is a relatively small behavior change, but worth calling out in case you notice that dbt is calculating freshness for _more_ sources than before. To exclude a source from freshness calculations, you have two options:
- Don't add a `freshness:` block.
- Explicitly set `freshness: null`
1. Don't add a `freshness:` block.
2. Explicitly set `freshness: null`

Beginning with v1.7, running [`dbt deps`](/reference/commands/deps) creates or updates the `package-lock.yml` file in the _project_root_ where `packages.yml` is recorded. The `package-lock.yml` file contains a record of all packages installed and, if subsequent `dbt deps` runs contain no updated packages in `dependencies.yml` or `packages.yml`, dbt-core installs from `package-lock.yml`.
Beginning with v1.7, running [`dbt deps`](/reference/commands/deps) creates or updates the `package-lock.yml` file in the _project_root_ where `packages.yml` is recorded. The `package-lock.yml` file contains a record of all packages installed and, if subsequent `dbt deps` runs contain no updated packages in `dependencies.yml` or `packages.yml`, dbt-core installs from `package-lock.yml`.

To retain the behavior prior to v1.7, there are two main options:
1. Use `dbt deps --upgrade` everywhere `dbt deps` was used previously.
2. Add `package-lock.yml` to your `.gitignore` file.

## New and changed features and functionality

Expand Down
8 changes: 7 additions & 1 deletion website/docs/reference/commands/deps.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,13 @@ The `package-lock.yml` file should be committed in Git initially, and then updat

The `package-lock.yml` file includes a `sha1_hash` of the `packages` config. This enables dbt to detect if the `packages` config has been updated, and to rerun dependency resolution. To only check for changes to the `packages` config and update the lock file accordingly without installing those packages, provide the `--lock` flag (that is, `dbt deps --lock`).

It's possible to force package resolution to rerun, even if the `packages` config hasn't changed, by running `dbt deps --upgrade`. This enables you to get the latest commits from the `main` branch of an internally maintained `git` package while accepting the risk of unpredictable builds. An alternative to running `dbt deps --upgrade` in production is to "ignore" the lock file by adding `package-lock.yml` to your project's `.gitignore` file. If you pursue either approach, dbt Labs strongly recommends adding version pins for third-party packages within your `packages` config.
### Forcing upgrades

It's possible to force package resolution to rerun, even if the `packages` config hasn't changed, by running `dbt deps --upgrade`. This enables you to get the latest commits from the `main` branch of an internally maintained `git` package while accepting the risk of unpredictable builds.

An alternative to running `dbt deps --upgrade` in production is to "ignore" the lock file by adding `package-lock.yml` to your project's `.gitignore` file.

If you pursue either approach, dbt Labs strongly recommends adding version pins for third-party packages within your `packages` config.

## Add specific packages

Expand Down

0 comments on commit e2b532a

Please sign in to comment.