dbt packages.yml conflict: Found duplicate project "dbt_utils"

[ilmari-aalto]

Describe the bug

dbt errors when importing the same package several times due to package dependencies.

Consider the following config in packages.yml:

packages:
  - git: "https://github.com/fishtown-analytics/dbt-utils.git"
    revision: 0.6.2
  - git: "https://github.com/calogica/dbt-expectations.git"
    revision: 0.2.0

Note how we first import dbt-utils directly. Next we import dbt-expectations which in turn installs dbt-date as seen in its package.yml: https://github.com/calogica/dbt-expectations/blob/main/packages.yml#L2. Now dbt-date requires dbt-utils as we can see in the corresponding package.yml: https://github.com/calogica/dbt-date/blob/main/packages.yml#L2.

So we end up importing dbt-utils twice: once explicitly, and a second time indirectly because it's referenced as a dependency in other packages. Unfortunately, this leads to the following error when executing dbt deps: Found duplicate project "dbt_utils". This occurs when a dependency has the same project name as some other dependency.

Steps To Reproduce

Use the package.yml definition from above and run dbt deps. This will lead dbt to error with the message: Found duplicate project "dbt_utils". This occurs when a dependency has the same project name as some other dependency.

Expected behavior

dbt knows that both references point to the same dbt-utils, imports it just once, and is happy.

Screenshots and log output

If applicable, add screenshots or log output to help explain your problem.

System information

Which database are you using dbt with?

• postgres
• [ x] redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

installed version: 0.18.1
   latest version: 0.18.1

Up to date!

Plugins:
  - bigquery: 0.18.1
  - snowflake: 0.18.1
  - redshift: 0.18.1
  - postgres: 0.18.1

The operating system you're using:
macOS Catalina 10.15.7

The output of python --version:

Python 2.7.16

Additional context

Of course we could remove the explicit entry for dbt-utils. But we want to make sure that dbt-utils will always be there, and the package.yml of the second package could change. So we don't want to rely on it to have dbt-utils available.

There could also be two distinct packages, and both of them would depend on dbt-utils (for example). Presumably you'd then get the same error and couldn't have both packages installed at the same time.

[jtcohen6]

Hey @ilmari-aalto, the best solution here is to use the package syntax instead of the git syntax:

packages:
  - package: fishtown-analytics/dbt_utils
    version: 0.6.2
  - package: calogica/dbt_expectations
    version: 0.2.0

This is what we recommend in the docs because it solves exactly the problem you're raising here. It's available for any package hosted at hub.getdbt.com. dbt is then able to figure out that dbt_expectations depends on dbt_utils and, depending on the version (or version boundaries!) you've provided, picks the best candidate versions, or raises an error if it can't find one.

$ dbt deps
Running with dbt=0.18.1
Installing fishtown-analytics/dbt_utils@0.6.2
  Installed from version 0.6.2

Installing calogica/dbt_expectations@0.2.0
  Installed from version 0.2.0

Installing https://github.com/calogica/dbt-date.git@0.2.2
  Installed from revision 0.2.2

Installing fishtown-analytics/spark_utils@0.1.0
  Installed from version 0.1.0

[ilmari-aalto]

Amazing, thanks so much for your answer @jtcohen6! I'm very glad that this had been thought of!

It works with a minor tweak to your example, had to change revision to version:

packages:
  - package: fishtown-analytics/dbt_utils
    version: 0.6.2
  - package: calogica/dbt_expectations
    version: 0.2.0

[jtcohen6]

@ilmari-aalto you're so right, I just edited my comment above to reflect. Thank you!