[CT-2584] [Feature] Allow users to define packages in dependencies.yml

[joellabes]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

I just watched Sung's speedrun of cross-project refs, and was introduced to dependencies.yml for the first time. I get how we got there, but I think of dependencies as a package-specific thing (cf npm's dependencies and devDependencies in package.json), and I wonder if people coming in cold will be confused about the difference between dependencies.yml and packages.yml.

Describe alternatives you've considered

Easy and covers most of the problem: rename dependencies.yml

Every other part of this process' artifacts talks about publication, so maybe something along the lines of publication_imports.yml or publication_discovery.yml.

Probably better, but a lot more lifting: unify packages.yml and dependencies.yml

Fundamentally, these are just two different ways of extending your project:

#tbc.yml
packages: 
  - package: dbt-labs/dbt_utils
      version: [">=1.0.0", "<2.0.0"]

published_projects: #don't know what to call this
  - project: jaffle_shop

Ideally if we did this, the file wouldn't be called packages.yml anymore - at this point it probably does deserve the dependencies.yml name! I don't think the rename is mandatory, but would be tidier for the long term - we'd just have to do a bit more checking that there's not both versions of the file in play for one project. `

Who will this benefit?

People coming to multi-project or packages for the first time, with assumptions about what content goes in what files

Are you interested in contributing this feature?

Nah, but I will applaud loudly from the sidelines

Anything else?

No response

[jtcohen6]

@joellabes Love it when we have the exact same idea!

We weren't planning for this to be in the initial scope of v1.6, but the idea would very much be to consolidate both package and project dependencies into a single file named dependencies.yml, and have it look like:

# dependencies.yml
packages: 
  - package: dbt-labs/dbt_utils
      version: [">=1.0.0", "<2.0.0"]

projects: # or something else?
  - project: jaffle_shop

In the event that a project is included in both spots — my thinking is, that would enable model access enforcement while resolving refs across projects. I wrote a bit about this here: https://docs.getdbt.com/docs/collaborate/govern/model-access#what-about-referencing-models-from-a-package

[jtcohen6]

Outcomes:

• We do not allow users to define projects (new thing) in packages.yml (old file name)
• We should allow users to define packages (old thing) in dependencies.yml (new file name)
• If users have a packages key in both packages.yml and dependencies.yml — we raise an error
• We do not support any Jinja rendering in dependencies.yml

[emmyoop]

Relates to #7736

[jtcohen6]

This was resolved by #7372. However, in local testing I found that Jinja-in-yaml conditional rendering is supported in the new dependencies.yml file. I'll open a new issue to track removing / disabling that capability.

[joellabes]

@jtcohen6 I've just created dbt-labs/hubcap#275 to ensure that all the dependencies still get captured by the Hub. idk if resolving this is a core eng job or something that a friendly neighbourhood Doug can knock out, but I know it's not something I can sort out myself 🙀

[jtcohen6]

@joellabes Good thinking!

• Check both packages.yml + dependencies.yml for packages deps hubcap#276