[Feature] Fail during dbt parse when any dependency is not installed via dbt deps

[davidharting]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

dbt parse succeeds when a package specified in packages.yml is not installed

Expected Behavior

dbt parse should fail and instruct me to run dbt deps

Steps To Reproduce

You can watch along with the reproduction in this Zoom clip.

1. Clone https://github.com/dbt-labs/jaffle_shop_duckdb and follow instructions to install dependencies and activate virtual environment
2. Run dbt parse. Notice that it succeeds.
3. Create a packages.yml. Make it look like this:

packages:
  - package: dbt-labs/codegen
    version: ["<1.0.0"]

4. Run dbt deps
5. Update packages.yml to add another package:

packages:
  - package: dbt-labs/codegen
    version: ["<1.0.0"]
  - package: fivetran/salesforce 
    version: [">=1.1.0", "<1.2.0"]

7. Run dbt parse (Do not run dbt deps first)
8. Notice that dbt parse succeeds 🐛 when it should actually fail due to missing fivetran/salesforce

Relevant log output

No response

Environment

- OS: MacOS 14.16.1
- Python: 3.12.5
- dbt: 1.8.1

Which database adapter are you using with dbt?

other (mention it in "Additional Context")

Additional Context

dbt-duckdb 1.8.1

Looks like Feature Request #4557 would be one way of resolving this bug?

[davidharting]

I added the Impact: Exp label to this.

The Cloud IDE has a feature that automatically runs deps for users when needed. However, this feature depends on parse failing with the error message about missing dependencies. When parse comes back clean but a dependency is missing, this feature no does not work.

[gshank]

I don't believe that this is a bug. Parsing processes the project files. If none of them reference a project that's been added to dependencies, then there is no error. This is a feature request. It seems like a reasonable feature request from Cloud's perspective, but what you're asking for is that "parsing" do a kind of mini-deps command.

[davidharting]

@gshank I see what you are saying, but I there is an inconsistency in behavior that lead me to file this as a bug.

In some circumstances, dbt parse does fail with a Compilation Error when missing packages:

$ dbt parse
14:43:53  Running with dbt=1.8.1
14:43:53  Registered adapter: duckdb=1.8.1
14:43:53  Encountered an error:
Compilation Error
  dbt found 2 package(s) specified in packages.yml, but only 0 package(s) installed in dbt_packages. Run "dbt deps" to install package dependencies.

but what you're asking for is that "parsing" do a kind of mini-deps command.

I'm not sure exactly what you mean by a "mini-deps command."

Because of the compilation error I pasted above, I am not expecting parse to install dependencies for me. But I am expecting it to detect when I am missing installed dependencies. Basically, I'm saying that that CompilationError has set that expectation, which is then not always upheld. Thus my take that this is a bug.

[gshank]

The code does a count of packages in packages.yml and compares that against the count of dependencies, which is where that error comes from. The problem is that there is no easy connection between the packages specified in packages.yml and the "dependencies" that are loaded. The actual name of the package comes from the dbt_project.yml file, not from the "name" of the package that's specified. We could store more information about the loaded dependencies for a better comparison, which is probably a good idea, but it's not "fixable" with what we have currently.

[dbeatty10]

Per @gshank's feedback, I'm going to recategorize this as a feature request (we can always switch it back to a bug report if that seems appropriate).

[ChenyuLInx]

An update to this should be a be behind behavior flag