[CT-3227] [Bug] KeyError: 'my_project://models/metrics.yml' after deleting YAML files related to semantic layer

[dbeatty10]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Failure with long stacktrace after deleting YAML files related to semantic models and/or metrics.

Workaround

The easiest workaround is do a dbt clean:

dbt clean

More detailed workaround

Here's a workaround that may work in the meantime:

Use --no-partial-parse to disable partial parsing:

dbt --no-partial-parse compile

Alternatively, delete the partial_parse.msgpack file from your target directory (for example, via dbt clean).

Expected Behavior

Either no failure or at least no stacktrace.

Steps To Reproduce

1. create a bunch of semantic models / metrics that reference each other
2. parse project: dbt parse
3. delete a bunch of them
4. re-parse project: dbt parse
5. get a KeyError

  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/parser/manifest.py", line 659, in parse_project
    block = FileBlock(self.manifest.files[file_id])
KeyError: 'salesforce_opportunity://models/marts/core/metrics.yml'

Relevant log output

05:53:09 Running with dbt=1.6.6
05:53:11 Registered adapter: snowflake=1.6.4
05:53:11 Encountered an error:
'salesforce_opportunity://models/marts/core/metrics.yml'
05:53:11 Traceback (most recent call last):
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/cli/requires.py", line 87, in wrapper
    result, success = func(*args, **kwargs)
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/cli/requires.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/cli/requires.py", line 143, in wrapper
    return func(*args, **kwargs)
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/cli/requires.py", line 172, in wrapper
    return func(*args, **kwargs)
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/cli/requires.py", line 219, in wrapper
    return func(*args, **kwargs)
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/cli/requires.py", line 246, in wrapper
    manifest = ManifestLoader.get_full_manifest(
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/parser/manifest.py", line 316, in get_full_manifest
    manifest = loader.load()
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/parser/manifest.py", line 492, in load
    self.parse_project(
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/parser/manifest.py", line 659, in parse_project
    block = FileBlock(self.manifest.files[file_id])
KeyError: 'salesforce_opportunity://models/marts/core/metrics.yml'
05:53:09 'salesforce_opportunity://models/marts/core/metrics.yml'

Environment

- OS:
- Python:
- dbt: 1.6.6

Which database adapter are you using with dbt?

snowflake

Additional Context

Maybe be related (or a duplicate of):

• [CT-3226] [Bug] Compilation Error 'metric.my_project.distinct_customers' depends on 'semantic_model.my_project.orderves' which is not in the graph! #8859
• [CT-3102] [Bug] semantic models don't support partial parsing #8609

[dbeatty10]

Workaround

The easiest workaround is do a dbt clean:

dbt clean

More detailed workaround

Here's a workaround that may work in the meantime:

Use --no-partial-parse to disable partial parsing:

dbt --no-partial-parse compile

Alternatively, delete the partial_parse.msgpack file from your target directory (for example, via dbt clean).

[dbeatty10]

The key thing to triggering this error was defining a ratio metric, parsing the project, and then deleting the file it appears within:

metrics:
  - name: average_value_per_opportunity
    description: The average value per opportunity
    type: ratio
    label: Average Value per Opportunity
    type_params:
      numerator: total_value
      denominator: total_opportunities

Reprex

👉 This reproducible example assumes dbt-snowflake>=1.7.0-rc1. You'll need to adjust the metricflow_time_spine model in order to use dbt_utils to make it work for dbt ~=1.6.0.

seeds/orders.csv

"order_id","location_id","customer_id","order_total","tax_paid","ordered_at","count_food_items","count_drink_items","count_items","subtotal_drink_items","subtotal_food_items","subtotal","order_cost","location_name","is_food_order","is_drink_order"
"3d12eb16-1bbb-47ad-a732-bf156c322fa6","7f790ed7-0fc4-4de2-a1b0-cce72e657fc4","7cd5e7f3-39c8-48ce-950a-1b70449e7829",6.3600000000000000,0.36000000000000000000,2016-09-01 19:58:00.000,0,1,1,6.0000000000000000,0,6.0000000000000000,0.82000000000000000000,Philadelphia,false,true
"2c54134f-2322-4704-9f82-e2e9b77bcb19","7f790ed7-0fc4-4de2-a1b0-cce72e657fc4","96c8937f-d99c-4515-acec-db24d07768f9",50.8800000000000000,2.8800000000000000,2016-09-01 09:35:00.000,3,2,5,11.0000000000000000,37.0000000000000000,48.0000000000000000,10.32000000000000000000,Philadelphia,true,true

Create models/metricflow_time_spine.sql as described here:

{{ config(materialized='table') }}

with days as (

    {{
        dbt.date_spine(
            'day',
            "to_date('01/01/2000','mm/dd/yyyy')",
            to_date('2027-01-02')
        )
    }}

),

final as (
    select cast(date_day as date) as date_day
    from days
)

select * from final

models/semantic_models.yml

semantic_models:
  - name: orders_2
    model: ref('orders')
    description: |
      Order fact table. This table is at the order grain with one row per order.
    defaults:
      agg_time_dimension: ordered_at
    entities:
      - name: order_id
        type: primary
    dimensions:
      - name: ordered_at
        expr: ordered_at
        type: time
        type_params:
          time_granularity: day
    measures:
      - name: customers_with_orders_2
        description: Distinct count of customers placing orders
        agg: count_distinct
        expr: customer_id
      - name: total_value
        agg: sum
        expr: order_total
      - name: total_opportunities
        agg: count_distinct
        expr: customer_id

models/metrics.yml

metrics:
  - name: average_value_per_opportunity
    description: The average value per opportunities
    type: ratio
    label: Average Value per Opportunity
    type_params:
      numerator: total_value
      denominator: total_opportunities

  - name: total_value
    description: The total opportunity value for the business
    type: simple
    label: Total Value
    type_params:
      measure: total_value

  - name: total_opportunities
    description: The total opportunities for the business
    type: simple
    label: Total Opportunities
    type_params:
      measure: total_opportunities

At the end, your project folder will look like this:

├── dbt_project.yml
├── models
│   ├── metricflow_time_spine.sql
│   ├── metrics.yml
│   └── semantic_models.yml
└── seeds
    └── orders.csv

Run these set of commands to see the error:

dbt clean
dbt parse
mv models/metrics.yml models/metrics.yml.x
dbt parse

Here's the error I got:

15:27:46  Encountered an error:
'my_project://models/metrics.yml'
...
  File "/venv/snowflake_1.7/lib/python3.10/site-packages/dbt/parser/manifest.py", line 663, in parse_project
    block = FileBlock(self.manifest.files[file_id])
KeyError: 'my_project://models/metrics.yml'

Then run this to make the problem "go away":

dbt --no-partial-parse parse

[graciegoheen]

@dbeatty10 Is this only affecting 1.6 and 1.7?

[dbeatty10]

@dbeatty10 Is this only affecting 1.6 and 1.7?

I didn't check 1.5 since the reprex was using semantic_models that first appeared in 1.6.

@graciegoheen Do you think we should try out a similar non-MetricFlow ratio metric with 1.5 also?

[graciegoheen]

Got it - let's contain this issue to semantic_models. I'll go ahead and add the 1.6 and 1.7 backport labels

[QMalcolm]

We had hoped this was resolved by some related work we did. However I just tried the reproduction with 1.7.latest and the error is still occurring.

[QMalcolm]

The error specifically happens when a yaml file which contains definitions for both semantic models and metrics is deleted

[QMalcolm]

But also it only does it sometimes...