[CT-998] [Bug] Provide a more helpful error when a project-level config is Undefined and cannot be deepcopied

[jesperdejby]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

DBT version 1.0.0
Plugins:

• bigquery: 1.0.0

I tried executing dbt run --select my_model with the following syntax in my dbt_project.yml file:

dwh:
  +database: db_name
  dataset:
    +schema: schema_name
    +grant_access_to:
      - project: "{{ 'dwh-prod' if target.name == 'prod' }}"
        dataset: mydataset

The IF statement - project: "{{ 'dwh-prod' if target.name == 'prod' }}" is incomplete and will return the error message:

cannot pickle 'Undefined' object

Expected Behavior

If you want to grant access to this project you could just write: - project: prod. I expect the error message to be more descriptive than:

cannot pickle 'Undefined' object

If you get a hint on that the if-statement can't be parsed or something along these lines you could identify where the problem lies. I managed to reverse engineer this by finding #2110 and concluding this was not caused by my models/refs and I started looking into the dbt_project.yml file.

Steps To Reproduce

1. Be connected to a BigQuery db
2. Replace the following targets in your project.yml files with ones relevant to your case:
   dwh:
   +database: db_name
   dataset:
   +schema: schema_name
   +grant_access_to:
   - project: "{{ 'dwh-prod' if target.name == 'prod' }}"
   dataset: mydataset
3. Create a model that uses this database/schema combination
4. Execute dbt run --select my_model and it should fail because of the incorrect IF-statement

Relevant log output

cannot pickle 'Undefined' object

### Environment

```markdown
- OS: macOS Monterey 12.4
- Python: 3.9.9
- dbt: 1.0.0

Which database adapter are you using with dbt?

bigquery

Additional Context

No response

[jtcohen6]

@jesperdejby Thanks for opening, and for the clear restatement! I was able to reproduce this quite easily.

Indeed, the issue here is that, when target.name is not prod, the Jinja expression resolves to None/Undefined, which Python cannot deepcopy. (dbt needs to deepcopy project-level configs, during config resolution at parse time.)

A temporary fix is to provide an else conditional value:

        - project: "{{ 'dwh-prod' if target.name == 'prod' else 'something_else' }}"

I agree that the error message here is not very helpful in identifying the root cause. I'm going to reframe this issue as: dbt should provide a more helpful error when a project-level config is Undefined and cannot be deepcopied.

[jesperdejby]

@jtcohen6 Thank you for the quick response and help 🙏 .

[emmyoop]

We're going to need to take a closer look to see if there is a reasonable place/way to determine when Undefined is being returned from jinja before this can be estimated.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.