[Bug] References to vars in dbt_project.yml don't work in source yamls

[ryantimjohn]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Hi there,

I've tried to reference vars with the method mentioned here by @jtcohen6 doesn't seem to be working:
#3450

I defined my variables like so:

vars:
    frakture_facebook : 
      - name: frakture_facebook
        database: bsd-vera
        schema: src_frakture
        tables:
            - name: facebook_bizman_hdx_ad_summary_by_date_pivot
            - name: facebook_bizman_hdx_message

And then tried to reference them like this:

version: 2
sources: "{{ var('frakture_facebook') }}"

But got this error:

Server error: RPC server failed to compile project, call the "status" method for compile status: Compilation Error
The schema file at models/sources/frakture_facebook.yml is invalid because the value of 'sources' is not a list

I know the variable is correctly formatted because when I reference it in a .sql statement like so:
{{ var('frakture_facebook')}}
I get:
[{'name': 'frakture_facebook', 'database': 'bsd-vera', 'schema': 'src_frakture', 'tables': [{'name': 'facebook_bizman_hdx_ad_summary_by_date_pivot'}, {'name': 'facebook_bizman_hdx_message'}]}]

Expected Behavior

The vars should be referenced in the source yml and correctly parse as sources.

Steps To Reproduce

1. Define sources within vars in dbt_profile.yml
2. References the sources in your source yaml file

Relevant log output

Server error: RPC server failed to compile project, call the "status" method for compile status: Compilation Error
The schema file at models/sources/frakture_facebook.yml is invalid because the value of 'sources' is not a list

Environment

- OS: dbtCloud
- Python: n/a
- dbt: 0.21.0

What database are you using dbt with?

bigquery

Additional Context

No response

[jtcohen6]

Hey @ryantimjohn, thanks for the issue!

There's an important distinction between the snippet you have above, and the one I shared in #3450 (comment). There, I was using a var to template just the source's list of tables. In this case, you're using a var to template out the full sources specification. That has tricky implications for our logic around file parsing and validation. One of the very first things dbt does when reading yaml files is validate its structure, at the highest level:

dbt-core/core/dbt/parser/read_files.py

Lines 47 to 67 in 40ae6b6

	# Do some minimal validation of the yaml in a schema file.
	# Check version, that key values are lists and that each element in
	# the lists has a 'name' key
	def validate_yaml(file_path, dct):
	check_format_version(file_path, dct)
	for key in schema_file_keys:
	if key in dct:
	if not isinstance(dct[key], list):
	msg = (f"The schema file at {file_path} is "
	f"invalid because the value of '{key}' is not a list")
	raise CompilationException(msg)
	for element in dct[key]:
	if not isinstance(element, dict):
	msg = (f"The schema file at {file_path} is "
	f"invalid because a list element for '{key}' is not a dictionary")
	raise CompilationException(msg)
	if 'name' not in element:
	msg = (f"The schema file at {file_path} is "
	f"invalid because a list element for '{key}' does not have a "
	"name attribute.")
	raise CompilationException(msg)

The primary goal is to power capabilities around partial parsing: If sources isn't a list right off the bat, before variables are templated, dbt is going to have a hard time storing its internal representation of the .yml file. This is one of the strong assumptions dbt needs to make in order to parse projects in a fast, reliable manner.

The code below does work just fine, though I get it's much less sleek:

version: 2
sources:
  - name: "{{ var('frakture_facebook')[0].name }}"
    database: "{{ var('frakture_facebook')[0].database }}"
    schema: "{{ var('frakture_facebook')[0].schema }}"
    tables: "{{ var('frakture_facebook')[0].tables }}"

I'm not sure we're going to be able to support this use case, without significant changes to some of the lowest-lying code in dbt. I wish I could get you the coolness you're after.