Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[state:modified] store unrendered_database and unrendered_schema on source definition for state:modified comparisons #10675

Merged
merged 42 commits into from
Sep 30, 2024
Merged
Changes from 1 commit
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
1659c7b
persist schema yml unrendered_config before rendering
MichelleArk Jul 26, 2024
8ad3c03
persist unrendered config on schema file instead of manifest
MichelleArk Jul 26, 2024
4f96f5c
update test_configs_in_schema_files.py
MichelleArk Jul 26, 2024
acd23b0
support versioning in SchemaSourceFile.add_unrendered_config
MichelleArk Jul 26, 2024
9e10428
Merge branch 'main' into state-modified-source-tests
MichelleArk Aug 12, 2024
fe9b572
fix: do not use None for version key in unrendered_configs
MichelleArk Aug 12, 2024
f368cdf
support reliable unrendered_config in .sql files
MichelleArk Aug 12, 2024
57db0bf
more reliably set unrendered_config_call_dict + prefer config_call_di…
MichelleArk Aug 19, 2024
1ce11ec
store + reset unrendered_config_call_dict on node
MichelleArk Aug 20, 2024
bad5723
Merge branch 'main' into state-modified-source-tests
MichelleArk Aug 22, 2024
ce2b150
Merge branch 'main' into state-modified-source-tests
MichelleArk Sep 4, 2024
bad6b92
first pass: put behind legacy flag
MichelleArk Sep 4, 2024
d8bf0e6
fix SnapshotParserTest unit test
MichelleArk Sep 5, 2024
fad1295
Return early to avoid creating jinja environemt if no config call in …
MichelleArk Sep 6, 2024
47b18b1
test var sensitivity with require_config_jinja_insensitivity_for_stat…
MichelleArk Sep 6, 2024
d625a23
cleanup comment
MichelleArk Sep 6, 2024
7d979be
store unrendered database on source definition
MichelleArk Sep 6, 2024
33e2359
Merge branch 'main' into state-modified-source-tests
MichelleArk Sep 10, 2024
4081627
rename behaviour flag
MichelleArk Sep 10, 2024
897c2e5
Merge branch 'main' into state-modified-source-tests
MichelleArk Sep 25, 2024
ac42ec4
changelog entry
MichelleArk Sep 25, 2024
71bda2a
fix state:modified for jinja if statements in configs
MichelleArk Sep 25, 2024
8677a9a
simplify constructing unrendered_config from jinja
MichelleArk Sep 25, 2024
9adb4b8
update unit tests for unrendered_config_call_dict values
MichelleArk Sep 25, 2024
8250705
Merge branch 'state-modified-source-tests' into state-modified-source…
MichelleArk Sep 25, 2024
428df36
clean up debugging print
MichelleArk Sep 25, 2024
40f1a67
put static parsing behind behaviour change flag
MichelleArk Sep 25, 2024
7986726
tests reflect behaviour flag setting
MichelleArk Sep 26, 2024
f4a5c0b
use resource_types_to_schema_file_keys to get patch unrendered configs
MichelleArk Sep 26, 2024
259d6f8
handle versioned unrendered_configs during partial parsing
MichelleArk Sep 26, 2024
1247c24
Merge branch 'state-modified-source-tests' into state-modified-source…
MichelleArk Sep 26, 2024
6f53845
fix unit tests
MichelleArk Sep 26, 2024
42d4bba
add unrendered_schema to source definition
MichelleArk Sep 26, 2024
996c46d
changelog entry
MichelleArk Sep 26, 2024
3aede61
refactor unrendered_schema and unrendered_database methods on schema …
MichelleArk Sep 26, 2024
e924bdc
Merge branch 'main' into state-modified-source-unrendered-database
MichelleArk Sep 26, 2024
1213867
refactor sources tests + pass unrendered properties to SourceDefinition
MichelleArk Sep 26, 2024
7a4892f
fix functional tests
MichelleArk Sep 30, 2024
be86bef
linting
MichelleArk Sep 30, 2024
c18e9ff
one more test
MichelleArk Sep 30, 2024
70327e7
Merge branch 'main' into state-modified-source-unrendered-database
MichelleArk Sep 30, 2024
1261b29
Merge branch 'main' into state-modified-source-unrendered-database
MichelleArk Sep 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
store + reset unrendered_config_call_dict on node
MichelleArk committed Aug 20, 2024
commit 1ce11ec11e203dfeaee7afb600d608612e74baf4
3 changes: 3 additions & 0 deletions core/dbt/artifacts/resources/v1/components.py
Original file line number Diff line number Diff line change
@@ -194,13 +194,16 @@ class ParsedResource(ParsedResourceMandatory):
unrendered_config: Dict[str, Any] = field(default_factory=dict)
created_at: float = field(default_factory=lambda: time.time())
config_call_dict: Dict[str, Any] = field(default_factory=dict)
unrendered_config_call_dict: Dict[str, Any] = field(default_factory=dict)
relation_name: Optional[str] = None
raw_code: str = ""

def __post_serialize__(self, dct: Dict, context: Optional[Dict] = None):
dct = super().__post_serialize__(dct, context)
if context and context.get("artifact") and "config_call_dict" in dct:
del dct["config_call_dict"]
if context and context.get("artifact") and "unrendered_config_call_dict" in dct:
del dct["unrendered_config_call_dict"]
return dct


1 change: 1 addition & 0 deletions core/dbt/parser/base.py
Original file line number Diff line number Diff line change
@@ -390,6 +390,7 @@ def update_parsed_node_config(
)

parsed_node.config_call_dict = config._config_call_dict
parsed_node.unrendered_config_call_dict = config._unrendered_config_call_dict

# do this once before we parse the node database/schema/alias, so
# parsed_node.config is what it would be if they did nothing
1 change: 1 addition & 0 deletions core/dbt/parser/models.py
Original file line number Diff line number Diff line change
@@ -301,6 +301,7 @@ def render_update(self, node: ModelNode, config: ContextConfig) -> None:
statically_parsed = self.run_experimental_parser(node)
# run the stable static parser unless it is explicitly turned off
else:
# TODO: consider running new parser here?
statically_parsed = self.run_static_parser(node)

# if the static parser succeeded, extract some data in easy-to-compare formats
2 changes: 1 addition & 1 deletion core/dbt/parser/schemas.py
Original file line number Diff line number Diff line change
@@ -634,7 +634,7 @@ def patch_node_config(self, node, patch) -> None:
)
# We need to re-apply the config_call_dict after the patch config
config._config_call_dict = node.config_call_dict
config._unrendered_config_call_dict = node.unrendered_config
config._unrendered_config_call_dict = node.unrendered_config_call_dict
self.schema_parser.update_parsed_node_config(
node,
config,

Unchanged files with check annotations Beta

global _TESTING_MACRO_CACHE
if test_caching_enabled() and _TESTING_MACRO_CACHE and string in _TESTING_MACRO_CACHE:
parsed = _TESTING_MACRO_CACHE.get(string, None)
func_calls = getattr(parsed, "_dbt_cached_calls")

Check warning on line 215 in core/dbt/clients/jinja_static.py

Codecov / codecov/patch

core/dbt/clients/jinja_static.py#L214-L215

Added lines #L214 - L215 were not covered by tests
else:
parsed = env.parse(string)
func_calls = tuple(parsed.find_all(jinja2.nodes.Call))
unrendered_kwargs = {}
for call_kwarg in kwarg.value.kwargs:
kwarg_value = _construct_static_kwarg_value(call_kwarg)

Check warning on line 265 in core/dbt/clients/jinja_static.py

Codecov / codecov/patch

core/dbt/clients/jinja_static.py#L265

Added line #L265 was not covered by tests
unrendered_kwargs[call_kwarg.key] = kwarg_value

Check warning on line 267 in core/dbt/clients/jinja_static.py

Codecov / codecov/patch

core/dbt/clients/jinja_static.py#L267

Added line #L267 was not covered by tests
formatted_unrendered_kwargs = [
f"{key}={value}" for key, value in unrendered_kwargs.items()
if yaml_key not in self.unrendered_configs:
return None
if versioned_name not in self.unrendered_configs[yaml_key]:
return None

Check warning on line 336 in core/dbt/contracts/files.py

Codecov / codecov/patch

core/dbt/contracts/files.py#L336

Added line #L336 was not covered by tests
return self.unrendered_configs[yaml_key][versioned_name]