Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warn error options invalid when warn or error set to none #10453

Merged
merged 6 commits into from
Jul 17, 2024

Conversation

gshank
Copy link
Contributor

@gshank gshank commented Jul 16, 2024

resolves #10452

Problem

If a field in warn_error_options was set to None, a valid WarnErrorOptions object was not constructed.

Solution

Fix the function to normalize the warn_error_options dictionary.

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
  • This PR includes type annotations for new and modified functions

@gshank gshank requested a review from a team as a code owner July 16, 2024 21:18
@cla-bot cla-bot bot added the cla:yes label Jul 16, 2024
Copy link

codecov bot commented Jul 16, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.79%. Comparing base (4c7d922) to head (0baf0c8).
Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10453      +/-   ##
==========================================
+ Coverage   88.76%   88.79%   +0.03%     
==========================================
  Files         180      180              
  Lines       22564    22562       -2     
==========================================
+ Hits        20029    20035       +6     
+ Misses       2535     2527       -8     
Flag Coverage Δ
integration 86.10% <100.00%> (+0.06%) ⬆️
unit 62.13% <100.00%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
Unit Tests 62.13% <100.00%> (+0.02%) ⬆️
Integration Tests 86.10% <100.00%> (+0.06%) ⬆️

Copy link
Contributor

@QMalcolm QMalcolm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First, thank you for doing this work ❤️ This is well done 🙂 However, I worry the changes as is if merged would remove a generalized utility that we'll likely continually need.

Specifically, exclusive_primary_alt_value_setting was created specifically to handle general name changes gracefully. Although it is only being used for exclude/warn and include/error at this time, it is not unlikely that there will be other keys we want to rename in the future. As such, altering the function to only work with warn_error_options seems like a step back.

I think there are two simpler changesets to fix the problem we are trying to address: extending exclusive_primary_alt_value_setting to default the associated value OR create an orchestrating function.

Alt (1): extending exclusive_primary_alt_value_setting

First the function would take an additional optional argument making the function signature look like

def exclusive_primary_alt_value_setting(
    dictionary: Optional[Dict[str, Any]],
    primary: str,
    alt: str,
    parent_config: Optional[str] = None,
    default_value: Optional[Any] = None,
) -> None:

Then at the end of the function we'd conditionally set the default via

if dictionary.get(primary) is None:
    dictionary[primary] = default_value

Alt (2): Orchestrating Function

If we didn't want to modify exclusive_primary_alt_value_setting at all, then we could put the defaulting logic into an orchestrating function specific for warn_error_options munging. Somthing like

def normalize_warn_error_options(dictionary: Dict[str, Any]) -> None:
    exclusive_primary_alt_value_setting(
        warn_error_options, "include", "error", "warn_error_options"
    )
    exclusive_primary_alt_value_setting(
        warn_error_options, "exclude", "warn", "warn_error_options"
    )
    if dictionary.get("include") is None:
        dictionary["include"] = []
    if dictionary.get("exclude") is None:
        dictionary["exclude"] = []

@gshank
Copy link
Contributor Author

gshank commented Jul 17, 2024

I see that you intended "exclusive_primary_alt_value_setting" as a general purpose utility, but it looks like premature generalization to me. What I've found is that when/if you actually get to some other use case that falls kind of in the same category, it will have other constraints or conditions and the intended "general" utility won't actually work at all without complicating changes.

I think it makes more sense to derive a general use case utility when there is more than one use case.

@QMalcolm
Copy link
Contributor

I see that you intended "exclusive_primary_alt_value_setting" as a general purpose utility, but it looks like premature generalization to me. What I've found is that when/if you actually get to some other use case that falls kind of in the same category, it will have other constraints or conditions and the intended "general" utility won't actually work at all without complicating changes.

I think it makes more sense to derive a general use case utility when there is more than one use case.

I'm not sure that I agree that it's a pre-mature generalization, but I'm absolutely open to discussing it further. Can you talk more about your concern that makes the existing function pre-mature? In my mind, there is more than one use case. The existing function addresses the use case of "A key in a dictionary is being renamed to another, but we want to support both as inputs". It is being used for two keys. It renames "error" to "include" and "warn" to "exclude".

I do agree that there could be more constraints. The issue that this PR aims to address is one of those, the ability to have a default value association. This could simply be addressed as outlined above, extending the existing function or external to it. There are additional constraints that could be imagined, such as supporting multiple "alt" key names for a given "primary". I don't recommend we do work to handle multiple "alts" as we don't have an existing or planned use case where that is necessary.

@QMalcolm
Copy link
Contributor

QMalcolm commented Jul 17, 2024

From your comment and my comments, my understanding is that we view the underlying discrete cases differently. I also think both your view and my view are valid. The work I did on renaming dictionaries keys in dictionaries has more than one use case. However, we need to default all the values for keys in warn_error_options to an empty list if the value is None. I think this could be included or excluded as part of the exclusive_primary_alt_value_setting, and which way it goes isn't too important in my mind. I think for the sake of defaulting all three values in the same manner though, that we should do it externally. A compromise solution being something like

def normalize_warn_error_options(
    dictionary: Dict[str, Any],
) -> None:
    convert = {
        "error": "include",
        "warn": "exclude",
        "silence": None,
    }

    for key, alt in convert.items():
        if alt is not None:
            exclusive_primary_alt_value_setting(dictionary, key, alt, "warn_error_options")
        
        if dictionary.get(key) is None:
            dictionary[key] = []

@gshank gshank requested a review from QMalcolm July 17, 2024 18:57
Copy link
Contributor

@QMalcolm QMalcolm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@QMalcolm QMalcolm merged commit 33161a3 into main Jul 17, 2024
66 checks passed
@QMalcolm QMalcolm deleted the warn_error_options_invalid_when_none branch July 17, 2024 21:13
Copy link
Contributor

The backport to 1.8.latest failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.8.latest 1.8.latest
# Navigate to the new working tree
cd .worktrees/backport-1.8.latest
# Create a new branch
git switch --create backport-10453-to-1.8.latest
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 33161a3035d965e43642dbcafa97b9ecc95ce384
# Push it to GitHub
git push --set-upstream origin backport-10453-to-1.8.latest
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.8.latest

Then, create a pull request where the base branch is 1.8.latest and the compare/head branch is backport-10453-to-1.8.latest.

@QMalcolm QMalcolm restored the warn_error_options_invalid_when_none branch July 17, 2024 21:15
QMalcolm pushed a commit that referenced this pull request Jul 17, 2024
…10453)

* Fix exclusive_primary_alt_value_setting to set warn_error_options correctly

* Add test

* Changie

* Fix unit test

* Replace conversion method

* Refactor normalize_warn_error_options
QMalcolm added a commit that referenced this pull request Jul 18, 2024
…10453) (#10462)

* Fix exclusive_primary_alt_value_setting to set warn_error_options correctly

* Add test

* Changie

* Fix unit test

* Replace conversion method

* Refactor normalize_warn_error_options

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

warn_error_options with empty warn or error fields does not use silence
2 participants