Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix error analysis predictions analyzer failing due to new filtering refactor #1625

Merged
merged 1 commit into from
Aug 11, 2022

Conversation

imatiach-msft
Copy link
Contributor

@imatiach-msft imatiach-msft commented Aug 10, 2022

Description

The simple scenario where only predictions and true labels are passed to error analysis dashboard is failing with the new released 0.3.7 version of erroranalysis package. This is due to the new filtering refactor, where model does not exist for the simple scenario, but we are still trying to access it. Supposedly our notebook tests should have caught this, but it seems because the dashboard python code is run on a separate thread the error doesn't actually lead to a notebook failure. This PR fixes the logic and adds additional unit tests for the both the predictions analyzer and model analyzer cases.

Checklist

  • I have added screenshots above for all UI changes.
  • I have added e2e tests for all UI changes.
  • Documentation was updated if it was needed.

@imatiach-msft imatiach-msft requested a review from gaugup as a code owner August 10, 2022 15:28
@imatiach-msft imatiach-msft force-pushed the ilmat/fix-ea-pred-analyzer branch from 8ca18d0 to a818de8 Compare August 10, 2022 19:59
@codecov-commenter
Copy link

Codecov Report

Merging #1625 (8ca18d0) into main (c5d7319) will decrease coverage by 3.30%.
The diff coverage is 100.00%.

❗ Current head 8ca18d0 differs from pull request most recent head a818de8. Consider uploading reports for the commit a818de8 to get more accurate results

@@            Coverage Diff             @@
##             main    #1625      +/-   ##
==========================================
- Coverage   88.86%   85.55%   -3.31%     
==========================================
  Files         105       58      -47     
  Lines        5449     2943    -2506     
==========================================
- Hits         4842     2518    -2324     
+ Misses        607      425     -182     
Flag Coverage Δ
unittests 85.55% <100.00%> (-3.31%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ranalysis/erroranalysis/_internal/cohort_filter.py 87.58% <100.00%> (+1.18%) ⬆️
erroranalysis/erroranalysis/version.py 100.00% <100.00%> (ø)
...ponsibleai/databalanceanalysis/balance_measures.py
...onsibleai/responsibleai/serialization_utilities.py
responsibleai/responsibleai/__init__.py
.../responsibleai/modelanalysis/constants/__init__.py
...ibleai/_tools/causal/dashboard_schemas/__init__.py
...ponsibleai/responsibleai/_tools/shared/__init__.py
responsibleai/responsibleai/_internal/constants.py
...ai/responsibleai/_tools/counterfactual/__init__.py
... and 42 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@imatiach-msft imatiach-msft enabled auto-merge (squash) August 11, 2022 04:39
@imatiach-msft imatiach-msft disabled auto-merge August 11, 2022 04:39
@imatiach-msft imatiach-msft merged commit 5f273e6 into main Aug 11, 2022
@imatiach-msft imatiach-msft deleted the ilmat/fix-ea-pred-analyzer branch August 11, 2022 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants