test: Update test_plot_results_no_axis baseline image #2009
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* matplotlib v3.6.0 results in a slightly different baseline image than matplotlib v3.5.x.
matthewfeickert
added
tests
|
Henry has warned us before about checking the actual output of plots, but I'm not sure how to properly test visualization related functions otherwise. I'm open to suggestions though. @alexander-held @andrzejnovak you might be seeing this issue as well for |
Codecov ReportBase: 98.28% // Head: 98.28% // No change to project coverage 👍
Additional details and impacted files@@ Coverage Diff @@
## master #2009 +/- ##
=======================================
Coverage 98.28% 98.28%
=======================================
Files 68 68
Lines 4479 4479
Branches 730 730
=======================================
Hits 4402 4402
Misses 45 45
Partials 32 32
Flags with carried forward coverage won't be shown. Click here to find out more. Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
|
Ah things get more complicated as |
matthewfeickert
commented
Sep 16, 2022
| @@ -67,6 +68,10 @@ def test_plot_results(datadir): | |||
|
|
|||
|
|
|||
| @pytest.mark.mpl_image_compare | |||
| @pytest.mark.xfail( | |||
There was a problem hiding this comment.
I'm going with xfail over skip here as skip just doesn't run the test at all, but xfail will guarantee that the behavior is failing as expected and so is a stronger test.
|
As this is a fix for comparison of baseline images and necessary to get the test suite passing for the v0.7.0 release, I'm going to approve and merge this myself. As always, PRs approved by a single core dev can be reverted as needed by the rest of the dev team. |
matthewfeickert
added a commit
that referenced
this pull request
Sep 20, 2022
Split warning for readability Add test for rtol warning Remove tmpdir fixture as not needed for these tests given no writing of output Add check for return_results More verbose fix: Correct concatenate lists instead of adding float to all list elements Test bounds expansion test: Update test_plot_results_no_axis baseline image (#2009) * matplotlib v3.6.0 results in a slightly different baseline image than matplotlib v3.5.x, so regenerate the baseline image using matplotlib v3.6.0 with `pytest --mpl-generate-path=tests/contrib/baseline tests/contrib/test_viz.py`. * Mark the test_plot_results_no_axis test as xfail for Python 3.7 as matplotlib v3.6.0 is Python 3.8+ and so the image is guaranteed to be different as Python 3.7 runtimes will install matplotlib v3.5.x. Add upperlimit_fixed_scan to API docs Add return_results test move to test_upperlimit_with_kwargs Move the pop out before evaluation to make everything very clean and clear Note what scan Rename to auto_scan docs: fix link Provide better coverage and use np.allclose docs: Add Beojan Stanislaus to contributor list change auto_scan to toms748_scan rename fixed_scan to linear_grid_scan Make intervals module and change API to upper_limit Rename to pyhf.infer.intervals.upper_limits get upper_limits.upper_limit working Also bring along old API limit to just upper_limit by default Rearrange feat: Add internal API to warn of deprecation and future removal * Add internal API pyhf.exceptions._deprecated_api_warning to alert users to API deprecation by raising a subclass of DeprecationWarning and future removal. * Add test for pyhf.exceptions._deprecated_api_warning to ensure it gets picked up as DeprecationWarning. Note deprecated API Seperate into condifence intervals section fix: Use function scope to avoid altering hypotest_args fixture Make test name explicit Use deprecated Sphinx note Add versionadded directives feat: Add internal API to warn of deprecation and future removal (#2012) * Add internal API pyhf.exceptions._deprecated_api_warning to alert users to API deprecation by raising a subclass of DeprecationWarning and future removal. * Add test for pyhf.exceptions._deprecated_api_warning to ensure it gets picked up as DeprecationWarning.
matthewfeickert
added a commit
that referenced
this pull request
Sep 20, 2022
Split warning for readability Add test for rtol warning Remove tmpdir fixture as not needed for these tests given no writing of output Add check for return_results More verbose fix: Correct concatenate lists instead of adding float to all list elements Test bounds expansion test: Update test_plot_results_no_axis baseline image (#2009) * matplotlib v3.6.0 results in a slightly different baseline image than matplotlib v3.5.x, so regenerate the baseline image using matplotlib v3.6.0 with `pytest --mpl-generate-path=tests/contrib/baseline tests/contrib/test_viz.py`. * Mark the test_plot_results_no_axis test as xfail for Python 3.7 as matplotlib v3.6.0 is Python 3.8+ and so the image is guaranteed to be different as Python 3.7 runtimes will install matplotlib v3.5.x. Add upperlimit_fixed_scan to API docs Add return_results test move to test_upperlimit_with_kwargs Move the pop out before evaluation to make everything very clean and clear Note what scan Rename to auto_scan docs: fix link Provide better coverage and use np.allclose docs: Add Beojan Stanislaus to contributor list change auto_scan to toms748_scan rename fixed_scan to linear_grid_scan Make intervals module and change API to upper_limit Rename to pyhf.infer.intervals.upper_limits get upper_limits.upper_limit working Also bring along old API limit to just upper_limit by default Rearrange feat: Add internal API to warn of deprecation and future removal * Add internal API pyhf.exceptions._deprecated_api_warning to alert users to API deprecation by raising a subclass of DeprecationWarning and future removal. * Add test for pyhf.exceptions._deprecated_api_warning to ensure it gets picked up as DeprecationWarning. Note deprecated API Seperate into condifence intervals section fix: Use function scope to avoid altering hypotest_args fixture Make test name explicit Use deprecated Sphinx note Add versionadded directives feat: Add internal API to warn of deprecation and future removal (#2012) * Add internal API pyhf.exceptions._deprecated_api_warning to alert users to API deprecation by raising a subclass of DeprecationWarning and future removal. * Add test for pyhf.exceptions._deprecated_api_warning to ensure it gets picked up as DeprecationWarning. Update lower bound on scipy as toms748 added in scipy v1.2.0
matthewfeickert
added a commit
that referenced
this pull request
Sep 20, 2022
Use fstrings for rtol warning Use level arg over hardcoded 0.05 add in lru_cache import fix: Pin codemetapy to v0.3.5 for `--no-extras` functionality (#1995) * Pin codemetapy to v0.3.5 in the 'current release' test workflow to keep the `--no-extras` CLI API option. - c.f. proycon/codemetapy#24 * Update lower bounds for scipy and click in codemeta.json and add lower bounds for importlib-resources and typing-extensions. Rename and add to public API Add in hypotest_kwargs Add FIXME notice for later. FIX BEFORE MERGE Update public API repr ci: Install release candidates for 'current release' test workflow (#1996) * Use release candidates that are on PyPI for verifiying that the public API passes tests. This verifies that the release candidates that users are being asked to test reflect the release API. * Use the latest version of pytest. refactor: Use urllib.parse.urlsplit over urlparse (#1997) * Use urllib.parse.urlsplit over urllib.parse.urlparse to avoid having to deal with urlparse's 'params' argument which incurs a performance cost. - c.f. https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urlsplit - c.f. https://youtu.be/ABJvdsIANds Indent docstrings correctly Use uncorrelated_background API for docstring example Use rtol in docstring example to avoid warning drop lru_cache fix kwargs for autoscan Bump version: 0.7.0rc3 → 0.7.0rc4 docs: Add Binder Python runtime environment specification (#1998) * Add binder/runtime.txt to specify the version of Python that is used for the Binder environment. Use Python 3.10 as this is the latest version that all pyhf backends will work with. - c.f. https://mybinder.readthedocs.io/en/latest/howto/languages.html#python fix: Update codemeta lower bounds for jsonschema, importlib-resources (#2000) * Update jsonschema lower bound to v4.15.0 and importlib-resources lower bound to v1.4.0 to match their versions in setup.cfg. - Amends PR #1979 docs: Add milestone for 2000 project GitHub items (#2001) * Add milestone to README for 2000 project GitHub issues and pull requests. fix: Use codemetapy v2.2.2+ API (#2002) * Update codemetapy to v2.2.2+ in 'current release' workflow to have access to the `--no-extras` CLI API in v2.0+ and reproducible runs. - c.f. proycon/codemetapy#24 - c.f. proycon/codemetapy#26 - Amends PR #1995 * Use the codemetapy v2.0 API which requires `--inputtype python` to be added. * Update codemeta.json to follow codemetapy v2.0+ general spec. fix: Add filterwarnings ignore for protobuf DeprecationWarning (#2005) * Add a ignore to filterwarnings to avoid a protobuf DeprecationWarning > DeprecationWarning: Call to deprecated create function FileDescriptor(). > Note: Create unlinked descriptors is going to go away. Please use get/find > descriptors from generated code or query the descriptor_pool. from TensorFlow's use of protobuf. fix: Specify encoding as utf-8 to enforce PEP 597 (#2007) * Explicitly specify the encoding as utf-8 while opening a file to enforce PEP 597. This is future-proofing work to some degree as Python 3.15+ will make utf-8 the default. - c.f. https://peps.python.org/pep-0597/ * Add the flake8-encodings pre-commit hook to enforce PEP 597. docs: Add FAQ on reasons for need to downgrade dependencies (#1529) * Add FAQ explaining how reasons why users might have to manually downgrade dependencies. - c.f. PR #1979 for context docs: Seperate docstrings semantically Apply sourcery suggestion for simplification Rename to cached for clarity Add test for auto through upperlimit API Use None instead of auto to simplify API Avoid function level globals Use np.asarray to avoid copy Use lower and upper to match scipy terms Split warning for readability Add test for rtol warning Remove tmpdir fixture as not needed for these tests given no writing of output Add check for return_results More verbose fix: Correct concatenate lists instead of adding float to all list elements Test bounds expansion test: Update test_plot_results_no_axis baseline image (#2009) * matplotlib v3.6.0 results in a slightly different baseline image than matplotlib v3.5.x, so regenerate the baseline image using matplotlib v3.6.0 with `pytest --mpl-generate-path=tests/contrib/baseline tests/contrib/test_viz.py`. * Mark the test_plot_results_no_axis test as xfail for Python 3.7 as matplotlib v3.6.0 is Python 3.8+ and so the image is guaranteed to be different as Python 3.7 runtimes will install matplotlib v3.5.x. Add upperlimit_fixed_scan to API docs Add return_results test move to test_upperlimit_with_kwargs Move the pop out before evaluation to make everything very clean and clear Note what scan Rename to auto_scan docs: fix link Provide better coverage and use np.allclose docs: Add Beojan Stanislaus to contributor list change auto_scan to toms748_scan rename fixed_scan to linear_grid_scan Make intervals module and change API to upper_limit Rename to pyhf.infer.intervals.upper_limits get upper_limits.upper_limit working Also bring along old API limit to just upper_limit by default Rearrange feat: Add internal API to warn of deprecation and future removal * Add internal API pyhf.exceptions._deprecated_api_warning to alert users to API deprecation by raising a subclass of DeprecationWarning and future removal. * Add test for pyhf.exceptions._deprecated_api_warning to ensure it gets picked up as DeprecationWarning. Note deprecated API Seperate into condifence intervals section fix: Use function scope to avoid altering hypotest_args fixture Make test name explicit Use deprecated Sphinx note Add versionadded directives feat: Add internal API to warn of deprecation and future removal (#2012) * Add internal API pyhf.exceptions._deprecated_api_warning to alert users to API deprecation by raising a subclass of DeprecationWarning and future removal. * Add test for pyhf.exceptions._deprecated_api_warning to ensure it gets picked up as DeprecationWarning. Update lower bound on scipy as toms748 added in scipy v1.2.0 fixup from autoscan test changes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
matplotlib v3.6.0 results in a slightly different baseline image (the location of the$\mathrm{CL}_{s}$
y-axislabel is shifted very slighlty) than matplotlib v3.5.x, so regenerate the baseline image using matplotlib v3.6.0 withAlso mark the
test_plot_results_no_axistest asxfailfor Python 3.7 asmatplotlibv3.6.0is Python 3.8+ and so the image is guaranteed to be different as Python 3.7 runtimes will installmatplotlibv3.5.x.Checklist Before Requesting Reviewer
Before Merging
For the PR Assignees: