Pytester runexamples #3623

RonnyPfannschmidt · 2018-06-26T21:01:17Z

this starts to make use of the example dir in testing

nicoddemus

Overall nice!

Missing some docs and changelog entry, otherwise looks like a good start.

nicoddemus · 2018-06-27T00:05:31Z

src/_pytest/pytester.py

@@ -628,6 +632,14 @@ def mkpydir(self, name):
        p.ensure("__init__.py")
        return p

+    def copy_example(self, name):
+        example_dir = self.request.config.getini("pytester_example_dir")


Probably makes sense to raise an error if pytester_example_dir is None.

Should we mark this as experimental?

yes we should - what mechanism do we want to use for that?

RonnyPfannschmidt · 2018-06-27T06:28:40Z

@nicoddemus done, waiting for ci

nicoddemus

Nice!

This PR updates [pytest](https://pypi.org/project/pytest) from **3.6.4** to **3.7.0**. <details> <summary>Changelog</summary> ### 3.7.0 ``` ========================= Deprecations and Removals ------------------------- - `2639 <https://github.com/pytest-dev/pytest/issues/2639>`_: ``pytest_namespace`` has been deprecated. See the documentation for ``pytest_namespace`` hook for suggestions on how to deal with this in plugins which use this functionality. - `3661 <https://github.com/pytest-dev/pytest/issues/3661>`_: Calling a fixture function directly, as opposed to request them in a test function, now issues a ``RemovedInPytest4Warning``. It will be changed into an error in pytest ``4.0``. This is a great source of confusion to new users, which will often call the fixture functions and request them from test functions interchangeably, which breaks the fixture resolution model. Features -------- - `2283 <https://github.com/pytest-dev/pytest/issues/2283>`_: New ``package`` fixture scope: fixtures are finalized when the last test of a *package* finishes. This feature is considered **experimental**, so use it sparingly. - `3576 <https://github.com/pytest-dev/pytest/issues/3576>`_: ``Node.add_marker`` now supports an ``append=True/False`` parameter to determine whether the mark comes last (default) or first. - `3579 <https://github.com/pytest-dev/pytest/issues/3579>`_: Fixture ``caplog`` now has a ``messages`` property, providing convenient access to the format-interpolated log messages without the extra data provided by the formatter/handler. - `3610 <https://github.com/pytest-dev/pytest/issues/3610>`_: New ``--trace`` option to enter the debugger at the start of a test. - `3623 <https://github.com/pytest-dev/pytest/issues/3623>`_: Introduce ``pytester.copy_example`` as helper to do acceptance tests against examples from the project. Bug Fixes --------- - `2220 <https://github.com/pytest-dev/pytest/issues/2220>`_: Fix a bug where fixtures overridden by direct parameters (for example parametrization) were being instantiated even if they were not being used by a test. - `3695 <https://github.com/pytest-dev/pytest/issues/3695>`_: Fix ``ApproxNumpy`` initialisation argument mixup, ``abs`` and ``rel`` tolerances were flipped causing strange comparsion results. Add tests to check ``abs`` and ``rel`` tolerances for ``np.array`` and test for expecting ``nan`` with ``np.array()`` - `980 <https://github.com/pytest-dev/pytest/issues/980>`_: Fix truncated locals output in verbose mode. Improved Documentation ---------------------- - `3295 <https://github.com/pytest-dev/pytest/issues/3295>`_: Correct the usage documentation of ``--last-failed-no-failures`` by adding the missing ``--last-failed`` argument in the presented examples, because they are misleading and lead to think that the missing argument is not needed. Trivial/Internal Changes ------------------------ - `3519 <https://github.com/pytest-dev/pytest/issues/3519>`_: Now a ``README.md`` file is created in ``.pytest_cache`` to make it clear why the directory exists. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pytest - Changelog: https://pyup.io/changelogs/pytest/ - Repo: https://github.com/pytest-dev/pytest/issues - Homepage: http://pytest.org </details>

This PR updates [pytest](https://pypi.org/project/pytest) from **3.6.2** to **3.7.1**. <details> <summary>Changelog</summary> ### 3.7.1 ``` ========================= Bug Fixes --------- - `3473 <https://github.com/pytest-dev/pytest/issues/3473>`_: Raise immediately if ``approx()`` is given an expected value of a type it doesn't understand (e.g. strings, nested dicts, etc.). - `3712 <https://github.com/pytest-dev/pytest/issues/3712>`_: Correctly represent the dimensions of an numpy array when calling ``repr()`` on ``approx()``. - `3742 <https://github.com/pytest-dev/pytest/issues/3742>`_: Fix incompatibility with third party plugins during collection, which produced the error ``object has no attribute '_collectfile'``. - `3745 <https://github.com/pytest-dev/pytest/issues/3745>`_: Display the absolute path if ``cache_dir`` is not relative to the ``rootdir`` instead of failing. - `3747 <https://github.com/pytest-dev/pytest/issues/3747>`_: Fix compatibility problem with plugins and the warning code issued by fixture functions when they are called directly. - `3748 <https://github.com/pytest-dev/pytest/issues/3748>`_: Fix infinite recursion in ``pytest.approx`` with arrays in ``numpy<1.13``. - `3757 <https://github.com/pytest-dev/pytest/issues/3757>`_: Pin pathlib2 to ``>=2.2.0`` as we require ``__fspath__`` support. - `3763 <https://github.com/pytest-dev/pytest/issues/3763>`_: Fix ``TypeError`` when the assertion message is ``bytes`` in python 3. ``` ### 3.7.0 ``` ========================= Deprecations and Removals ------------------------- - `2639 <https://github.com/pytest-dev/pytest/issues/2639>`_: ``pytest_namespace`` has been deprecated. See the documentation for ``pytest_namespace`` hook for suggestions on how to deal with this in plugins which use this functionality. - `3661 <https://github.com/pytest-dev/pytest/issues/3661>`_: Calling a fixture function directly, as opposed to request them in a test function, now issues a ``RemovedInPytest4Warning``. It will be changed into an error in pytest ``4.0``. This is a great source of confusion to new users, which will often call the fixture functions and request them from test functions interchangeably, which breaks the fixture resolution model. Features -------- - `2283 <https://github.com/pytest-dev/pytest/issues/2283>`_: New ``package`` fixture scope: fixtures are finalized when the last test of a *package* finishes. This feature is considered **experimental**, so use it sparingly. - `3576 <https://github.com/pytest-dev/pytest/issues/3576>`_: ``Node.add_marker`` now supports an ``append=True/False`` parameter to determine whether the mark comes last (default) or first. - `3579 <https://github.com/pytest-dev/pytest/issues/3579>`_: Fixture ``caplog`` now has a ``messages`` property, providing convenient access to the format-interpolated log messages without the extra data provided by the formatter/handler. - `3610 <https://github.com/pytest-dev/pytest/issues/3610>`_: New ``--trace`` option to enter the debugger at the start of a test. - `3623 <https://github.com/pytest-dev/pytest/issues/3623>`_: Introduce ``pytester.copy_example`` as helper to do acceptance tests against examples from the project. Bug Fixes --------- - `2220 <https://github.com/pytest-dev/pytest/issues/2220>`_: Fix a bug where fixtures overridden by direct parameters (for example parametrization) were being instantiated even if they were not being used by a test. - `3695 <https://github.com/pytest-dev/pytest/issues/3695>`_: Fix ``ApproxNumpy`` initialisation argument mixup, ``abs`` and ``rel`` tolerances were flipped causing strange comparsion results. Add tests to check ``abs`` and ``rel`` tolerances for ``np.array`` and test for expecting ``nan`` with ``np.array()`` - `980 <https://github.com/pytest-dev/pytest/issues/980>`_: Fix truncated locals output in verbose mode. Improved Documentation ---------------------- - `3295 <https://github.com/pytest-dev/pytest/issues/3295>`_: Correct the usage documentation of ``--last-failed-no-failures`` by adding the missing ``--last-failed`` argument in the presented examples, because they are misleading and lead to think that the missing argument is not needed. Trivial/Internal Changes ------------------------ - `3519 <https://github.com/pytest-dev/pytest/issues/3519>`_: Now a ``README.md`` file is created in ``.pytest_cache`` to make it clear why the directory exists. ``` ### 3.6.4 ``` ========================= Bug Fixes --------- - Invoke pytest using ``-mpytest`` so ``sys.path`` does not get polluted by packages installed in ``site-packages``. (`742 <https://github.com/pytest-dev/pytest/issues/742>`_) Improved Documentation ---------------------- - Use ``smtp_connection`` instead of ``smtp`` in fixtures documentation to avoid possible confusion. (`3592 <https://github.com/pytest-dev/pytest/issues/3592>`_) Trivial/Internal Changes ------------------------ - Remove obsolete ``__future__`` imports. (`2319 <https://github.com/pytest-dev/pytest/issues/2319>`_) - Add CITATION to provide information on how to formally cite pytest. (`3402 <https://github.com/pytest-dev/pytest/issues/3402>`_) - Replace broken type annotations with type comments. (`3635 <https://github.com/pytest-dev/pytest/issues/3635>`_) - Pin ``pluggy`` to ``<0.8``. (`3727 <https://github.com/pytest-dev/pytest/issues/3727>`_) ``` ### 3.6.3 ``` ========================= Bug Fixes --------- - Fix ``ImportWarning`` triggered by explicit relative imports in assertion-rewritten package modules. (`3061 <https://github.com/pytest-dev/pytest/issues/3061>`_) - Fix error in ``pytest.approx`` when dealing with 0-dimension numpy arrays. (`3593 <https://github.com/pytest-dev/pytest/issues/3593>`_) - No longer raise ``ValueError`` when using the ``get_marker`` API. (`3605 <https://github.com/pytest-dev/pytest/issues/3605>`_) - Fix problem where log messages with non-ascii characters would not appear in the output log file. (`3630 <https://github.com/pytest-dev/pytest/issues/3630>`_) - No longer raise ``AttributeError`` when legacy marks can't be stored in functions. (`3631 <https://github.com/pytest-dev/pytest/issues/3631>`_) Improved Documentation ---------------------- - The description above the example for ``pytest.mark.skipif`` now better matches the code. (`3611 <https://github.com/pytest-dev/pytest/issues/3611>`_) Trivial/Internal Changes ------------------------ - Internal refactoring: removed unused ``CallSpec2tox ._globalid_args`` attribute and ``metafunc`` parameter from ``CallSpec2.copy()``. (`3598 <https://github.com/pytest-dev/pytest/issues/3598>`_) - Silence usage of ``reduce`` warning in Python 2 (`3609 <https://github.com/pytest-dev/pytest/issues/3609>`_) - Fix usage of ``attr.ib`` deprecated ``convert`` parameter. (`3653 <https://github.com/pytest-dev/pytest/issues/3653>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pytest - Changelog: https://pyup.io/changelogs/pytest/ - Repo: https://github.com/pytest-dev/pytest/issues - Homepage: http://pytest.org </details>

This PR updates [pytest](https://pypi.org/project/pytest) from **3.6.4** to **3.7.2**. <details> <summary>Changelog</summary> ### 3.7.2 ``` ========================= Bug Fixes --------- - `3671 <https://github.com/pytest-dev/pytest/issues/3671>`_: Fix ``filterwarnings`` not being registered as a builtin mark. - `3768 <https://github.com/pytest-dev/pytest/issues/3768>`_, `3789 <https://github.com/pytest-dev/pytest/issues/3789>`_: Fix test collection from packages mixed with normal directories. - `3771 <https://github.com/pytest-dev/pytest/issues/3771>`_: Fix infinite recursion during collection if a ``pytest_ignore_collect`` hook returns ``False`` instead of ``None``. - `3774 <https://github.com/pytest-dev/pytest/issues/3774>`_: Fix bug where decorated fixtures would lose functionality (for example ``mock.patch``). - `3775 <https://github.com/pytest-dev/pytest/issues/3775>`_: Fix bug where importing modules or other objects with prefix ``pytest_`` prefix would raise a ``PluginValidationError``. - `3788 <https://github.com/pytest-dev/pytest/issues/3788>`_: Fix ``AttributeError`` during teardown of ``TestCase`` subclasses which raise an exception during ``__init__``. - `3804 <https://github.com/pytest-dev/pytest/issues/3804>`_: Fix traceback reporting for exceptions with ``__cause__`` cycles. Improved Documentation ---------------------- - `3746 <https://github.com/pytest-dev/pytest/issues/3746>`_: Add documentation for ``metafunc.config`` that had been mistakenly hidden. ``` ### 3.7.1 ``` ========================= Bug Fixes --------- - `3473 <https://github.com/pytest-dev/pytest/issues/3473>`_: Raise immediately if ``approx()`` is given an expected value of a type it doesn't understand (e.g. strings, nested dicts, etc.). - `3712 <https://github.com/pytest-dev/pytest/issues/3712>`_: Correctly represent the dimensions of an numpy array when calling ``repr()`` on ``approx()``. - `3742 <https://github.com/pytest-dev/pytest/issues/3742>`_: Fix incompatibility with third party plugins during collection, which produced the error ``object has no attribute '_collectfile'``. - `3745 <https://github.com/pytest-dev/pytest/issues/3745>`_: Display the absolute path if ``cache_dir`` is not relative to the ``rootdir`` instead of failing. - `3747 <https://github.com/pytest-dev/pytest/issues/3747>`_: Fix compatibility problem with plugins and the warning code issued by fixture functions when they are called directly. - `3748 <https://github.com/pytest-dev/pytest/issues/3748>`_: Fix infinite recursion in ``pytest.approx`` with arrays in ``numpy<1.13``. - `3757 <https://github.com/pytest-dev/pytest/issues/3757>`_: Pin pathlib2 to ``>=2.2.0`` as we require ``__fspath__`` support. - `3763 <https://github.com/pytest-dev/pytest/issues/3763>`_: Fix ``TypeError`` when the assertion message is ``bytes`` in python 3. ``` ### 3.7.0 ``` ========================= Deprecations and Removals ------------------------- - `2639 <https://github.com/pytest-dev/pytest/issues/2639>`_: ``pytest_namespace`` has been deprecated. See the documentation for ``pytest_namespace`` hook for suggestions on how to deal with this in plugins which use this functionality. - `3661 <https://github.com/pytest-dev/pytest/issues/3661>`_: Calling a fixture function directly, as opposed to request them in a test function, now issues a ``RemovedInPytest4Warning``. It will be changed into an error in pytest ``4.0``. This is a great source of confusion to new users, which will often call the fixture functions and request them from test functions interchangeably, which breaks the fixture resolution model. Features -------- - `2283 <https://github.com/pytest-dev/pytest/issues/2283>`_: New ``package`` fixture scope: fixtures are finalized when the last test of a *package* finishes. This feature is considered **experimental**, so use it sparingly. - `3576 <https://github.com/pytest-dev/pytest/issues/3576>`_: ``Node.add_marker`` now supports an ``append=True/False`` parameter to determine whether the mark comes last (default) or first. - `3579 <https://github.com/pytest-dev/pytest/issues/3579>`_: Fixture ``caplog`` now has a ``messages`` property, providing convenient access to the format-interpolated log messages without the extra data provided by the formatter/handler. - `3610 <https://github.com/pytest-dev/pytest/issues/3610>`_: New ``--trace`` option to enter the debugger at the start of a test. - `3623 <https://github.com/pytest-dev/pytest/issues/3623>`_: Introduce ``pytester.copy_example`` as helper to do acceptance tests against examples from the project. Bug Fixes --------- - `2220 <https://github.com/pytest-dev/pytest/issues/2220>`_: Fix a bug where fixtures overridden by direct parameters (for example parametrization) were being instantiated even if they were not being used by a test. - `3695 <https://github.com/pytest-dev/pytest/issues/3695>`_: Fix ``ApproxNumpy`` initialisation argument mixup, ``abs`` and ``rel`` tolerances were flipped causing strange comparsion results. Add tests to check ``abs`` and ``rel`` tolerances for ``np.array`` and test for expecting ``nan`` with ``np.array()`` - `980 <https://github.com/pytest-dev/pytest/issues/980>`_: Fix truncated locals output in verbose mode. Improved Documentation ---------------------- - `3295 <https://github.com/pytest-dev/pytest/issues/3295>`_: Correct the usage documentation of ``--last-failed-no-failures`` by adding the missing ``--last-failed`` argument in the presented examples, because they are misleading and lead to think that the missing argument is not needed. Trivial/Internal Changes ------------------------ - `3519 <https://github.com/pytest-dev/pytest/issues/3519>`_: Now a ``README.md`` file is created in ``.pytest_cache`` to make it clear why the directory exists. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pytest - Changelog: https://pyup.io/changelogs/pytest/ - Homepage: http://pytest.org </details>

RonnyPfannschmidt added 2 commits June 26, 2018 22:48

enable pytester to run examples copied from the cwd

0672bc6

port some acceptance tests over to copy_example

e860ff7

nicoddemus requested changes Jun 27, 2018

View reviewed changes

nicoddemus mentioned this pull request Jun 27, 2018

Add acceptance tests #2664

Closed

RonnyPfannschmidt added 3 commits June 27, 2018 06:52

add docs and changelog

581d496

regendoc

8a63455

regendoc and invocation fixes

17e0199

RonnyPfannschmidt changed the title ~~[wip] Pytester runexamples~~ Pytester runexamples Jun 27, 2018

nicoddemus approved these changes Jun 27, 2018

View reviewed changes

nicoddemus merged commit ea379ba into pytest-dev:features Jun 27, 2018

RonnyPfannschmidt deleted the pytester-runexamples branch August 2, 2018 10:31

blueyed mentioned this pull request Feb 24, 2020

tests: check result with test_510 - test likely needs fixing #6813

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pytester runexamples #3623

Pytester runexamples #3623

RonnyPfannschmidt commented Jun 26, 2018

nicoddemus left a comment

nicoddemus Jun 27, 2018

nicoddemus Jun 27, 2018

RonnyPfannschmidt Jun 27, 2018

RonnyPfannschmidt commented Jun 27, 2018

nicoddemus left a comment

Pytester runexamples #3623

Pytester runexamples #3623

Conversation

RonnyPfannschmidt commented Jun 26, 2018

nicoddemus left a comment

Choose a reason for hiding this comment

nicoddemus Jun 27, 2018

Choose a reason for hiding this comment

nicoddemus Jun 27, 2018

Choose a reason for hiding this comment

RonnyPfannschmidt Jun 27, 2018

Choose a reason for hiding this comment

RonnyPfannschmidt commented Jun 27, 2018

nicoddemus left a comment

Choose a reason for hiding this comment