Feature/improve handling of local test files

Changes.md

@@ -2,6 +2,21 @@
 
 Note: development is tracked on the [`develop` branch](https://github.com/chmp/ipytest/tree/develop).
 
+## `development`
+
+- Support glob patterns in `ipytest.force_reload`. E.g.,
+  `ipytest.force_reload("test_*")`
+- Add `ipytest.autoconfig(force_reload=["test_*"])` to clear test modules on
+  each `ipytest` invocation
+
+## `0.14.2`
+
+- Support collecting branch coverage in notebooks (e.g., via `--cov--branch`)
+- Add `ipytest.autoconfig(coverage=True)` to simplify using `pytest-cov` inside
+  notebooks
+- Add experimental `ipytest.cov.translate_cell_filenames()` to simplify
+  interpretation of collected coverage information
+
 ## `0.14.1`
 
 - Add a [Coverage.py](https://coverage.readthedocs.io/en/latest/index.html)

Readme.md

@@ -1,4 +1,4 @@
-# ipytest - Pytest in Jupyter notebooks
+# ipytest - pytest in Jupyter notebooks
 
 [PyPI](https://pypi.org/project/ipytest)
 | [Usage](#usage)
@@ -10,7 +10,7 @@
 | [Related packages](#related-packages)
 | [License](#license)
 
-`ipytest` allows you to run [Pytest](https://pytest.org) in Jupyter notebooks.
+`ipytest` allows you to run [pytest](https://pytest.org) in Jupyter notebooks.
 `ipytest` aims to give access to the full `pytest` experience and to make it
 easy to transfer tests out of notebooks into separate test files.
 
@@ -44,39 +44,47 @@ and then run pytest. For further details on how to use `ipytest` see the
 
 ## Global state
 
-There are multiple sources of global state when using pytest inside the notebook:
-
-1. pytest will find any test function ever defined. This behavior can lead to
-   unexpected results when test functions are renamed, as their previous
-   definition is still available inside the kernel. Running
-   [`%%ipytest`][ipytest.ipytest] per default deletes any previously defined
-   tests. As an alternative the [`ipytest.clean()`][ipytest.clean]
-   function allows to delete previously defined tests.
-2. Python's module system caches imports and therefore acts as a global state.
-   To test the most recent version of any module, the module needs to be
-   reloaded. `ipytest` offers the
-   [`ipytest.force_reload()`][ipytest.force_reload] function. The `autoreload`
-   extension of IPython may also help here. To test local packages, it is
-   advisable to install them as development packages, e.g., `pip install -e .`.
-3. For async code, IPython will create an event loop in the current thread. This
-   setup may interfere with async tests. To support these use cases, ipytest
-   supports running tests in a separate thread. Simply setup ipytest via
-   `ipytest.autoconfig(run_in_thread=True)`.
+There are multiple sources of global state when using `pytest` inside the notebook:
+
+1. `pytest` will find any test function ever defined. This behavior can lead to unexpected results
+   when test functions are renamed, as their previous definition is still available inside the
+   kernel. Running [`%%ipytest`][ipytest.ipytest] per default deletes any previously defined tests.
+   The [`ipytest.clean()`][ipytest.clean] function allows to delete previously defined tests, as
+   well.
+2. Python's module system caches imports and therefore acts as a global state. To test the most
+   recent version of any module, the module needs to be reloaded. `ipytest` offers the
+   [`ipytest.force_reload()`][ipytest.force_reload] function. The `autoreload` extension of IPython
+   may also help here. To test local packages, it is advisable to install them as development
+   packages, e.g., `pip install -e .`.
+
+   Note that local test files are imported by `pytest` and fall under the same restriction as other
+   modules. To test the most recent version of the tests, the corresponding modules need to be
+   reloaded. The [import behavior][pytest-import-docs] depends on both whether the tests are
+   organized as packages (are `__init__.py` files present?) and the `pytest` configuration. For the
+   default `pytest` configuration, `ipytest` can be configured with
+   `ipytest.autoconfig(force_reload="test_*")`, assuming test modules prefixed with `test_` not
+   organized into packages, or `ipytest.autoconfig(force_reload="tests")`, assuming tests grouped in
+   a `tests` package.
+3. For async code, IPython will create an event loop in the current thread. This setup may interfere
+   with async tests. To support these use cases, ipytest supports running tests in a separate
+   thread. Simply setup ipytest via `ipytest.autoconfig(run_in_thread=True)`.
+
+[pytest-import-docs]: https://docs.pytest.org/en/latest/explanation/pythonpath.html
 
 ## How does it work?
 
 In its default configuration (via `autoconfig()`), `ipytest` performs the
 following steps:
 
-1. Register pytest's assertion rewriter with the IPython kernel. The rewriter
+1. Register `pytest`'s assertion rewriter with the IPython kernel. The rewriter
    will rewrite any assert statements entered into the notebook to give better
    error messages. This change will affect also non test based code, but should
    generally improve the development experience.
 2. Ensure the notebook can be mapped to a file. `ipytest` will create a
    temporary file in the current directory and remove if afterwards.
 3. Register the notebook scope temporarily as a module. This step is necessary
-   to allow pytest's doctest plugin to import the notebook.
-4. Call pytest with the name of the temporary module
+   to allow `pytest`'s doctest plugin to import the notebook.
+4. Call `pytest` with the name of the temporary module
 
 **NOTE:** Some notebook implementations modify the core IPython package and
 magics may not work correctly (see [here][issue-47] or [here][issue-50]). In
@@ -99,22 +107,24 @@ this case, using [`ipytest.run()`][ipytest.run] and
 | [`ipytest.cov`](#ipytestcov)
 
 <!-- minidoc "function": "ipytest.autoconfig", "header_depth": 3 -->
-### `ipytest.autoconfig(rewrite_asserts=<default>, magics=<default>, clean=<default>, addopts=<default>, run_in_thread=<default>, defopts=<default>, display_columns=<default>, raise_on_error=<default>)`
+### `ipytest.autoconfig(rewrite_asserts=<default>, magics=<default>, clean=<default>, addopts=<default>, run_in_thread=<default>, defopts=<default>, display_columns=<default>, raise_on_error=<default>, coverage=<default>, force_reload=<default>)`
 
-[ipytest.autoconfig]: #ipytestautoconfigrewrite_assertsdefault-magicsdefault-cleandefault-addoptsdefault-run_in_threaddefault-defoptsdefault-display_columnsdefault-raise_on_errordefault
+[ipytest.autoconfig]: #ipytestautoconfigrewrite_assertsdefault-magicsdefault-cleandefault-addoptsdefault-run_in_threaddefault-defoptsdefault-display_columnsdefault-raise_on_errordefault-coveragedefault-force_reloaddefault
 
 Configure `ipytest` with reasonable defaults.
 
 Specifically, it sets:
 
-* `rewrite_asserts`: `True`
-* `magics`: `True`
-* `clean`: `'[Tt]est*'`
 * `addopts`: `('-q', '--color=yes')`
-* `run_in_thread`: `False`
+* `clean`: `'[Tt]est*'`
+* `coverage`: `False`
 * `defopts`: `'auto'`
 * `display_columns`: `100`
+* `magics`: `True`
 * `raise_on_error`: `False`
+* `rewrite_asserts`: `True`
+* `run_in_thread`: `False`
+* `force_reload`: `()`
 
 See [`ipytest.config`][ipytest.config] for details.
 
@@ -136,7 +146,7 @@ current cell are executed. To disable this behavior, use
 [`ipytest.config(clean=False)`][ipytest.config].
 
 Any arguments passed on the magic line are interpreted as command line
-arguments to to pytest. For example calling the magic as
+arguments to to `pytest`. For example calling the magic as
 
 ```python
 %%ipytest -qq
@@ -169,9 +179,9 @@ inside a CI/CD context, use `ipytest.autoconfig(raise_on_error=True)`.
 <!-- minidoc -->
 
 <!-- minidoc "function": "ipytest.config", "header_depth": 3 -->
-### `ipytest.config(rewrite_asserts=<keep>, magics=<keep>, clean=<keep>, addopts=<keep>, run_in_thread=<keep>, defopts=<keep>, display_columns=<keep>, raise_on_error=<keep>)`
+### `ipytest.config(rewrite_asserts=<keep>, magics=<keep>, clean=<keep>, addopts=<keep>, run_in_thread=<keep>, defopts=<keep>, display_columns=<keep>, raise_on_error=<keep>, coverage=<keep>, force_reload=<keep>)`
 
-[ipytest.config]: #ipytestconfigrewrite_assertskeep-magicskeep-cleankeep-addoptskeep-run_in_threadkeep-defoptskeep-display_columnskeep-raise_on_errorkeep
+[ipytest.config]: #ipytestconfigrewrite_assertskeep-magicskeep-cleankeep-addoptskeep-run_in_threadkeep-defoptskeep-display_columnskeep-raise_on_errorkeep-coveragekeep-force_reloadkeep
 
 Configure `ipytest`
 
@@ -186,6 +196,12 @@ The following settings are supported:
 * `rewrite_asserts` (default: `False`): enable ipython AST transforms
   globally to rewrite asserts
 * `magics` (default: `False`): if set to `True` register the ipytest magics
+* `coverage` (default: `False`): if `True` configure `pytest` to collect
+  coverage information. This functionality requires the `pytest-cov` package
+  to be installed. It adds `--cov --cov-config={GENERATED_CONFIG}` to the
+  arguments when invoking `pytest`. **WARNING**: this option will hide
+  existing coverage configuration files. See [`ipytest.cov`](#ipytestcov)
+  for details
 * `clean` (default: `[Tt]est*`): the pattern used to clean variables
 * `addopts` (default: `()`): pytest command line arguments to prepend to
   every pytest invocation. For example setting
@@ -204,9 +220,14 @@ The following settings are supported:
 * `display_columns` (default: `100`): if not `False`, configure pytest to
   use the given number of columns for its output. This option will
   temporarily override the `COLUMNS` environment variable.
-* `raise_on_error` (default `False` ): if `True`,
+* `raise_on_error` (default `False`): if `True`,
   [`ipytest.run`][ipytest.run] and [`%%ipytest`][ipytest.ipytest] will raise
   an `ipytest.Error` if pytest fails.
+* `force_reload` (default `()`): a sequence of modules to remove from the
+  global module cache before executing tests. The listed modules are passed
+  to [`ipytest.force_reload`][ipytest.force_reload]. For simplicity, a
+  single module can also be specified as a string. Glob-style wildcards are
+  supported.
 
 <!-- minidoc -->
 
@@ -217,9 +238,9 @@ The following settings are supported:
 The return code of the last pytest invocation.
 
 <!-- minidoc "function": "ipytest.run", "header_depth": 3 -->
-### `ipytest.run(*args, module=None, plugins=(), run_in_thread=<default>, raise_on_error=<default>, addopts=<default>, defopts=<default>, display_columns=<default>)`
+### `ipytest.run(*args, module=None, plugins=(), run_in_thread=<default>, raise_on_error=<default>, addopts=<default>, defopts=<default>, display_columns=<default>, coverage=<default>, force_reload=<default>)`
 
-[ipytest.run]: #ipytestrunargs-modulenone-plugins-run_in_threaddefault-raise_on_errordefault-addoptsdefault-defoptsdefault-display_columnsdefault
+[ipytest.run]: #ipytestrunargs-modulenone-plugins-run_in_threaddefault-raise_on_errordefault-addoptsdefault-defoptsdefault-display_columnsdefault-coveragedefault-force_reloaddefault
 
 Execute all tests in the passed module (defaults to `__main__`) with pytest.
 
@@ -239,13 +260,9 @@ inside a CI/CD context, use `ipytest.autoconfig(raise_on_error=True)`.
 
 The following parameters override the config options set with
 [`ipytest.config()`][ipytest.config] or
-[`ipytest.autoconfig()`][ipytest.autoconfig].
-
-- `run_in_thread`: if given, override the config option "run_in_thread".
-- `raise_on_error`: if given, override the config option "raise_on_error".
-- `addopts`: if given, override the config option "addopts".
-- `defopts`: if given, override the config option "defopts".
-- `display_columns`: if given, override the config option "display_columns".
+[`ipytest.autoconfig()`][ipytest.autoconfig]: `run_in_thread`,
+`raise_on_error`, `addopts`, `defopts`, `display_columns`, `coverage`,
+`force_reload`.
 
 **Returns**: the exit code of `pytest.main`.
 
@@ -283,10 +300,11 @@ as expected.
 Ensure following imports of the listed modules reload the code from disk
 
 The given modules and their submodules are removed from `sys.modules`.
-Next time the modules are imported, they are loaded from disk.
+Next time the modules are imported, they are loaded from disk. The module
+names can use glob patterns, e.g., `test_*` to delete all test modules.
 
-If given, the parameter `modules` should be a dictionary of modules to use
-instead of `sys.modules`.
+If given, the parameter `modules` should be a dictionary of modules to work
+on instead of `sys.modules`.
 
 Usage:
 
@@ -295,6 +313,14 @@ ipytest.force_reload("my_package")
 from my_package.submodule import my_function
 ```
 
+This function can be used to ensure that the most recent version of test
+files is used inside notebook via:
+
+```python
+ipytest.force_reload("test_*")
+ipytest.run(".")
+```
+
 <!-- minidoc -->
 <!-- minidoc "class": "ipytest.Error", "header_depth": 3 -->
 ### `ipytest.Error(exit_code)`
@@ -320,19 +346,45 @@ plugins =
     ipytest.cov
 ```
 
-With this config file, the coverage can be collected using
+With this config file, coverage information can be collected using
 [pytest-cov][ipytest-cov-pytest-cov] with
 
-```pyhton
+```python
 %%ipytest --cov
 
 def test():
     ...
 ```
 
+`ipytest.autoconfig(coverage=True)` automatically adds the `--cov` flag and the
+path of a generated config file to the Pytest invocation. In this case no
+further configuration is required.
+
+There are some known issues of `ipytest.cov`
+
+- Each notebook cell is reported as an individual file
+- Lines that are executed at import time may not be encountered in tracing and
+  may be reported as not-covered (One example is the line of a function
+  definition)
+- Marking code to be excluded in branch coverage is currently not supported
+  (incl. coveragepy pragmas)
+
 [coverage-py-config-docs]: https://coverage.readthedocs.io/en/latest/config.html
 [ipytest-cov-pytest-cov]: https://pytest-cov.readthedocs.io/en/latest/config.html
 
+#### `ipytest.cov.translate_cell_filenames(enabled=True)`
+
+[ipytest.cov.translate_cell_filenames]: #ipytestcovtranslate_cell_filenamesenabledtrue
+
+Translate the filenames of notebook cells in coverage information.
+
+If enabled, `ipytest.cov` will translate the temporary file names generated
+by ipykernel (e.g, `ipykernel_24768/3920661193.py`) to their cell names
+(e.g., `In[6]`).
+
+**Warning**: this is an experimental feature and not subject to any
+stability guarantees.
+
 <!-- minidoc -->
 
 ## Development

ipytest/_config.py

@@ -5,25 +5,29 @@
 default_clean = "[Tt]est*"
 
 defaults = {
-    "rewrite_asserts": True,
-    "magics": True,
-    "clean": default_clean,
     "addopts": ("-q", "--color=yes"),
-    "run_in_thread": False,
+    "clean": default_clean,
+    "coverage": False,
     "defopts": "auto",
     "display_columns": 100,
+    "magics": True,
     "raise_on_error": False,
+    "rewrite_asserts": True,
+    "run_in_thread": False,
+    "force_reload": (),
 }
 
 current_config = {
-    "rewrite_asserts": False,
-    "magics": False,
-    "clean": default_clean,
     "addopts": (),
-    "run_in_thread": False,
+    "clean": default_clean,
+    "coverage": False,
     "defopts": "auto",
     "display_columns": 100,
+    "magics": False,
     "raise_on_error": False,
+    "rewrite_asserts": False,
+    "run_in_thread": False,
+    "force_reload": (),
 }
 
 _rewrite_transformer = None
@@ -67,6 +71,8 @@ def autoconfig(
     defopts=default,
     display_columns=default,
     raise_on_error=default,
+    coverage=default,
+    force_reload=default,
 ):
     """Configure `ipytest` with reasonable defaults.
 
@@ -91,6 +97,8 @@ def config(
     defopts=keep,
     display_columns=keep,
     raise_on_error=keep,
+    coverage=keep,
+    force_reload=keep,
 ):
     """Configure `ipytest`
 
@@ -105,6 +113,12 @@ def config(
     * `rewrite_asserts` (default: `False`): enable ipython AST transforms
       globally to rewrite asserts
     * `magics` (default: `False`): if set to `True` register the ipytest magics
+    * `coverage` (default: `False`): if `True` configure `pytest` to collect
+      coverage information. This functionality requires the `pytest-cov` package
+      to be installed. It adds `--cov --cov-config={GENERATED_CONFIG}` to the
+      arguments when invoking `pytest`. **WARNING**: this option will hide
+      existing coverage configuration files. See [`ipytest.cov`](#ipytestcov)
+      for details
     * `clean` (default: `[Tt]est*`): the pattern used to clean variables
     * `addopts` (default: `()`): pytest command line arguments to prepend to
       every pytest invocation. For example setting
@@ -123,9 +137,14 @@ def config(
     * `display_columns` (default: `100`): if not `False`, configure pytest to
       use the given number of columns for its output. This option will
       temporarily override the `COLUMNS` environment variable.
-    * `raise_on_error` (default `False` ): if `True`,
+    * `raise_on_error` (default `False`): if `True`,
       [`ipytest.run`][ipytest.run] and [`%%ipytest`][ipytest.ipytest] will raise
       an `ipytest.Error` if pytest fails.
+    * `force_reload` (default `()`): a sequence of modules to remove from the
+      global module cache before executing tests. The listed modules are passed
+      to [`ipytest.force_reload`][ipytest.force_reload]. For simplicity, a
+      single module can also be specified as a string. Glob-style wildcards are
+      supported.
     """
     args = collect_args()
     new_config = {