Add code to test notebook code examples (#471)

This PR adds a script to automatically execute notebooks in CI.

To keep PRs small and focussed, this only adds the scripts to test
notebooks, it doesn't actually fix those notebooks or add anything to
CI.

<details>
  <summary>Example: Success</summary>

![Screenshot 2023-12-04 at 14 22
15](https://github.com/Qiskit/documentation/assets/36071638/2c6f698d-f990-456c-b20d-241db4d0007c)

</details>

<details>
  <summary>Example: Failure</summary>

![Screenshot 2023-12-04 at 14 23
17](https://github.com/Qiskit/documentation/assets/36071638/4983fcae-6d32-4c26-bdb3-4d6035d0a2fa)

</details>

<details>
  <summary>Why <code>tox</code>?</summary>

An important feature is to be able to run a single notebook in the
virtual environment. With `tox`, we can use commands like this:

```sh
tox -- path/to/notebook
```

Whereas with `nox`, this is the simplest I could get it:

```sh
nox --session 'test(path="path/to/notebook")'
```

Note this also doesn't autocomplete the path.

The main reason for choosing `nox` seems to be more flexible
configuration, but we only need simple configuration in this repo so
this isn't a problem.

</details>

***

Part of #166

---------

Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>

.gitignore

@@ -2,6 +2,7 @@
 .vscode/
 .fleet/
 .venv/
+.tox/
 .envrc
 
 .ipynb_checkpoints/

README.md

@@ -99,6 +99,28 @@ We also re-deploy the docs every time we merge into `main` at the site https://q
 
 This staging environment can be useful to see how the docs are rendering before we push it live to production.
 
+## Execute notebooks
+
+To execute notebooks in a fixed Python environment, first install `tox` using
+[pipx](https://pipx.pypa.io/stable/):
+
+```sh
+pipx install tox
+```
+
+- To execute all notebooks, run tox.
+  ```sh
+  tox
+  ```
+- To only execute specific notebooks, pass them as arguments.
+  ```sh
+  tox -- path/to/notebook.ipynb path/to/another-notbook.ipynb
+  ```
+- To write the execution results to the file, pass the `--write` argument.
+  ```sh
+  tox -- optional/paths/to/notebooks.ipynb --write
+  ```
+
 ## Check for broken links
 
 CI will check for broken links. You can also check locally:

scripts/nb-tester/requirements.txt

@@ -0,0 +1,8 @@
+# We pin to exact versions for a more reproducible and
+# stable build.
+nbconvert~=7.11.0
+nbformat~=5.9.2
+ipykernel~=6.27.1
+qiskit[all]~=0.45.1
+qiskit-aer~=0.13.1
+qiskit-ibm-runtime~=0.16.1

scripts/nb-tester/test-notebook.py

@@ -0,0 +1,96 @@
+# This code is a Qiskit project.
+#
+# (C) Copyright IBM 2023.
+#
+# This code is licensed under the Apache License, Version 2.0. You may obtain a
+# copy of this license in the LICENSE file in the root directory of this source
+# tree or at http://www.apache.org/licenses/LICENSE-2.0.
+#
+# Any modifications or derivative works of this code must retain this copyright
+# notice, and modified files need to carry a notice indicating that they have
+# been altered from the originals.
+
+import sys
+from pathlib import Path
+import nbconvert
+import nbformat
+
+WRITE_FLAG = "--write"
+NOTEBOOKS_GLOB = "docs/**/*.ipynb"
+NOTEBOOKS_EXCLUDE = [
+    "docs/api/**",
+    "**/.ipynb_checkpoints/**",
+    # Following notebooks have code errors
+    "docs/build/circuit-construction.ipynb",
+    "docs/build/pulse.ipynb",
+    "docs/transpile/custom-transpiler-pass.ipynb",
+    "docs/transpile/transpiler-stages.ipynb",
+    # Following notebook has some non-python dependency I can't figure out
+    "docs/build/circuit-visualization.ipynb",
+    # Following notebooks make requests so can't be tested yet
+    "docs/run/get-backend-information.ipynb",
+    "docs/start/hello-world.ipynb",
+]
+
+
+def execute_notebook(path: Path, write=False) -> bool:
+    """
+    Wrapper function for `_execute_notebook` to print status
+    """
+    print(f"▶️  {path}", end="", flush=True)
+    try:
+        _execute_notebook(path, write)
+    except nbconvert.preprocessors.CellExecutionError as err:
+        print("\r❌\n")
+        print(err)
+        return False
+    print("\r✅")
+    return True
+
+
+def _execute_notebook(filepath: Path, write=False) -> None:
+    """
+    Use nbconvert to execute notebook
+    """
+    nb = nbformat.read(filepath, as_version=4)
+
+    processor = nbconvert.preprocessors.ExecutePreprocessor(
+        timeout=100,
+        kernel_name="python3",
+    )
+
+    processor.preprocess(nb)
+
+    if not write:
+        return
+
+    for cell in nb.cells:
+        cell.metadata.pop("execution", None)
+    nbformat.write(nb, filepath)
+
+
+def find_notebooks() -> list[Path]:
+    """
+    Get paths to all notebooks in NOTEBOOKS_GLOB that are not excluded by
+    NOTEBOOKS_EXCLUDE
+    """
+    all_notebooks = Path(".").rglob(NOTEBOOKS_GLOB)
+    return [
+        path
+        for path in all_notebooks
+        if not any(path.match(glob) for glob in NOTEBOOKS_EXCLUDE)
+    ]
+
+
+if __name__ == "__main__":
+    args = sys.argv[1:]
+    write = WRITE_FLAG in args
+    if write:
+        args.remove(WRITE_FLAG)
+
+    notebook_paths = args or find_notebooks()
+    print("Executing notebooks:")
+    results = [execute_notebook(path, write) for path in notebook_paths]
+    if not all(results):
+        sys.exit(1)
+    sys.exit(0)

tox.ini

@@ -0,0 +1,9 @@
+[tox]
+min_version = 4.0
+env_list = py311
+skipsdist = true
+
+[testenv]
+deps = -r scripts/nb-tester/requirements.txt
+setenv = PYDEVD_DISABLE_FILE_VALIDATION=1
+commands = python scripts/nb-tester/test-notebook.py {posargs}