Merge pull request #1845 from kedro-org/merge-main-to-develop

[AUTO-MERGE] Merge main into develop via merge-main-to-develop

MANIFEST.in

@@ -3,5 +3,5 @@ include LICENSE.md
 include requirements.txt
 include test_requirements.txt
 include kedro/framework/project/default_logging.yml
-include kedro/extras/extensions/*.png
+include kedro/ipython/*.png
 recursive-include templates *

RELEASE.md

@@ -11,13 +11,14 @@
 
 ### Other
 * Removed deprecated `kedro.extras.ColorHandler`.
+* The Kedro IPython extension is no longer available as `%load_ext kedro.extras.extensions.ipython`; use `%load_ext kedro.ipython` instead.
 
 ## Migration guide from Kedro 0.18.* to 0.19.*
 
-
 # Upcoming Release 0.18.3
 
 ## Major features and improvements
+* The Kedro IPython extension should now be loaded with `%load_ext kedro.ipython`.
 * The line magic `%reload_kedro` now accepts keywords arguments, e.g. `%reload_kedro --env=prod`.
 
 ## Bug fixes and other changes
@@ -35,6 +36,7 @@
 * Relaxed `delta-spark` upper bound to allow compatibility with Spark 3.1.x and 3.2.x.
 
 ## Upcoming deprecations for Kedro 0.19.0
+* The Kedro IPython extension will no longer be available as `%load_ext kedro.extras.extensions.ipython`; use `%load_ext kedro.ipython` instead.
 
 # Release 0.18.2
 
@@ -140,7 +142,7 @@ main(
 * Added `save_args` to `feather.FeatherDataSet`.
 
 ### Jupyter and IPython integration
-* The [only recommended way to work with Kedro in Jupyter or IPython is now the Kedro IPython extension](https://kedro.readthedocs.io/en/0.18.0/tools_integration/ipython.html). Managed Jupyter instances should load this via `%load_ext kedro.extras.extensions.ipython` and use the line magic `%reload_kedro`.
+* The [only recommended way to work with Kedro in Jupyter or IPython is now the Kedro IPython extension](https://kedro.readthedocs.io/en/0.18.0/tools_integration/ipython.html). Managed Jupyter instances should load this via `%load_ext kedro.ipython` and use the line magic `%reload_kedro`.
 * `kedro ipython` launches an IPython session that preloads the Kedro IPython extension.
 * `kedro jupyter notebook/lab` creates a custom Jupyter kernel that preloads the Kedro IPython extension and launches a notebook with that kernel selected. There is no longer a need to specify `--all-kernels` to show all available kernels.
 

docs/source/deployment/databricks.md

@@ -230,7 +230,7 @@ Your complete notebook should look similar to this (the results are hidden):
 
 ### 9. Using the Kedro IPython Extension
 
-You can interact with Kedro in Databricks through the Kedro [IPython extension](https://ipython.readthedocs.io/en/stable/config/extensions/index.html), `kedro.extras.extensions.ipython`.
+You can interact with Kedro in Databricks through the Kedro [IPython extension](https://ipython.readthedocs.io/en/stable/config/extensions/index.html), `kedro.ipython`.
 
 The Kedro IPython extension launches a [Kedro session](../kedro_project_setup/session.md) and makes available the useful Kedro variables `catalog`, `context`, `pipelines` and `session`. It also provides the `%reload_kedro` [line magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html) that reloads these variables (for example, if you need to update `catalog` following changes to your Data Catalog).
 
@@ -239,7 +239,7 @@ The IPython extension can be used in a Databricks notebook in a similar way to h
 If you encounter a `ContextualVersionConflictError`, it is likely caused by Databricks using an old version of `pip`. Hence there's one additional step you need to do in the Databricks notebook to make use of the IPython extension. After you load the IPython extension using the below command:
 
 ```ipython
-In [1]: %load_ext kedro.extras.extensions.ipython
+In [1]: %load_ext kedro.ipython
 ```
 
 You must explicitly upgrade your `pip` version by doing the below:

docs/source/tools_integration/ipython.md

@@ -13,12 +13,12 @@ There are reasons why you may want to use a Notebook, although in general, the p
 
 ## Kedro IPython extension
 
-The recommended way to interact with Kedro in IPython and Jupyter is through the Kedro [IPython extension](https://ipython.readthedocs.io/en/stable/config/extensions/index.html), `kedro.extras.extensions.ipython`. An [IPython extension](https://ipython.readthedocs.io/en/stable/config/extensions/) is an importable Python module that has a couple of special functions to load and unload it.
+The recommended way to interact with Kedro in IPython and Jupyter is through the Kedro [IPython extension](https://ipython.readthedocs.io/en/stable/config/extensions/index.html), `kedro.ipython`. An [IPython extension](https://ipython.readthedocs.io/en/stable/config/extensions/) is an importable Python module that has a couple of special functions to load and unload it.
 
 The Kedro IPython extension launches a [Kedro session](../kedro_project_setup/session.md) and makes available the useful Kedro variables `catalog`, `context`, `pipelines` and `session`. It also provides the `%reload_kedro` [line magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html) that reloads these variables (for example, if you need to update `catalog` following changes to your Data Catalog).
 
 The simplest way to make use of the Kedro IPython extension is through the following commands:
-* `kedro ipython`. This launches an IPython shell with the extension already loaded and is equivalent to the command `ipython --ext kedro.extras.extensions.ipython`.
+* `kedro ipython`. This launches an IPython shell with the extension already loaded and is equivalent to the command `ipython --ext kedro.ipython`.
 * `kedro jupyter notebook`. This creates a custom Jupyter kernel that automatically loads the extension and launches Jupyter Notebook with this kernel selected.
 * `kedro jupyter lab`. This creates a custom Jupyter kernel that automatically loads the extension and launches JupyterLab with this kernel selected.
 
@@ -32,7 +32,7 @@ If these variables are not available then Kedro has not been able to load your p
 
 If the above commands are not available to you (e.g. you work in a managed Jupyter service such as a Databricks Notebook) then equivalent behaviour can be achieved by explicitly loading the Kedro IPython extension with the `%load_ext` line magic:
 ```ipython
-In [1]: %load_ext kedro.extras.extensions.ipython
+In [1]: %load_ext kedro.ipython
 ```
 
 If your IPython or Jupyter instance was launched from outside your Kedro project then you will need to run a second line magic to set the project path so that Kedro can load the `catalog`, `context`, `pipelines` and `session` variables:
@@ -42,7 +42,7 @@ In [2]: %reload_kedro <project_root>
 The Kedro IPython extension remembers the project path so that subsequent calls to `%reload_kedro` do not need to specify it:
 
 ```ipython
-In [1]: %load_ext kedro.extras.extensions.ipython
+In [1]: %load_ext kedro.ipython
 In [2]: %reload_kedro <project_root>
 In [3]: %reload_kedro
 ```
@@ -204,7 +204,7 @@ If you are not able to execute `kedro jupyter notebook` or `kedro jupyter lab` t
 
 ### Manage Jupyter kernels
 
-Behind the scenes, the `kedro jupyter notebook` and `kedro jupyter lab` commands create a Jupyter kernel named `kedro_<package_name>`. This kernel is identical to the [default IPython kernel](https://ipython.readthedocs.io/en/stable/install/kernel_install.html) but with a slightly customised [kernel specification](https://jupyter-client.readthedocs.io/en/stable/kernels.html#kernel-specs) that automatically loads `kedro.extras.extensions.ipython` when the kernel is started. The kernel specification is installed at a user level rather than system-wide.
+Behind the scenes, the `kedro jupyter notebook` and `kedro jupyter lab` commands create a Jupyter kernel named `kedro_<package_name>`. This kernel is identical to the [default IPython kernel](https://ipython.readthedocs.io/en/stable/install/kernel_install.html) but with a slightly customised [kernel specification](https://jupyter-client.readthedocs.io/en/stable/kernels.html#kernel-specs) that automatically loads `kedro.ipython` when the kernel is started. The kernel specification is installed at a user level rather than system-wide.
 
 ```{note}
 If a Jupyter kernel with the name `kedro_<package_name>` already exists then it is replaced. This ensures that the kernel always points to the correct Python executable. For example, if you change conda environment in a Kedro project then you should re-run `kedro jupyter notebook/lab` to replace the kernel specification with one that points to the new environment.

kedro/framework/cli/jupyter.py

@@ -117,7 +117,7 @@ def _create_kernel(kernel_name: str, display_name: str) -> None:
       "-f",
       "{connection_file}",
       "--ext",
-      "kedro.extras.extensions.ipython"
+      "kedro.ipython"
      ],
      "display_name": "Kedro (spaceflights)",
      "language": "python",
@@ -151,14 +151,14 @@ def _create_kernel(kernel_name: str, display_name: str) -> None:
 
         kernel_json = Path(kernel_path) / "kernel.json"
         kernel_spec = json.loads(kernel_json.read_text(encoding="utf-8"))
-        kernel_spec["argv"].extend(["--ext", "kedro.extras.extensions.ipython"])
+        kernel_spec["argv"].extend(["--ext", "kedro.ipython"])
         # indent=1 is to match the default ipykernel style (see
         # ipykernel.write_kernel_spec).
         kernel_json.write_text(json.dumps(kernel_spec, indent=1), encoding="utf-8")
 
-        kedro_extensions_dir = Path(__file__).parents[2] / "extras" / "extensions"
-        shutil.copy(kedro_extensions_dir / "logo-32x32.png", kernel_path)
-        shutil.copy(kedro_extensions_dir / "logo-64x64.png", kernel_path)
+        kedro_ipython_dir = Path(__file__).parents[2] / "ipython"
+        shutil.copy(kedro_ipython_dir / "logo-32x32.png", kernel_path)
+        shutil.copy(kedro_ipython_dir / "logo-64x64.png", kernel_path)
     except Exception as exc:
         raise KedroCliError(
             f"Cannot setup kedro kernel for Jupyter.\nError: {exc}"

kedro/framework/cli/pipeline.py

@@ -223,7 +223,6 @@ def _create_pipeline(name: str, output_dir: Path) -> Path:
     return result_path
 
 
-# pylint: disable=missing-raises-doc
 def _sync_dirs(source: Path, target: Path, prefix: str = "", overwrite: bool = False):
     """Recursively copies `source` directory (or file) into `target` directory without
     overwriting any existing files/directories in the target using the following

kedro/framework/cli/project.py

@@ -124,7 +124,7 @@ def ipython(
 
     if env:
         os.environ["KEDRO_ENV"] = env
-    call(["ipython", "--ext", "kedro.extras.extensions.ipython"] + list(args))
+    call(["ipython", "--ext", "kedro.ipython"] + list(args))
 
 
 @project_group.command()

kedro/extras/extensions/ipython.py → kedro/ipython/__init__.py

@@ -1,4 +1,4 @@
-# pylint: disable=import-outside-toplevel,global-statement,invalid-name,too-many-locals
+# pylint: disable=global-statement,invalid-name
 """
 This script creates an IPython extension to load Kedro-related variables in
 local scope.
@@ -8,24 +8,31 @@
 from pathlib import Path
 from typing import Any, Dict
 
+from IPython import get_ipython
+from IPython.core.magic import needs_local_scope, register_line_magic
+from IPython.core.magic_arguments import argument, magic_arguments, parse_argstring
+
+from kedro.framework.cli import load_entry_points
 from kedro.framework.cli.project import PARAMS_ARG_HELP
 from kedro.framework.cli.utils import ENV_HELP, _split_params
+from kedro.framework.project import LOGGING  # noqa
+from kedro.framework.project import configure_project, pipelines
+from kedro.framework.session import KedroSession
+from kedro.framework.startup import _is_project, bootstrap_project
 
 logger = logging.getLogger(__name__)
 default_project_path = Path.cwd()
 
 
-def _remove_cached_modules(package_name):
+def _remove_cached_modules(package_name):  # pragma: no cover
     to_remove = [mod for mod in sys.modules if mod.startswith(package_name)]
     # `del` is used instead of `reload()` because: If the new version of a module does not
     # define a name that was defined by the old version, the old definition remains.
     for module in to_remove:
-        del sys.modules[module]  # pragma: no cover
+        del sys.modules[module]
 
 
 def _find_kedro_project(current_dir: Path):  # pragma: no cover
-    from kedro.framework.startup import _is_project
-
     while current_dir != current_dir.parent:
         if _is_project(current_dir):
             return current_dir
@@ -36,18 +43,9 @@ def _find_kedro_project(current_dir: Path):  # pragma: no cover
 
 def reload_kedro(
     path: str = None, env: str = None, extra_params: Dict[str, Any] = None
-):
-    """Line magic which reloads all Kedro default variables.
-    Setting the path will also make it default for subsequent calls.
-    """
-    from IPython import get_ipython
-    from IPython.core.magic import needs_local_scope, register_line_magic
-
-    from kedro.framework.cli import load_entry_points
-    from kedro.framework.project import LOGGING  # noqa # pylint:disable=unused-import
-    from kedro.framework.project import configure_project, pipelines
-    from kedro.framework.session import KedroSession
-    from kedro.framework.startup import bootstrap_project
+):  # pragma: no cover
+    """Function that underlies the %reload_kedro Line magic. This should not be imported
+    or run directly but instead invoked through %reload_kedro."""
 
     # If a path is provided, set it as default for subsequent calls
     global default_project_path
@@ -86,43 +84,40 @@ def reload_kedro(
         logger.info("Registered line magic '%s'", line_magic.__name__)  # type: ignore
 
 
+@magic_arguments()
+@argument(
+    "path",
+    type=str,
+    help=(
+        "Path to the project root directory. If not given, use the previously set"
+        "project root."
+    ),
+    nargs="?",
+    default=None,
+)
+@argument("-e", "--env", type=str, default=None, help=ENV_HELP)
+@argument(
+    "--params",
+    type=lambda value: _split_params(None, None, value),
+    default=None,
+    help=PARAMS_ARG_HELP,
+)
+def magic_reload_kedro(line: str):
+    """
+    The `%reload_kedro` IPython line magic. See
+     https://kedro.readthedocs.io/en/stable/tools_integration/ipython.html for more.
+    """
+    args = parse_argstring(magic_reload_kedro, line)
+    reload_kedro(args.path, args.env, args.params)
+
+
 def load_ipython_extension(ipython):
     """
-    Main entry point when %load_ext is executed.
+    Main entry point when %load_ext kedro.ipython is executed, either manually or
+    automatically through `kedro ipython` or `kedro jupyter lab/notebook`.
     IPython will look for this function specifically.
     See https://ipython.readthedocs.io/en/stable/config/extensions/index.html
-
-    This function is called when users do `%load_ext kedro.extras.extensions.ipython`.
-    When user use `kedro jupyter notebook` or `jupyter ipython`, this extension is
-    loaded automatically.
     """
-    from IPython.core.magic_arguments import argument, magic_arguments, parse_argstring
-
-    @magic_arguments()
-    @argument(
-        "path",
-        type=str,
-        help=(
-            "Path to the project root directory. If not given, use the previously set"
-            "project root."
-        ),
-        nargs="?",
-        default=None,
-    )
-    @argument("-e", "--env", type=str, default=None, help=ENV_HELP)
-    @argument(
-        "--params",
-        type=lambda value: _split_params(None, None, value),
-        default=None,
-        help=PARAMS_ARG_HELP,
-    )
-    def magic_reload_kedro(line: str):
-        """
-        The `%reload_kedro` IPython line magic. See
-         https://kedro.readthedocs.io/en/stable/tools_integration/ipython.html for more.
-        """
-        args = parse_argstring(magic_reload_kedro, line)
-        reload_kedro(args.path, args.env, args.params)
 
     global default_project_path
 

tests/framework/cli/test_jupyter.py

@@ -139,7 +139,7 @@ def test_create_new_kernel(self):
         kernel_spec = get_kernel_spec("my_kernel_name")
         assert kernel_spec.display_name == "My display name"
         assert kernel_spec.language == "python"
-        assert kernel_spec.argv[-2:] == ["--ext", "kedro.extras.extensions.ipython"]
+        assert kernel_spec.argv[-2:] == ["--ext", "kedro.ipython"]
         kernel_files = {file.name for file in Path(kernel_spec.resource_dir).iterdir()}
         assert kernel_files == {"logo-32x32.png", "logo-64x64.png", "kernel.json"}
 

tests/framework/cli/test_project.py

@@ -249,7 +249,7 @@ def test_happy_path(
             [
                 "ipython",
                 "--ext",
-                "kedro.extras.extensions.ipython",
+                "kedro.ipython",
                 "--random-arg",
                 "value",
             ]