[KED-3023] Enable plugins to extend starter-aliases default list

.pre-commit-config.yaml

@@ -47,6 +47,7 @@ repos:
             - types-requests
             - types-setuptools
             - types-toml
+            - attrs
 
     - repo: https://github.com/asottile/blacken-docs
       rev: v1.12.1

Makefile

@@ -20,7 +20,7 @@ test-no-spark:
 	pytest tests --no-cov --ignore tests/extras/datasets/spark --numprocesses 4 --dist loadfile
 
 e2e-tests:
-	behave
+	behave --no-capture
 
 pip-compile:
 	pip-compile -q -o -

RELEASE.md

@@ -18,6 +18,7 @@
 * Added `abfss` to list of cloud protocols, enabling abfss paths.
 * Kedro now uses the [Rich](https://github.com/Textualize/rich) library to format terminal logs and tracebacks.
 * The file `conf/base/logging.yml` is now optional. See [our documentation](https://kedro.readthedocs.io/en/0.18.2/logging/logging.html) for details.
+* Introduced a `kedro.starters` entry point. This enables plugins to create custom starter aliases used by `kedro starter list` and `kedro new`.
 
 ## Bug fixes and other changes
 * Bumped `pyyaml` upper bound to make Kedro compatible with the [pyodide](https://pyodide.org/en/stable/usage/loading-packages.html#micropip) stack.

docs/source/extend_kedro/create_kedro_starters.md

@@ -87,3 +87,7 @@ Here is the layout of the project as a Cookiecutter template:
     ├── setup.py
     └── tests
 ```
+
+```{note}
+You can add an alias by creating a plugin using [`kedro.starters` entry point](../plugin.md#extend-starter-aliases), which will allows you to do `kedro new --starter=your_starters` and shows up on shows up on `kedro starter list`.
+```

docs/source/extend_kedro/plugins.md

@@ -44,6 +44,31 @@ Once the plugin is installed, you can run it as follows:
 kedro to_json
 ```
 
+## Extend starter aliases
+It is possible to extend the list of starter aliases built into Kedro. This means that a [custom Kedro starter](create_kedro_starters.md) can be used directly through the `starter` argument in `kedro new` rather than needing to explicitly provide the `template` and `directory` arguments. A custom starter alias behaves in the same way as an official Kedro starter alias and is also picked up by `kedro starter list`.
+
+You need to extend the starters by providing a list of  `KedroStarterSpec`, in this example it is defined in a file called `plugin.py`.
+
+```python
+starters = [
+    KedroStarterSpec(
+        alias="test_plugin_starter",
+        template_path="https://github.com/kedro-org/kedro-starters/",
+        directory="pandas-iris",
+    )
+]
+
+The `directory` argument is optional and should be used when you have multiple templates in one repository as for the [official kedro-starters](https://github.com/kedro-org/kedro-starters). If you only have one template, your top-level directory will be treated as the template. You can take the [pandas-iris](https://github.com/kedro-org/kedro-starters/tree/main/pandas-iris) example for reference.
+```
+
+In your `setup.py`, you need to register the specifications to `kedro.starters`.
+
+```python
+setup(
+    entry_points={"kedro.starters": ["starter =  plugin:starters"]},
+)
+```
+
 ## Working with `click`
 
 Commands must be provided as [`click` `Groups`](https://click.palletsprojects.com/en/7.x/api/#click.Group)

features/activate_nbstripout.feature

@@ -2,7 +2,7 @@ Feature: Activate_nbstripout target in new project
 
   Scenario: Check nbstripout git post commit hook functionality
     Given I have prepared a config file
-    And I have run a non-interactive kedro new with starter
+    And I have run a non-interactive kedro new with starter "default"
     And I have added a test jupyter notebook
     And I have initialized a git repository
     And I have added the project directory to staging

features/build_docs.feature

@@ -3,7 +3,7 @@ Feature: build-docs target in new project
   @fresh_venv
   Scenario: Execute build-docs target
     Given I have prepared a config file
-    And I have run a non-interactive kedro new with starter
+    And I have run a non-interactive kedro new with starter "default"
     And I have updated kedro requirements
     And I have installed the project dependencies
     When I execute the kedro command "build-docs"

features/build_reqs.feature

@@ -3,7 +3,7 @@ Feature: build-reqs target in new project
   @fresh_venv
   Scenario: Execute build-reqs target
     Given I have prepared a config file
-    And I have run a non-interactive kedro new with starter
+    And I have run a non-interactive kedro new with starter "default"
     And I have updated kedro requirements
     And I have executed the kedro command "build-reqs"
     When I add scrapy>=1.7.3 to the requirements

features/info.feature

@@ -1,7 +1,7 @@
 Feature: Run kedro info
     Background:
         Given I have prepared a config file
-        And I have run a non-interactive kedro new with starter
+    And I have run a non-interactive kedro new with starter "default"
 
     Scenario: Plugins are installed and detected by kedro info
         Given I have installed the test plugin

features/ipython.feature

@@ -2,7 +2,7 @@ Feature: IPython target in new project
 
   Scenario: Execute ipython target
     Given I have prepared a config file
-    And I have run a non-interactive kedro new with starter
+    And I have run a non-interactive kedro new with starter "default"
     When I execute the kedro command "ipython"
     Then I should get a message including "An enhanced Interactive Python"
     And I should get a message including "Kedro project project-dummy"

features/jupyter.feature

@@ -2,7 +2,7 @@ Feature: Jupyter targets in new project
 
   Background:
     Given I have prepared a config file
-    And I have run a non-interactive kedro new with starter
+    And I have run a non-interactive kedro new with starter "default"
 
   Scenario: Execute jupyter-notebook target
     When I execute the kedro jupyter command "notebook --no-browser"

features/load_context.feature

@@ -1,7 +1,7 @@
 Feature: Custom Kedro project
     Background:
         Given I have prepared a config file
-        And I have run a non-interactive kedro new with starter
+    And I have run a non-interactive kedro new with starter "default"
 
     Scenario: Update the source directory to be nested
         When I move the package to "src/nested"

features/new_project.feature

@@ -1,13 +1,19 @@
 Feature: New Kedro project
+  Background:
+    Given I have prepared a config file
 
   Scenario: Create a new kedro project without example code
-    Given I have prepared a config file
     When I run a non-interactive kedro new without starter
     Then the expected project directories and files should be created
     And the pipeline should contain no nodes
 
   Scenario: Create a new kedro project with example code
-    Given I have prepared a config file
-    When I run a non-interactive kedro new with starter
+    When I run a non-interactive kedro new with starter "default"
+    Then the expected project directories and files should be created
+    And the pipeline should contain nodes
+
+  Scenario: Plugins are installed and create a new kedro project with custom plugin starter
+    Given I have installed the test plugin
+    When I run a non-interactive kedro new with starter "test_plugin_starter"
     Then the expected project directories and files should be created
     And the pipeline should contain nodes

features/package.feature

@@ -2,7 +2,7 @@ Feature: Package target in new project
 
   Background:
     Given I have prepared a config file
-    And I have run a non-interactive kedro new with starter
+    And I have run a non-interactive kedro new with starter "default"
     And I have installed the project dependencies
 
   @fresh_venv

features/run.feature

@@ -6,14 +6,14 @@ Feature: Run Project
     Local environment should be used by default when no env option is specified.
 
     Given I have prepared a config file
-    And I have run a non-interactive kedro new with starter
+    And I have run a non-interactive kedro new with starter "default"
     When I execute the kedro command "run"
     Then I should get a successful exit code
     And the logs should show that 4 nodes were run
 
   Scenario: Run parallel runner with default python entry point with example code
     Given I have prepared a config file
-    And I have run a non-interactive kedro new with starter
+    And I have run a non-interactive kedro new with starter "default"
     When I execute the kedro command "run --runner=ParallelRunner"
     Then I should get a successful exit code
     And the logs should show that "split_data" was run
@@ -30,23 +30,23 @@ Feature: Run Project
 
   Scenario: Run kedro run with config file
     Given I have prepared a config file
-    And I have run a non-interactive kedro new with starter
+    And I have run a non-interactive kedro new with starter "default"
     And I have prepared a run_config file with config options
     When I execute the kedro command "run --config run_config.yml"
     Then I should get a successful exit code
     And the logs should show that 1 nodes were run
 
   Scenario: Run kedro run with config file and override option
     Given I have prepared a config file
-    And I have run a non-interactive kedro new with starter
+    And I have run a non-interactive kedro new with starter "default"
     And I have prepared a run_config file with config options
     When I execute the kedro command "run --config run_config.yml --pipeline __default__"
     Then I should get a successful exit code
     And the logs should show that 4 nodes were run
 
   Scenario: Run kedro run with extra parameters
     Given I have prepared a config file
-    And I have run a non-interactive kedro new with starter
+    And I have run a non-interactive kedro new with starter "default"
     When I execute the kedro command "run --params extra1:1,extra2:value2"
     Then I should get a successful exit code
     And the logs should show that 4 nodes were run

features/starter.feature

@@ -0,0 +1,9 @@
+Feature: List Kedro Starters
+
+  Scenario: List all starters with custom starters from plugin
+    Given I have prepared a config file
+    And I have installed the test plugin
+    And I have run a non-interactive kedro new with starter "default"
+    When I execute the kedro command "starter list"
+    Then I should get a successful exit code
+    And I should get a message including "test_plugin_starter"

features/steps/cli_steps.py

@@ -222,19 +222,21 @@ def add_test_jupyter_nb(context):
         test_nb_fh.write(TEST_JUPYTER_ORG)
 
 
-@given("I have run a non-interactive kedro new with starter")
-@when("I run a non-interactive kedro new with starter")
-def create_project_with_starter(context):
+@given('I have run a non-interactive kedro new with starter "{starter}"')
+@when('I run a non-interactive kedro new with starter "{starter}"')
+def create_project_with_starter(context, starter):
     """Behave step to run kedro new given the config I previously created."""
-    starter_dir = Path(__file__).parent / "test_starter"
+    if starter == "default":
+        starter = Path(__file__).parent / "test_starter"
+
     res = run(
         [
             context.kedro,
             "new",
             "-c",
             str(context.config_file),
             "--starter",
-            str(starter_dir),
+            str(starter),
         ],
         env=context.env,
         cwd=context.temp_dir,
@@ -468,7 +470,7 @@ def check_correct_nodes_run(context, node):
         "Expected the following message segment to be printed on stdout: "
         f"{expected_log_line},\nbut got {stdout}"
     )
-    assert expected_log_line in info_log.read_text()
+    assert expected_log_line in info_log.read_text(), info_log.read_text()
 
 
 @then("I should get a successful exit code")

features/steps/test_plugin/plugin.py

@@ -1,6 +1,7 @@
 """Dummy plugin with simple hook implementations."""
 import logging
 
+from kedro.framework.cli.starters import KedroStarterSpec
 from kedro.framework.hooks import hook_impl
 
 logger = logging.getLogger(__name__)
@@ -11,8 +12,16 @@ class MyPluginHook:
     @hook_impl
     def after_catalog_created(
         self, catalog
-    ):  # pylint: disable=unused-argument,no-self-use
+    ):  # pylint: disable=unused-argument, no-self-use
         logger.info("Reached after_catalog_created hook")
 
 
+starters = [
+    KedroStarterSpec(
+        "test_plugin_starter",
+        template_path="https://github.com/kedro-org/kedro-starters/",
+        directory="pandas-iris",
+    )
+]
+
 hooks = MyPluginHook()

features/steps/test_plugin/setup.py

@@ -3,7 +3,10 @@
 setup(
     name="test_plugin",
     version="0.1",
-    description="Dummy plugin with hook implementations",
+    description="Dummy plugin with hook implementations and custom starters",
     packages=find_packages(),
-    entry_points={"kedro.hooks": ["test_plugin = plugin:hooks"]},
+    entry_points={
+        "kedro.hooks": ["test_plugin = plugin:hooks"],
+        "kedro.starters": ["starter =  plugin:starters"],
+    },
 )

features/test.feature

@@ -2,7 +2,7 @@ Feature: Test target in new project
 
   Background:
     Given I have prepared a config file
-    And I have run a non-interactive kedro new with starter
+    And I have run a non-interactive kedro new with starter "default"
 
   Scenario: Execute successful test in new project
     When I execute the kedro command "test"

kedro/framework/cli/cli.py

@@ -11,7 +11,6 @@
 from typing import Sequence
 
 import click
-import importlib_metadata
 
 # pylint: disable=unused-import
 import kedro.config.default_logger  # noqa
@@ -29,6 +28,7 @@
     ENTRY_POINT_GROUPS,
     CommandCollection,
     KedroCliError,
+    _get_entry_points,
     load_entry_points,
 )
 from kedro.framework.startup import _is_project, bootstrap_project
@@ -67,8 +67,8 @@ def info():
 
     plugin_versions = {}
     plugin_entry_points = defaultdict(set)
-    for plugin_entry_point, group in ENTRY_POINT_GROUPS.items():
-        for entry_point in importlib_metadata.entry_points().select(group=group):
+    for plugin_entry_point in ENTRY_POINT_GROUPS:
+        for entry_point in _get_entry_points(plugin_entry_point):
             module_name = entry_point.module.split(".")[0]
             plugin_versions[module_name] = entry_point.dist.version
             plugin_entry_points[module_name].add(plugin_entry_point)

kedro/framework/cli/micropkg.py

@@ -255,7 +255,7 @@ def _package_micropkgs_from_manifest(metadata: ProjectMetadata) -> None:
 @click.pass_obj  # this will pass the metadata as first argument
 def package_micropkg(
     metadata: ProjectMetadata, module_path, env, alias, destination, all_flag
-):  # pylint: disable=too-many-arguments
+):
     """Package up a modular pipeline or micro-package as a Python source distribution."""
     if not module_path and not all_flag:
         click.secho(