0.19.0

🚀 Major Features and improvements

• Dropped Python 3.7 support.
• Introduced project tools and example to the kedro new CLI flow.
• The new spaceflights starters, spaceflights-pandas, spaceflights-pandas-viz, spaceflights-pyspark, and spaceflights-pyspark-viz can be used with the kedro new command with the --starter flag.
• Added the --conf-source option to %reload_kedro, allowing users to specify a source for project configuration.
• Added the functionality to choose a merging strategy for config files loaded with OmegaConfigLoader.
• Modified the mechanism of importing datasets, raise more explicit error when dependencies are missing.
• Added validation for configuration file used to override run commands via the CLI.
• Moved the default environment base and local from config loader to _ProjectSettings. This enables the use of config loader as a standalone class without affecting existing Kedro Framework users.

🪲 Bug fixes and other changes

• Added a new field tools to pyproject.toml when a project is created.
• Reduced spaceflights data to minimise waiting times during tutorial execution.
• Added validation to node tags to be consistent with node names.
• Removed pip-tools as a dependency.
• Accepted path-like filepaths more broadly for datasets.

💥 Breaking changes

• Removed ConfigLoader and TemplatedConfigLoader.
• Removed kedro.extras.datasets and tests (use kedro-datasets instead)
• Removed PartitionedDataset and IncrementalDataset from kedro.io (import them from kedro-datasets instead)
• logging is removed from OmegaConfigLoader in favour of the environment variable KEDRO_LOGGING_CONFIG.
• Removed support for defining the layer attribute at top-level within DataCatalog.
• Renamed data_set and DataSet to dataset and Dataset everywhere.
• Removed the create_default_data_set() method in the Runner in favour of using dataset factories to create default dataset instances.
• The default project template now has only one pyproject.toml at the root of the project (containing both the packaging metadata and the Kedro build config).

✍️ Documentation changes

• Added new top navigation to easily switch between Framework, Viz, and Datasets.
• Added new search-as-you-type to improve the search experience.

New Contributors

• @MinuraPunchihewa made their first contribution in #3115
• @mustious made their first contribution in #3181
• @JayOaks made their first contribution in #3239
• @adamkells made their first contribution in #3203
• @HKABIG made their first contribution in #3270
• @pdave34 made their first contribution in #3213
• @hermlon made their first contribution in #3303

Full Changelog: 0.18.14...0.19.0

🚨 If you are upgrading from Kedro 0.18, have a look at the migration guide.

We welcome every community contribution, large or small. See what we're working on now and report bugs or suggest future features.
Until next time,
The Kedro Team 💛

0.18.14

Release 0.18.14

Major features and improvements

• Allowed using of custom cookiecutter templates for creating pipelines with --template flag for kedro pipeline create or via template/pipeline folder.
• Allowed overriding of configuration keys with runtime parameters using the runtime_params resolver with OmegaConfigLoader.

Bug fixes and other changes

• Updated dataset factories to resolve nested catalog config properly.
• Updated OmegaConfigLoader to handle paths containing dots outside of conf_source.
• Made settings.py optional.

Documentation changes

• Added documentation to clarify execution order of hooks.
• Added a notebook example for spaceflights to illustrate how to incrementally add Kedro features.
• Moved documentation for the standalone-datacatalog starter into its README file.
• Added new documentation about deploying a Kedro project with Amazon EMR.
• Added new documentation about how to publish a Kedro-Viz project to make it shareable.
• New TSC members added to the page and the organisation of each member is also now listed.
• Plus some minor bug fixes and changes across the documentation.

Upcoming deprecations for Kedro 0.19.0

• All dataset classes will be removed from the core Kedro repository (kedro.extras.datasets). Install and import them from the kedro-datasets package instead.
• All dataset classes ending with DataSet are deprecated and will be removed in Kedro 0.19.0 and kedro-datasets 2.0.0. Instead, use the updated class names ending with Dataset.
• The starters pandas-iris, pyspark-iris, pyspark, and standalone-datacatalog are deprecated and will be archived in Kedro 0.19.0.
• PartitionedDataset and IncrementalDataset have been moved to kedro-datasets and will be removed in Kedro 0.19.0. Install and import them from the kedro-datasets package instead.

Community contributions

Many thanks to the following Kedroids for contributing PRs to this release:

• Jason Hite
• IngerMathilde
• Laíza Milena Scheid Parizotto
• Richard
• flpvvvv
• qheuristics
• Miguel Ortiz
• rxm7706
• Iñigo Hidalgo
• harmonys-qb
• Yi Kuang
• Jens Lordén

0.18.13

Release 0.18.13

Major features and improvements

• Added support for Python 3.11. This includes tackling challenges like dependency pinning and test adjustments to ensure a smooth experience. Detailed migration tips are provided below for further context.
• Added new OmegaConfigLoader features:
  • Allowed registering of custom resolvers to OmegaConfigLoader through CONFIG_LOADER_ARGS.
  • Added support for global variables to OmegaConfigLoader.
• Added kedro catalog resolve CLI command that resolves dataset factories in the catalog with any explicit entries in the project pipeline.
• Implemented a flat conf/ structure for modular pipelines, and accordingly, updated the kedro pipeline create and kedro catalog create command.
• Updated new Kedro project template and Kedro starters:
  • Change Kedro starters and new Kedro projects to use OmegaConfigLoader.
  • Converted setup.py in new Kedro project template and Kedro starters to pyproject.toml and moved flake8 configuration
    to dedicated file .flake8.
  • Updated the spaceflights starter to use the new flat conf/ structure.

Bug fixes and other changes

• Updated OmegaConfigLoader to ignore config from hidden directories like .ipynb_checkpoints.

Documentation changes

• Revised the data section to restructure beginner and advanced pages about the Data Catalog and datasets.
• Moved contributor documentation to the GitHub wiki.
• Updated example of using generator functions in nodes.
• Added migration guide from the ConfigLoader and the TemplatedConfigLoader to the OmegaConfigLoader. The ConfigLoader and the TemplatedConfigLoader are deprecated and will be removed in the 0.19.0 release.

Migration Tips for Python 3.11:

• PyTables on Windows: Users on Windows with Python >=3.8 should note we've pinned pytables to 3.8.0 due to compatibility issues.
• Spark Dependency: We've set an upper version limit for pyspark at <3.4 due to breaking changes in 3.4.
• Testing with Python 3.10: The latest moto version now supports parallel test execution for Python 3.10, resolving previous issues.

Breaking changes to the API

Upcoming deprecations for Kedro 0.19.0

• Renamed abstract dataset classes, in accordance with the Kedro lexicon. Dataset classes ending with "DataSet" are deprecated and will be removed in 0.19.0. Note that all of the below classes are also importable from kedro.io; only the module where they are defined is listed as the location.

Type	Deprecated Alias	Location
AbstractDataset	AbstractDataSet	kedro.io.core
AbstractVersionedDataset	AbstractVersionedDataSet	kedro.io.core

• Using the layer attribute at the top level is deprecated; it will be removed in Kedro version 0.19.0. Please move layer inside the metadata -> kedro-viz attributes.

Community contributions

Thanks to Laíza Milena Scheid Parizotto and Jonathan Cohen.

0.18.12

Release 0.18.12

Major features and improvements

• Added dataset factories feature which uses pattern matching to reduce the number of catalog entries.
• Activated all built-in resolvers by default for OmegaConfigLoader except for oc.env.
• Added kedro catalog rank CLI command that ranks dataset factories in the catalog by matching priority.

Bug fixes and other changes

• Consolidated dependencies and optional dependencies in pyproject.toml.
• Made validation of unique node outputs much faster.
• Updated kedro catalog list to show datasets generated with factories.

Documentation changes

• Recommended ruff as the linter and removed mentions of pylint, isort, flake8.

Community contributions

Thanks to Laíza Milena Scheid Parizotto and Chris Schopp.

Breaking changes to the API

Upcoming deprecations for Kedro 0.19.0

• ConfigLoader and TemplatedConfigLoader will be deprecated. Please use OmegaConfigLoader instead.

0.18.11

Release 0.18.11

Major features and improvements

• Added databricks-iris as an official starter.

Bug fixes and other changes

• Reworked micropackaging workflow to use standard Python packaging practices.
• Make kedro micropkg package accept --verbose.

Documentation changes

• Significant improvements to the documentation that covers working with Databricks and Kedro, including a new page for workspace-only development, and a guide to choosing the best workflow for your use case.
• Updated documentation for deploying with Prefect for version 2.0.

0.18.10

Major features and improvements

• Rebrand across all documentation and Kedro assets.
• Added support for variable interpolation in the catalog with the OmegaConfigLoader.

0.18.9

Major features and improvements

• kedro run --params now updates interpolated parameters correctly when using OmegaConfigLoader.
• Added metadata attribute to kedro.io datasets. This is ignored by Kedro, but may be consumed by users or external plugins.
• Added kedro.logging.RichHandler. This replaces the default rich.logging.RichHandler and is more flexible, user can turn off the rich traceback if needed.

Bug fixes and other changes

• OmegaConfigLoader will return a dict instead of DictConfig.
• OmegaConfigLoader does not show a MissingConfigError when the config files exist but are empty.

Documentation changes

• Added documentation for collaborative experiment tracking within Kedro-Viz.
• Revised section on deployment to better organise content and reflect how recently docs have been updated.
• Minor improvements to fix typos and revise docs to align with engineering changes.

Breaking changes to the API

• kedro package does not produce .egg files anymore, and now relies exclusively on .whl files.

Community contributions

Many thanks to the following Kedroids for contributing PRs to this release:

• tomasvanpottelbergh
• https://github.com/debugger24

0.18.8

Major features and improvements

• Added KEDRO_LOGGING_CONFIG environment variable, which can be used to configure logging from the beginning of the kedro process.
• Removed logs folder from the kedro new project template. File-based logging will remain but just be level INFO and above and go to project root instead.

Bug fixes and other changes

• Improvements to Jupyter E2E tests.
• Added full kedro run CLI command to session store to improve run reproducibility using Kedro-Viz experiment tracking.

Documentation changes

• Improvements to documentation about configuration.
• Improvements to Sphinx toolchain including incrementing to use a newer version.
• Improvements to documentation on visualising Kedro projects on Databricks, and additional documentation about the development workflow for Kedro projects on Databricks.
• Updated Technical Steering Committee membership documentation.
• Revised documentation section about linting and formatting and extended to give details of flake8 configuration.
• Updated table of contents for documentation to reduce scrolling.
• Expanded FAQ documentation.
• Added a 404 page to documentation.
• Added deprecation warnings about the removal of kedro.extras.datasets.

0.18.7

Release 0.18.7

Major features and improvements

• Added new Kedro CLI kedro jupyter setup to setup Jupyter Kernel for Kedro.
• kedro package now includes the project configuration in a compressed tar.gz file.
• Added functionality to the OmegaConfigLoader to load configuration from compressed files of zip or tar format. This feature requires fsspec>=2023.1.0.
• Significant improvements to on-boarding documentation that covers setup for new Kedro users. Also some major changes to the spaceflights tutorial to make it faster to work through. We think it's a better read. Tell us if it's not.

Bug fixes and other changes

• Added a guide and tooling for developing Kedro for Databricks.
• Implement missing dict-like interface for _ProjectPipeline.

0.18.6

Release 0.18.6

Bug fixes and other changes

• Fixed bug that didn't allow to read or write datasets with s3a or s3n filepaths
• Fixed bug with overriding nested parameters using the --params flag
• Fixed bug that made session store incompatible with Kedro-Viz experiment tracking

Migration guide from Kedro 0.18.5 to 0.18.6

A regression introduced in Kedro version 0.18.5 caused the Kedro-Viz console to fail to show experiment tracking correctly. If you experienced this issue, you will need to:

• upgrade to Kedro version 0.18.6
• delete any erroneous session entries created with Kedro 0.18.5 from your session_store.db stored at <project-path>/data/session_store.db.

Thanks to Kedroids tomohiko kato, tsanikgr and maddataanalyst for very detailed reports about the bug.