Release 0.15.2

.github/PULL_REQUEST_TEMPLATE.md

@@ -17,6 +17,6 @@ What testing strategies have you used?
 - [ ] Read the [contributing](/CONTRIBUTING.md) guidelines
 - [ ] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
 - [ ] Updated the documentation to reflect the code changes
-- [ ] Added new entries to the `RELEASE.md` file
+- [ ] Added a description of this change and added my name to the list of supporting contributions in the [`RELEASE.md`](/RELEASE.md) file
 - [ ] Added tests to cover my changes
 - [ ] Assigned myself to the PR

.pre-commit-config.yaml

@@ -4,116 +4,116 @@
 default_stages: [commit, manual]
 
 repos:
--   repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v2.2.3
-    hooks:
-    -   id: trailing-whitespace
-        stages: [commit, manual]
-    -   id: end-of-file-fixer
-        stages: [commit, manual]
-    -   id: check-yaml  # Checks yaml files for parseable syntax.
-        exclude: "^kedro/template/"
-    -   id: check-json  # Checks json files for parseable syntax.
-    -   id: check-added-large-files
-    -   id: check-case-conflict  # Check for files that would conflict in case-insensitive filesystems
-    -   id: check-merge-conflict  # Check for files that contain merge conflict strings.
-    -   id: debug-statements  # Check for debugger imports and py37+ `breakpoint()` calls in python source.
-        exclude: "^kedro/template/"
-    -   id: detect-private-key  # Detects the presence of private keys
-    -   id: requirements-txt-fixer  # Sorts entries in requirements.txt
-    -   id: flake8
-        exclude: "^kedro/template/"
+    - repo: https://github.com/pre-commit/pre-commit-hooks
+      rev: v2.2.3
+      hooks:
+          - id: trailing-whitespace
+            stages: [commit, manual]
+          - id: end-of-file-fixer
+            stages: [commit, manual]
+          - id: check-yaml # Checks yaml files for parseable syntax.
+            exclude: "^kedro/template/"
+          - id: check-json # Checks json files for parseable syntax.
+          - id: check-added-large-files
+          - id: check-case-conflict # Check for files that would conflict in case-insensitive filesystems
+          - id: check-merge-conflict # Check for files that contain merge conflict strings.
+          - id: debug-statements # Check for debugger imports and py37+ `breakpoint()` calls in python source.
+            exclude: "^kedro/template/"
+          - id: detect-private-key # Detects the presence of private keys
+          - id: requirements-txt-fixer # Sorts entries in requirements.txt
+          - id: flake8
+            exclude: "^kedro/template/"
 
--   repo: https://github.com/pre-commit/mirrors-isort
-    rev: v4.3.21
-    hooks:
-    -   id: isort
-        exclude: "^kedro/template/"
+    - repo: https://github.com/pre-commit/mirrors-isort
+      rev: v4.3.21
+      hooks:
+          - id: isort
+            exclude: "^kedro/template/"
 
--   repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v0.720
-    hooks:
-        - id: mypy
-          args: [--allow-redefinition, --ignore-missing-imports]
-          exclude: |
-            (?x)(
-              ^kedro/template/|
-              ^docs/
-            )
+    - repo: https://github.com/pre-commit/mirrors-mypy
+      rev: v0.720
+      hooks:
+          - id: mypy
+            args: [--allow-redefinition, --ignore-missing-imports]
+            exclude: |
+                (?x)(
+                  ^kedro/template/|
+                  ^docs/
+                )
 
--   repo: local
-    hooks:
-    # It's impossible to specify per-directory configuration, so we just run it many times.
-    # https://github.com/PyCQA/pylint/issues/618
-    # The first set of pylint checks if for local pre-commit, it only runs on the files changed.
-    -   id: pylint-quick-kedro
-        name: "Quick PyLint on kedro/*"
-        language: system
-        types: [file, python]
-        files: ^kedro/
-        exclude: ^kedro/template/
-        entry: pylint -j0 --disable=unnecessary-pass
-        stages: [commit]
-    -   id: pylint-quick-features
-        name: "Quick PyLint on features/*"
-        language: system
-        types: [file, python]
-        files: ^features/
-        entry: pylint -j0 --disable=missing-docstring,no-name-in-module
-        stages: [commit]
-    -   id: pylint-quick-extras
-        name: "Quick PyLint on extras/*"
-        language: system
-        types: [file, python]
-        files: ^extras/
-        entry: pylint -j0
-        stages: [commit]
-    -   id: pylint-quick-tests
-        name: "Quick PyLint on tests/*"
-        language: system
-        types: [file, python]
-        files: ^tests/
-        entry: pylint -j0 --disable=missing-docstring,redefined-outer-name,no-self-use,invalid-name
-        stages: [commit]
+    - repo: local
+      hooks:
+          # It's impossible to specify per-directory configuration, so we just run it many times.
+          # https://github.com/PyCQA/pylint/issues/618
+          # The first set of pylint checks if for local pre-commit, it only runs on the files changed.
+          - id: pylint-quick-kedro
+            name: "Quick PyLint on kedro/*"
+            language: system
+            types: [file, python]
+            files: ^kedro/
+            exclude: ^kedro/template/
+            entry: pylint --disable=unnecessary-pass
+            stages: [commit]
+          - id: pylint-quick-features
+            name: "Quick PyLint on features/*"
+            language: system
+            types: [file, python]
+            files: ^features/
+            entry: pylint --disable=missing-docstring,no-name-in-module
+            stages: [commit]
+          - id: pylint-quick-extras
+            name: "Quick PyLint on extras/*"
+            language: system
+            types: [file, python]
+            files: ^extras/
+            entry: pylint
+            stages: [commit]
+          - id: pylint-quick-tests
+            name: "Quick PyLint on tests/*"
+            language: system
+            types: [file, python]
+            files: ^tests/
+            entry: pylint --disable=missing-docstring,redefined-outer-name,no-self-use,invalid-name
+            stages: [commit]
 
-    # The same pylint checks, but running on all files. It's for manual run with `make lint`
-    -   id: pylint-kedro
-        name: "PyLint on kedro/*"
-        language: system
-        pass_filenames: false
-        stages: [manual]
-        entry: pylint -j0 --disable=unnecessary-pass kedro
-    -   id: pylint-features
-        name: "PyLint on features/*"
-        language: system
-        pass_filenames: false
-        stages: [manual]
-        entry: pylint -j0 --disable=missing-docstring,no-name-in-module features
-    -   id: pylint-extras
-        name: "PyLint on extras/*"
-        language: system
-        pass_filenames: false
-        stages: [manual]
-        entry: pylint -j0 extras
-    -   id: pylint-tests
-        name: "PyLint on tests/*"
-        language: system
-        pass_filenames: false
-        stages: [manual]
-        entry: pylint -j0 --disable=missing-docstring,redefined-outer-name,no-self-use,invalid-name tests
-    # We need to make some exceptions for 3.5, that's why it's a custom runner.
-    -   id: black
-        name: "Black"
-        language: system
-        pass_filenames: false
-        entry: python -m tools.min_version 3.6 "pip install black" "black kedro extras features tests"
-    -   id: legal
-        name: "Licence check"
-        language: system
-        pass_filenames: false
-        entry: make legal
-    -   id: imports
-        name: "Import Linter"
-        language: system
-        pass_filenames: false
-        entry: python -m tools.min_version 3.6 "pip install import-linter" lint-imports
+          # The same pylint checks, but running on all files. It's for manual run with `make lint`
+          - id: pylint-kedro
+            name: "PyLint on kedro/*"
+            language: system
+            pass_filenames: false
+            stages: [manual]
+            entry: pylint --disable=unnecessary-pass kedro
+          - id: pylint-features
+            name: "PyLint on features/*"
+            language: system
+            pass_filenames: false
+            stages: [manual]
+            entry: pylint --disable=missing-docstring,no-name-in-module features
+          - id: pylint-extras
+            name: "PyLint on extras/*"
+            language: system
+            pass_filenames: false
+            stages: [manual]
+            entry: pylint extras
+          - id: pylint-tests
+            name: "PyLint on tests/*"
+            language: system
+            pass_filenames: false
+            stages: [manual]
+            entry: pylint --disable=missing-docstring,redefined-outer-name,no-self-use,invalid-name tests
+          # We need to make some exceptions for 3.5, that's why it's a custom runner.
+          - id: black
+            name: "Black"
+            language: system
+            pass_filenames: false
+            entry: python -m tools.min_version 3.6 "pip install black" "black kedro extras features tests"
+          - id: legal
+            name: "Licence check"
+            language: system
+            pass_filenames: false
+            entry: make legal
+          - id: imports
+            name: "Import Linter"
+            language: system
+            pass_filenames: false
+            entry: python -m tools.min_version 3.6 "pip install import-linter" lint-imports

.pylintrc

@@ -50,7 +50,7 @@ confidence=
 # --enable=similarities". If you want to run only the classes checker, but have
 # no Warning level messages displayed, use"--disable=all --enable=classes
 # --disable=W"
-disable=ungrouped-imports,bad-continuation
+disable=ungrouped-imports,bad-continuation,duplicate-code
 
 # Enable the message, report, category or checker with the given id(s). You can
 # either give multiple identifier separated by comma (,) or put this option

CONTRIBUTING.md

@@ -35,7 +35,7 @@ If you're unsure where to begin contributing to Kedro, please start by looking t
 We focus on three areas for contribution: `core`, [`contrib`](/kedro/contrib/) or `plugin`:
 - `core` refers to the primary Kedro library
 - [`contrib`](/kedro/contrib/) refers to features that could be added to `core` that do not introduce too many depencies or require new Kedro CLI commands to be created e.g. adding a new dataset to the `io` data management module
-- [`plugin`](https://kedro.readthedocs.io/en/latest/04_user_guide/09_developing_plugins.html) refers to new functionality that requires a Kedro CLI command e.g. adding in Airflow functionality
+- [`plugin`](https://kedro.readthedocs.io/en/latest/04_user_guide/10_developing_plugins.html) refers to new functionality that requires a Kedro CLI command e.g. adding in Airflow functionality
 
 Typically, we only accept small contributions for the `core` Kedro library but accept new features as `plugin`s or additions to the [`contrib`](/kedro/contrib/) module. We regularly review [`contrib`](/kedro/contrib/) and may migrate modules to `core` if they prove to be essential for the functioning of the framework or if we believe that they are used by most projects.
 
@@ -101,7 +101,7 @@ You can add new work to `contrib` if you do not need to create a new Kedro CLI c
 
 ## `plugin` contribution process
 
-See the [`plugin` development documentation](https://kedro.readthedocs.io/en/latest/04_user_guide/09_developing_plugins.html) for guidance on how to design and develop a Kedro `plugin`.
+See the [`plugin` development documentation](https://kedro.readthedocs.io/en/latest/04_user_guide/10_developing_plugins.html) for guidance on how to design and develop a Kedro `plugin`.
 
 ## CI / CD and running checks locally
 To run E2E tests you need to install the test requirements which includes `behave`.

RELEASE.md

@@ -1,3 +1,30 @@
+# Release 0.15.2
+
+## Major features and improvements
+* Added `--load-version`, a `kedro run` argument that allows you run the pipeline with a particular load version of a dataset.
+* Support for modular pipelines in `src/`, break the pipeline into isolated parts with reusability in mind.
+* Support for multiple pipelines, an ability to have multiple entry point pipelines and choose one with `kedro run --pipeline NAME`.
+* Added a `MatplotlibWriter` dataset in `contrib` for saving Matplotlib images.
+* An ability to template/parameterize configuration files with `kedro.contrib.config.TemplatedConfigLoader`.
+* Parameters are exposed as a context property for ease of access in iPython / Jupyter Notebooks with `context.params`.
+* Added `max_workers` parameter for ``ParallelRunner``.
+
+## Bug fixes and other changes
+* Users will override the `_get_pipeline` abstract method in `ProjectContext(KedroContext)` in `run.py` rather than the `pipeline` abstract property. The `pipeline` property is not abstract anymore.
+* Improved an error message when versioned local dataset is saved and unversioned path already exists.
+* Add `catalog` global variable to `00-kedro-init.py`, allowing you to load datasets with `catalog.load()`.
+* Enabled tuples to be returned from a node.
+* Disallowed the ``ConfigLoader`` loading the same file more than once, and deduplicated the `conf_paths` passed in.
+* Added a `--open` flag to `kedro build-docs` that opens the documentation on build.
+* Updated the ``Pipeline`` representation to include name of the pipeline, also making it readable as a context property.
+* `kedro.contrib.io.pyspark.SparkDataSet` and `kedro.contrib.io.azure.CSVBlobDataSet` now support versioning.
+
+## Breaking changes to the API
+* `KedroContext.run()` no longer accepts `catalog` and `pipeline` arguments.
+
+## Thanks for supporting contributions
+[Deepyaman Datta](https://github.com/deepyaman), [Luciano Issoe](https://github.com/Lucianois), [Joost Duisters](https://github.com/JoostDuisters), [Zain Patel](https://github.com/mzjp2), [William Ashford](https://github.com/williamashfordQB), [Karlson Lee](https://github.com/i25959341)
+
 # Release 0.15.1
 
 ## Major features and improvements
@@ -15,7 +42,7 @@
 * Updated documentation in `03_configuration` regarding how to modify the configuration path.
 * Documented the architecture of Kedro showing how we think about library, project and framework components.
 * `extras/kedro_project_loader.py` renamed to `extras/ipython_loader.py` and now runs any IPython startup scripts without relying on the Kedro project structure.
-* Fixed TypeError when validating partial function's signature
+* Fixed TypeError when validating partial function's signature.
 * After a node failure during a pipeline run, a resume command will be suggested in the logs. This command will not work if the required inputs are MemoryDataSets.
 
 ## Breaking changes to the API

docs/conf.py

@@ -86,7 +86,13 @@
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path .
-exclude_patterns = ["**.ipynb_checkpoints", "_templates", "modules.rst", "source"]
+exclude_patterns = [
+    "**.ipynb_checkpoints",
+    "_templates",
+    "modules.rst",
+    "source",
+    "README.md",
+]
 
 
 # The name of the Pygments (syntax highlighting) style to use.

docs/source/03_tutorial/02_tutorial_template.md

@@ -27,4 +27,4 @@ The project template has a default configuration, but you should reconfigure it
   - To do this in the terminal, type the following from within the project's root directory: `mv ./conf/base/credentials.yml ./conf/local/`.
 * [optional] Add in any credentials to `conf/local/credentials.yml` that you would need to load specific data sources like usernames and passwords. Some examples are given within the file to illustrate how you store credentials.
 * [optional] Set up local configuration. Additional information can be found in the section on [configuration](../04_user_guide/03_configuration.md) in the user guide.
-* [optional] Set up [logging](../04_user_guide/06_logging.md).
+* [optional] Set up [logging](../04_user_guide/07_logging.md).