Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Allow for customization of pipeline templates (#2701)
* Remove non-user-facing release notes about pip pin (#2854) Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Allow for customization of pipeline templates Controlled via `settings.PIPELINE_TEMPLATE_PATH`. Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Update lookup based on feedback Instead of lookups on `settings.py` just statically look for a `templates` directory in the project root. If this folder is present it will override the default global template, otherwise fall back to the previous global default. Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Add CLI flag for pipeline creation template -t/--template Precedence for template_path is: command line > project templates/pipeline dir > global default pipeline create now will also print the template folder used. Signed-off-by: Jason Hite <jasonmhite@gmail.com> * First basic unit test TODO is testing the CLI flag too, and maybe testing that the cleanup works with custom templates. Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Add test to check overriding template path on the CLI Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Linter formatting fixes Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Release 0.18.12 (#2871) * bump version and prepare for new release Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update RELEASE.md Added community contributions * add DeltaTableDataSet to rst Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update setup.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update setup.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update setup.py Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * temporarily use the fixed kedro-plugins branch Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update setup.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * test docsbuild on merels branch Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * use kedro-datasets new release Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> --------- Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Fix broken link to Prefect website in deploy guide (#2885) Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Clarify the <micropkg_name> argument to kedro micropkg package (#2835) Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Allow registering of custom resolvers to `OmegaConfigLoader` (#2869) * Allow registering of custom resolvers to OCL Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Complete doc string Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add test for overwritten resolvers Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update test for overwritten resolvers Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Remove replace=True by default Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update release notes Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update release notes Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add debug level log for registering new resolver Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Document the use of custom resolvers with `OmegaConfigLoader` (#2896) * Update docs for custom resolvers for OCL Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update FAQ with correct link Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Fix typo Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> * Add note for oc.env Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Suggestions from code review Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Update kedro pipeline create to use new /conf file structure (#2856) * add parameters_pipelineNane.yml template in the conf folder Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * del yml template from parameters folder Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * modify delete pipeline command to use correct path to yml files Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * modify test_pipeline move parameters foldername to the yml filename Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * collect configs to package not only from parameters folder, but from core conf folder also Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * move parameters foldername to the yml filename, remove nesting Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * move parameters foldername to the yml filename, remove nesting Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * Add changes to RELEASE.md Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * Update RELEASE.md Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> * Update RELEASE.md, change to release 0.18.13 Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * save old structure for 'pipeline delete' command for backward compatibility Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * modify test_pipeline for 'pipeline delete' backward compability Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> --------- Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Update kedro catalog create to use new /conf structure (#2884) * modify test_catalog, remove nesting in the catalog folder Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * move catalog foldername to the yml filename, remove nesting Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * Update RELEASE.md Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * Modify docs, fix broken link Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> --------- Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Add migration steps for `ConfigLoader` to `OmegaConfigLoader` (#2887) * Add migration steps for CL to OCL Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Fix #2498 Adding logging issue 2498 (#2842) * Fixing typo n on line 114 file /docs/source/nodes_and_pipelines/run_a_pipeline.md from issue #2607 Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com> * Removed redundant packages at lines 66, 67 and 68 on file kedro/docs/source/nodes_and_pipelines/run_a_pipeline.md, packages: from kedro.io import AbstractDataSet, DataCatalog, MemoryDataSet, from kedro.pipeline import Pipeline and from kedro.runner.runner import AbstractRunner Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com> * Add logger object for logging when computing micropackage version Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com> * Added new format to the logging Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com> --------- Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Try only trigger docs build when release notes updated (#2907) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Add Python 3.11 support to Kedro (#2851) * add 3.11 Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update setup.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * limit number of python version tests Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * fix setup.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * delta-spark version Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * try original setup Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * remove 3.11 from circleci Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update setup.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update setup.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * pin matplotlib to 3.7.2 for python 3.11 Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update test_import.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * add noks test adjustments Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update setup.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update setup.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * replace eq with is for multi_catalog Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * minor fixes Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * remove matplotlib 3.11 pin Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * matplotlib pin for 3.11 Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * run original matplotlib pin with ubuntu only Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * run circleci Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * pin spark < 3.4 as its breaking Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * spark datasets revert earlier changes back to original Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * delta Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * remove all windows tests from GHA Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * skip coverage as not needed Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Checking only WIN tests (will revert) Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update unit-tests.yml Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * undo previous commit Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * add msbuild Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * add ubuntu tests back in and remove 3.11 tests from circleci add in msbuild for the other tests Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * set hdf5 disable check and lint Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * pin tables for 3.8 Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * pin 3.8 tables Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * lint Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * remove 3.11 completely from circleci and tidy up Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * add 3.11 to pyarrow pin for docs Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * undo previous commit Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * update rtd to python 3.11 Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * try 3.10 on rtd Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * fix outdated pylint to ruff Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * changes based on review Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * add mock for pyarrow and deltalake for docs Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * undo previous commit Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update setup.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update .readthedocs.yml Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update conf.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * changes based on review Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * fix comments Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * test running 3.10 parallel Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Add to release notes and update badges on docs Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Add migration tips Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> --------- Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Revise FAQs and README (#2909) * Revise FAQs and README Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add back the data layers FAQ as I've no idea where else it fits Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * minor changes from review Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> --------- Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Update Generator example (#2858) * update release note Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * update to binary mode when opening file Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Simplify the implementation of Dataset Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * simplify the example - wip Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Update the nodes Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * update example Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Update example Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Apply suggestions from code review Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Apply suggestions from code review Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Apply suggestions from code review Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Apply suggestions from code review Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * add more instruction to guide user to create the custom dataset Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * update the reference from `pipeline.py` -> `nodes.py` Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Collapse the nodes.py function into a expandable section Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Update nodes.md * Update nodes.md --------- Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * update docs to reflect change to /conf file structure (#2913) * update docs to reflect change to /conf file structure Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * update docs, folder structure diagrams Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> --------- Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Change CONTRIBUTING.md file based on PR #2894 (#2914) Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Move contribution guidelines from CONTRIBUTING.md to the Wiki (#2894) * Add drafts for new contributing.md and wiki page Signed-off-by: lrcouto <laurarccouto@gmail.com> * Add contribution details to wiki page draft Signed-off-by: lrcouto <laurarccouto@gmail.com> * Remove contributing-to-Kedro.md draft from the branch Signed-off-by: lrcouto <laurarccouto@gmail.com> * Fix broken link to Prefect website in deploy guide (#2885) Signed-off-by: lrcouto <laurarccouto@gmail.com> * Clarify the <micropkg_name> argument to kedro micropkg package (#2835) Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: lrcouto <laurarccouto@gmail.com> * Allow registering of custom resolvers to `OmegaConfigLoader` (#2869) * Allow registering of custom resolvers to OCL Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Complete doc string Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add test for overwritten resolvers Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update test for overwritten resolvers Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Remove replace=True by default Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update release notes Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update release notes Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add debug level log for registering new resolver Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: lrcouto <laurarccouto@gmail.com> * Some adjustments to ordering of text Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: lrcouto <laurarccouto@gmail.com> * Document the use of custom resolvers with `OmegaConfigLoader` (#2896) * Update docs for custom resolvers for OCL Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update FAQ with correct link Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Fix typo Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> * Add note for oc.env Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Suggestions from code review Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: lrcouto <laurarccouto@gmail.com> * Update kedro pipeline create to use new /conf file structure (#2856) * add parameters_pipelineNane.yml template in the conf folder Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * del yml template from parameters folder Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * modify delete pipeline command to use correct path to yml files Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * modify test_pipeline move parameters foldername to the yml filename Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * collect configs to package not only from parameters folder, but from core conf folder also Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * move parameters foldername to the yml filename, remove nesting Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * move parameters foldername to the yml filename, remove nesting Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * Add changes to RELEASE.md Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * Update RELEASE.md Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> * Update RELEASE.md, change to release 0.18.13 Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * save old structure for 'pipeline delete' command for backward compatibility Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * modify test_pipeline for 'pipeline delete' backward compability Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> --------- Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: lrcouto <laurarccouto@gmail.com> * Update CONTRIBUTING.md Signed-off-by: lrcouto <laurarccouto@gmail.com> * Fix broken link Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: lrcouto <laurarccouto@gmail.com> * Update kedro catalog create to use new /conf structure (#2884) * modify test_catalog, remove nesting in the catalog folder Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * move catalog foldername to the yml filename, remove nesting Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * Update RELEASE.md Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> * Modify docs, fix broken link Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> --------- Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Signed-off-by: lrcouto <laurarccouto@gmail.com> * Add migration steps for `ConfigLoader` to `OmegaConfigLoader` (#2887) * Add migration steps for CL to OCL Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: lrcouto <laurarccouto@gmail.com> * Fix #2498 Adding logging issue 2498 (#2842) * Fixing typo n on line 114 file /docs/source/nodes_and_pipelines/run_a_pipeline.md from issue #2607 Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com> * Removed redundant packages at lines 66, 67 and 68 on file kedro/docs/source/nodes_and_pipelines/run_a_pipeline.md, packages: from kedro.io import AbstractDataSet, DataCatalog, MemoryDataSet, from kedro.pipeline import Pipeline and from kedro.runner.runner import AbstractRunner Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com> * Add logger object for logging when computing micropackage version Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com> * Added new format to the logging Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com> --------- Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: lrcouto <laurarccouto@gmail.com> * Try only trigger docs build when release notes updated (#2907) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: lrcouto <laurarccouto@gmail.com> * Add Python 3.11 support to Kedro (#2851) * add 3.11 Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update setup.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * limit number of python version tests Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * fix setup.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * delta-spark version Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * try original setup Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * remove 3.11 from circleci Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update setup.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update setup.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * pin matplotlib to 3.7.2 for python 3.11 Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update test_import.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * add noks test adjustments Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update setup.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update setup.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * replace eq with is for multi_catalog Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * minor fixes Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * remove matplotlib 3.11 pin Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * matplotlib pin for 3.11 Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * run original matplotlib pin with ubuntu only Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * run circleci Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * pin spark < 3.4 as its breaking Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * spark datasets revert earlier changes back to original Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * delta Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * remove all windows tests from GHA Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * skip coverage as not needed Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Checking only WIN tests (will revert) Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update unit-tests.yml Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * undo previous commit Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * add msbuild Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * add ubuntu tests back in and remove 3.11 tests from circleci add in msbuild for the other tests Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * set hdf5 disable check and lint Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * pin tables for 3.8 Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * pin 3.8 tables Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * lint Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * remove 3.11 completely from circleci and tidy up Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * add 3.11 to pyarrow pin for docs Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * undo previous commit Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * update rtd to python 3.11 Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * try 3.10 on rtd Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * fix outdated pylint to ruff Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * changes based on review Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * add mock for pyarrow and deltalake for docs Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * undo previous commit Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update setup.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update .readthedocs.yml Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update conf.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * changes based on review Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * fix comments Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * test running 3.10 parallel Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Add to release notes and update badges on docs Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Add migration tips Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> --------- Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: lrcouto <laurarccouto@gmail.com> * Revise FAQs and README (#2909) * Revise FAQs and README Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add back the data layers FAQ as I've no idea where else it fits Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * minor changes from review Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> --------- Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update Generator example (#2858) * update release note Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * update to binary mode when opening file Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Simplify the implementation of Dataset Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * simplify the example - wip Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Update the nodes Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * update example Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Update example Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Apply suggestions from code review Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Apply suggestions from code review Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Apply suggestions from code review Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Apply suggestions from code review Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * add more instruction to guide user to create the custom dataset Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * update the reference from `pipeline.py` -> `nodes.py` Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Collapse the nodes.py function into a expandable section Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Update nodes.md * Update nodes.md --------- Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: lrcouto <laurarccouto@gmail.com> * Update CONTRIBUTING.md --------- Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com> Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Co-authored-by: Jonathan Cohen <jon@joncohen.dev> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Dmitry Sorokin <40151847+DimedS@users.noreply.github.com> Co-authored-by: Laíza Milena Scheid Parizotto <41296740+laizaparizotto@users.noreply.github.com> Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Remove redundant pages and direct users to wiki (#2917) * Remove redundant pages and direct users to wiki Signed-off-by: lrcouto <laurarccouto@gmail.com> * Lint docs changes Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Few tweaks to index page Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> --------- Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Deprecate abstract "DataSet" in favor of "Dataset" (#2746) * Deprecate abstract "DataSet" in favor of "Dataset" Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Update docs references to abstract "DataSet" class Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Fix `AbstractDataSet` reference to `AbstractDataset` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Update docs/source/conf.py Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Change remaining `_DEPRECATED_ERROR_CLASSES` to `_DEPRECATED_CLASSES`, update type hints Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Update RELEASE.md * Reformat kedro/io/__init__.py with Black and isort Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Remove duplicate imports Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Move imports Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Reorganise and improve the data catalog documentation (#2888) * First drop of newly organised data catalog docs Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * linter Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Added to-do notes Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Afternoon's work in rewriting/reorganising content Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * More changes Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Further changes Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Another chunk of changes Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Final changes Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Revise ordering of pages Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add new CLI commands to dataset factory docs (#2935) * Add changes from #2930 Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Lint Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Apply suggestions from code review Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Make code snippets collapsable Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> --------- Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Bunch of changes from feedback Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * A few more tweaks Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update h1,h2,h3 font sizes Signed-off-by: Tynan DeBold <thdebold@gmail.com> * Add code snippet for using DataCatalog with Kedro config Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Few more tweaks Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/data/data_catalog.md * Upgrade kedro-datasets for docs Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Improve prose Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> --------- Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> Signed-off-by: Tynan DeBold <thdebold@gmail.com> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Ahdra Merali <90615669+AhdraMeraliQB@users.noreply.github.com> Co-authored-by: Tynan DeBold <thdebold@gmail.com> Co-authored-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Add line about Viz to PR template (#2929) Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Add architecture graphic back to docs with revisions (#2916) * Revise FAQs and README Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add back the data layers FAQ as I've no idea where else it fits Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * minor changes from review Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Re-add kedro arch diagram, with revised graphic Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * fix broken anchor Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * fix broken anchor Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/extend_kedro/architecture_overview.md Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> * Update docs/source/extend_kedro/architecture_overview.md Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> * Update docs/source/extend_kedro/architecture_overview.md Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> * Update docs/source/extend_kedro/architecture_overview.md Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> * Changes to architecture page following review Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Change diagram following reivew * Add links to API docs Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Added in type of users * Fix linting error Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> --------- Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Co-authored-by: Yetunde Dada <43755008+yetudada@users.noreply.github.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Add kedro catalog resolve command (#2891) * Implement working draft Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Remove comments Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Remove raw_config Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Use config loader to access catalog config Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Use dict comprehension Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Remove pipeline filtering Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Prevent overwrite or param inclusion Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Trim filepath Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Appease linter Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add test for resolve Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add test to exclude params Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add test for overwrite (not working) Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Fix test Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Remove print Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add changes to RELEASE.md Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Trim unreachable code Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Make helper function private Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Refactor Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add changes to the docs Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Refactor 2 Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Fix docs build Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add suggestions from code review (docs) Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add suggestions from code review (code) Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Rename catalog_config variable to explicit_datasets Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Apply suggestions from code review Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> * Fix mocking in test Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Change test fixture name Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Lint Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> --------- Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Replace "DataSet" with "Dataset" in Markdown files (#2735) * LambdaDataSet->LambdaDataset in .md files Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * MemoryDataSet->MemoryDataset in .md files Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * PartitionedDataSet->PartitionedDataset in .md files Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * IncrementalDataSet->IncrementalDataset in .md files Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * CachedDataSet->CachedDataset in .md files Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * DataSetError->DatasetError in .md files Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * DataSetNotFoundError->DatasetNotFoundError in .md files Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Replace "DataSet" with "Dataset" in Markdown files * Update RELEASE.md * Fix remaining instance of "*DataSet*"->"*Dataset*" Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * `find . -name '*.md' -print0 | xargs -0 sed -i "" "s/\([^A-Za-z]\)DataSet/\1Dataset/g"` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Change non-class instances of Dataset to dataset * Replace any remaining instances of DataSet in docs * Fix a broken link to docs for `PartitionedDataset` --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Update h1,h2,h3 font sizes in the docs pages (#2938) * Update h1,h2,h3 font sizes Signed-off-by: Tynan DeBold <thdebold@gmail.com> * Update h2 font size Signed-off-by: Tynan DeBold <thdebold@gmail.com> --------- Signed-off-by: Tynan DeBold <thdebold@gmail.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Automatically trigger `kedro-starters` release on the release of `kedro` (#2889) * Add GA for triggering starters release Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update token name Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Create issues_metrics.yml (#2814) * Create issues_metrics.yml * Update issues_metrics.yml Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Clean up setuptools and wheel requirements to align with PEP-518 (#2927) Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Clean up `kedro pipeline create` outdated docs (#2945) * remove the README template generate by kedro pipeline create Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * fix docstring for kedro catalog create Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Fix broken link Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * fix broken link Signed-off-by: Nok <nok.lam.chan@quantumblack.com> --------- Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Add globals feature for `OmegaConfigLoader` using a globals resolver (#2921) * Refactor load_and_merge_dir() Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Try adding globals resolver Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Minor change Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add globals resolver Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Revert refactoring Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add test + remove self.globals Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Allow for nested variables in globals Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add documentation Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Typo Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add error message + test Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Apply suggestions from code review Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> * Split test into multiple tests Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Restrict the globals config_patterns Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Release notes Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update docs/source/configuration/advanced_configuration.md Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> * Add helpful error message for keys starting with _ Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Enable setting default value for globals resolver Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Typo Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Move test for keys starting with _ to the top Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Fix cross ref link in docs Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Consolidate two `ruff-pre-commit` entries into one (#2881) * Clean up Ruff configuration in base pyproject.toml Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Consolidate two `ruff-pre-commit` entries into one Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Move Ruff hook to the top, because it's aggressive Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Add `kedro` as `known-first-party` for `isort` run Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Add noqa in docs/source/conf.py Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Fix ruff linter precommit config Signed-off-by: Nok <nok.lam.chan@quantumblack.com> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Co-authored-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Fix typos across the documentation (#2956) * Fix typos across docs Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Capitalisation stuff Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Setup Vale linter as a GHA workflow (#2953) * Add vale GHA Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update condition Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update .github/styles/Kedro/ignore.txt Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * add some styles Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Remove styles for now Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Fix README to show graphics on PyPI (#2961) * Fix for PyPI Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Few more tweaks Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Few more tweaks Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Few more tweaks Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Few more tweaks Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update README.md * Update README.md --------- Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Add some Vale styles (#2963) * add some styles Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Tweaks to styles Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * more tweaks Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Revise some styles Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Revise severity of some styles Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update .github/styles/Kedro/words.yml Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Minor changes to test + release notes + docs Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Update anyconfig requirement from ~=0.10.0 to >=0.10,<0.14 (#2876) Updates the requirements on [anyconfig](https://github.com/ssato/python-anyconfig) to permit the latest version. - [Changelog](https://github.com/ssato/python-anyconfig/blob/next/NEWS) - [Commits](https://github.com/ssato/python-anyconfig/compare/RELEASE_0.10.0...RELEASE_0.13.0) Signed-off-by: Jason Hite <jasonmhite@gmail.com> --- updated-dependencies: - dependency-name: anyconfig dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Move default template to static `pyproject.toml`, take 2 (#2853) * Remove leftover reference to micropkg setup.py Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Replace outdated extras requirement with kedro-datasets Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Remove outdated mention to environment.yml Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Move flake8 configuration to dedicated file Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Replace setup.py with pyproject.toml in default project template Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Adapt link to moving file in docs Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Revert "Replace outdated extras requirement with kedro-datasets" This reverts commit dfa8a9e8377ca60dcc3026e93087c5538791a0b5. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add release notes for pyproject.toml change Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Add deprecation warnings to top-level use of layer in catalog definition (#2964) * add deprecation Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * lint Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Use FutureWarning Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * changes based on review Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * add link to docs Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update RELEASE.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> --------- Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Update on credentials.md (#2787) * Update on credentials.md Updating example code in credentials.md to make it usable when copy/pasting. Since as of now it will generate a TypeError Signed-off-by: Jose <jmnunezd123@gmail.com> * adding a more explicit code example, now we directly show the user that project_path is a pathlib.Path object Signed-off-by: Jose <jmnunezd123@gmail.com> * Update docs/source/configuration/credentials.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> --------- Signed-off-by: Jose <jmnunezd123@gmail.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Cap pluggy 1.3 release (#2981) Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Make vale linter only run when PR opened or reopened (#2982) Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Update merge-gatekeeper.yml (#2960) Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Configure starters to use OmegaConfigLoader (#2974) * Change starters to use OmegaConfigLoader Signed-off-by: lrcouto <laurarccouto@gmail.com> * Fix linter error Signed-off-by: lrcouto <laurarccouto@gmail.com> * Suppress import outside toplevel linting for starters template Signed-off-by: L. R. Couto <laurarccouto@gmail.com> * Fix linter error Signed-off-by: lrcouto <laurarccouto@gmail.com> * Add changes to release notes Signed-off-by: lrcouto <laurarccouto@gmail.com> --------- Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: L. R. Couto <laurarccouto@gmail.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Stop OmegaConfigLoader from reading config from hidden directory like `ipynb_checkpoints` (#2977) * Check plugins implement valid hooks Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Add release note Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Staging work - add custom functions to check hidden folder and files. Tests still failing Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Fix test - checkpoints should use the same environment Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Revert "Check plugins implement valid hooks" This reverts commit f10bede0401b80e3385e93458995835a2db721c9. * Update RELEASE.md Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> * fix lint Signed-off-by: Nok <nok.lam.chan@quantumblack.com> --------- Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Add migration steps for `TemplatedConfigLoader` to `OmegaConfigLoader` (#2904) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Introduce a sentinel value _NO_VALUE to improve Global resolvers to support defaults `0` or `None` (#2976) * Add None support to globals Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add warning when default value is used Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Check keys Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Nok's suggestions Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Create the test to check the non-existing keys Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * add more tests to catch case when global key is not a dict Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Fix the null test Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Introduce sentinel value _NO_VALUE Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * rename test Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Improve error mesasge and raise InterpolationResolutionError when key does not exist and no default Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Fix non exist default test Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Fix test Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Use omegaconf to replace the custom resolving logic Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * uncommented some tests Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Remove dead code Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update error message Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Co-authored-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Release 0.18.13 (#2988) Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Fix docstrings on kedro/extras/datasets (#2995) * Fix docstrings on kedro/extras/datasets Signed-off-by: lrcouto <laurarccouto@gmail.com> * Fix formatting error Signed-off-by: lrcouto <laurarccouto@gmail.com> * Explicitly define code block Signed-off-by: lrcouto <laurarccouto@gmail.com> * Fix empty line under code block Signed-off-by: lrcouto <laurarccouto@gmail.com> * Fix formatting error Signed-off-by: lrcouto <laurarccouto@gmail.com> * Fix broken link Signed-off-by: lrcouto <laurarccouto@gmail.com> * Bump kedro-datasets version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update links in partitioned and incremental datasets Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update links in partitioned and incremental datasets Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update links in partitioned dataset Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update links in partitioned dataset Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add polars.GenericDataSet to .rst Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Co-authored-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Minor docs changes on data section to create a PR and test Vale styles (#2966) * Minor changes to create a PR and test Vale styles Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * fix some vale warnings Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> --------- Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * PiP/pyproject.toml and Conda/Meta.yaml Sync (#2922) * Resynced Signed-off-by: rxm7706 <95496360+rxm7706@users.noreply.github.com> * restore lower bound for pluggy Signed-off-by: rxm7706 <95496360+rxm7706@users.noreply.github.com> --------- Signed-off-by: rxm7706 <95496360+rxm7706@users.noreply.github.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * fix typo for build (#3001) Signed-off-by: rxm7706 <95496360+rxm7706@users.noreply.github.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Add hook example to access `metadata` (#2998) * Check plugins implement valid hooks Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * add Metadtahook Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Fix docs according to comments Signed-off-by: Nok <nok.lam.chan@quantumblack.com> --------- Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Bump release notes version Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Expand docs for customized pipeline templates. Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Update release notes and docs Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Update language linter to also run when PR converted ready for review Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Fix ci - Use `--resolver=backtracking` with `kedro build-reqs` in e2e tests (#3010) * Bump build version Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Remove telemetry from test default starter Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add package_name back Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Pin build only for 3.7 Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Try upgrade pip Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add constraint Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update for windows Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Run build-reqs with backtracking resolver Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Run build-reqs with backtracking resolver Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Update docs Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Update style to catch some more US spellings Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Make dataset factory resolve nested dict properly (#2993) Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Apply suggestions from code review Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Resolve suggestions from code review Signed-off-by: Jason Hite <jasonmhite@gmail.com> * Revert changes ukspelling.yml Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Jason Hite <jasonmhite@gmail.com> Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com> Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com> Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> Signed-off-by: lrcouto <laurarccouto@gmail.com> Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Ahdra Merali <ahdra.merali…
- Loading branch information
19 people
authored
Sep 12, 2023
1 parent
e949c6c
commit d3171f1