Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add hook example to access metadata #2998

Merged
merged 8 commits into from
Sep 5, 2023
Merged

Add hook example to access metadata #2998

merged 8 commits into from
Sep 5, 2023

Conversation

noklam
Copy link
Contributor

@noklam noklam commented Sep 1, 2023

NOTE: Kedro datasets are moving from kedro.extras.datasets to a separate kedro-datasets package in
kedro-plugins repository. Any changes to the dataset implementations
should be done by opening a pull request in that repository.

Description

fix #2676

Development notes

Checklist

  • Read the contributing guidelines
  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added a description of this change in the RELEASE.md file
  • Added tests to cover my changes
  • Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

noklam added 5 commits July 30, 2023 18:53
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
@noklam
Copy link
Contributor Author

noklam commented Sep 1, 2023

I will keep the PR simple and I created the follow up in #2999

@noklam noklam requested a review from merelcht September 1, 2023 12:33
Comment on lines 213 to 217
conf_catalog: Dict[str, Any],
conf_creds: Dict[str, Any],
feed_dict: Dict[str, Any],
save_version: str,
load_versions: Dict[str, str],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All these parameters are not used right? Maybe we can remove them from the function. Pluggy allows this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point. We do this inconsistently, sometimes we do include signature that wasn't used (I was asked to add that before, can't find it exactly which PR it is).

For example, after_node_run has a redundant node argument.
https://docs.kedro.org/en/stable/hooks/examples.html#add-observability-to-your-pipeline

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stichbury do you think we should keep all our example minimals? We can do a clean up if this is agreed.

In this case

   def after_catalog_created(
        self,
        catalog: DataCatalog,
        conf_catalog: Dict[str, Any],
        conf_creds: Dict[str, Any],
        feed_dict: Dict[str, Any],
        save_version: str,
        load_versions: Dict[str, str])

or

   def after_catalog_created(
        self,
        catalog: DataCatalog)

will both work.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine to remove them too and keep it minimal.

@@ -200,3 +200,22 @@ HOOKS = (AzureSecretsHook(),)
```{note}
Note: `DefaultAzureCredential()` is Azure's recommended approach to authorise access to data in your storage accounts. For more information, consult the [documentation about how to authenticate to Azure and authorize access to blob data](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python).
```

## Use Hook to read `metadata` from `DataCatalog`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
## Use Hook to read `metadata` from `DataCatalog`
## Use a Hook to read `metadata` from `DataCatalog`

@@ -200,3 +200,22 @@ HOOKS = (AzureSecretsHook(),)
```{note}
Note: `DefaultAzureCredential()` is Azure's recommended approach to authorise access to data in your storage accounts. For more information, consult the [documentation about how to authenticate to Azure and authorize access to blob data](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python).
```

## Use Hook to read `metadata` from `DataCatalog`
We recommend to use the `after_catalog_created` hook if you need to access `metadata` to extend Kedro.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
We recommend to use the `after_catalog_created` hook if you need to access `metadata` to extend Kedro.
Use the `after_catalog_created` Hook to access `metadata` to extend Kedro.

Copy link
Contributor

@stichbury stichbury left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with minor changes. What's Vale saying about this page?

stichbury and others added 2 commits September 4, 2023 08:26
@noklam noklam requested a review from astrojuanlu September 4, 2023 14:55
@noklam noklam enabled auto-merge (squash) September 4, 2023 14:55
Copy link
Member

@astrojuanlu astrojuanlu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this example 👍🏽 but I'm wondering, is this a full solution for #2676? Wondering if we should have some prose somewhere else in the data catalog documentation, otherwise I fear it might be a bit difficult to find. Essentially what @stichbury wrote in #2940 as a response to the reviews on #2888.

What do you think?

@noklam
Copy link
Contributor Author

noklam commented Sep 5, 2023

@astrojuanlu Maybe we split the ticket too much, that should be kedro-org/kedro-viz#1650 ?

Copy link
Member

@astrojuanlu astrojuanlu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too many tickets 🥲 Thanks @noklam !

@noklam noklam merged commit 17f404f into main Sep 5, 2023
@noklam noklam deleted the docs/metadata-catalog branch September 5, 2023 13:23
ankatiyar added a commit that referenced this pull request Sep 12, 2023
* Remove non-user-facing release notes about pip pin (#2854)

Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Allow for customization of pipeline templates

Controlled via `settings.PIPELINE_TEMPLATE_PATH`.

Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Update lookup based on feedback

Instead of lookups on `settings.py` just statically look for a `templates`
directory in the project root. If this folder is present it will override
the default global template, otherwise fall back to the previous global
default.

Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Add CLI flag for pipeline creation template -t/--template

Precedence for template_path is:
   command line > project templates/pipeline dir > global default

pipeline create now will also print the template folder used.

Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* First basic unit test

TODO is testing the CLI flag too, and maybe testing that the cleanup works
with custom templates.

Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Add test to check overriding template path on the CLI

Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Linter formatting fixes

Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Release 0.18.12 (#2871)

* bump version and prepare for new release

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update RELEASE.md

Added community contributions

* add DeltaTableDataSet to rst

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update setup.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update setup.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update setup.py

Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* temporarily use the fixed kedro-plugins branch

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update setup.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* test docsbuild on merels branch

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* use kedro-datasets new release

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

---------

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Fix broken link to Prefect website in deploy guide (#2885)

Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Clarify the <micropkg_name> argument to kedro micropkg package (#2835)

Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Allow registering of custom resolvers to `OmegaConfigLoader` (#2869)

* Allow registering of custom resolvers to OCL

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Complete doc string

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Add test for overwritten resolvers

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update test for overwritten resolvers

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Remove replace=True by default

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update release notes

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update release notes

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Add debug level log for registering new resolver

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

---------

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Document the use of custom resolvers with `OmegaConfigLoader` (#2896)

* Update docs for custom resolvers for OCL

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update FAQ with correct link

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Fix typo

Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>

* Add note for oc.env

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Suggestions from code review

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

---------

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Update kedro pipeline create to use new /conf file structure (#2856)

* add parameters_pipelineNane.yml template in the conf folder

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* del yml template from parameters folder

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* modify delete pipeline command to use correct path to yml files

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* modify test_pipeline move parameters foldername to the yml filename

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* collect configs to package not only from parameters folder, but from core conf folder also

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* move parameters foldername to the yml filename, remove nesting

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* move parameters foldername to the yml filename, remove nesting

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* Add changes to RELEASE.md

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* Update RELEASE.md

Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>

* Update RELEASE.md, change to release 0.18.13

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* save old structure for 'pipeline delete' command for backward compatibility

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* modify test_pipeline for 'pipeline delete' backward compability

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

---------

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>
Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Update kedro catalog create to use new /conf structure (#2884)

* modify test_catalog, remove nesting in the catalog folder

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* move catalog foldername to the yml filename, remove nesting

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* Update RELEASE.md

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* Modify docs, fix broken link

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

---------

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Add migration steps for `ConfigLoader` to `OmegaConfigLoader` (#2887)

* Add migration steps for CL to OCL

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Fix #2498 Adding logging issue 2498 (#2842)

* Fixing typo n on line 114 file /docs/source/nodes_and_pipelines/run_a_pipeline.md from issue #2607

Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com>

* Removed redundant packages at lines 66, 67 and 68 on file kedro/docs/source/nodes_and_pipelines/run_a_pipeline.md, packages:  from kedro.io import AbstractDataSet, DataCatalog, MemoryDataSet,  from kedro.pipeline import Pipeline and  from kedro.runner.runner import AbstractRunner

Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com>

* Add logger object for logging when computing micropackage version

Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com>

* Added new format to the logging

Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com>

---------

Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com>
Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Try only trigger docs build when release notes updated (#2907)

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Add Python 3.11 support to Kedro (#2851)

* add 3.11

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update setup.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* limit number of python version tests

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* fix setup.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* delta-spark version

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* try original setup

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* remove 3.11 from circleci

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update setup.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update setup.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* pin matplotlib to 3.7.2 for python 3.11

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update test_import.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* add noks test adjustments

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update setup.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update setup.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* replace eq with is for multi_catalog

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* minor fixes

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* remove matplotlib 3.11 pin

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* matplotlib pin for 3.11

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* run original matplotlib pin with ubuntu only

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* run circleci

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* pin spark < 3.4 as its breaking

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* spark datasets revert earlier changes back to original

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* delta

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* remove all windows tests from GHA

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* skip coverage as not needed

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Checking only WIN tests (will revert)

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update unit-tests.yml

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* undo previous commit

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* add msbuild

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* add ubuntu tests back in and remove 3.11 tests from circleci add in msbuild for the other tests

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* set hdf5 disable check and lint

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* pin tables for 3.8

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* pin 3.8 tables

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* lint

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* remove 3.11 completely from circleci and tidy up

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* add 3.11 to pyarrow pin for docs

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* undo previous commit

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* update rtd to python 3.11

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* try 3.10 on rtd

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* fix outdated pylint to ruff

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* changes based on review

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* add mock for pyarrow and deltalake for docs

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* undo previous commit

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update setup.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update .readthedocs.yml

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update conf.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* changes based on review

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* fix comments

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* test running 3.10 parallel

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Add to release notes and update badges on docs

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Add migration tips

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

---------

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Revise FAQs and README (#2909)

* Revise FAQs and README

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Add back the data layers FAQ as I've no idea where else it fits

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* minor changes from review

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

---------

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Update Generator example (#2858)

* update release note

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* update to binary mode  when opening file

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>

* Simplify the implementation of Dataset

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>

* simplify the example - wip

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>

* Update the nodes

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>

* update example

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>

* Update example

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* Apply suggestions from code review

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Apply suggestions from code review

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Apply suggestions from code review

Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Apply suggestions from code review

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* add more instruction to guide user to create the custom dataset

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* update the reference from `pipeline.py` -> `nodes.py`

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* Collapse the nodes.py function into a expandable section

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* Update nodes.md

* Update nodes.md

---------

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* update docs to reflect change to /conf file structure (#2913)

* update docs to reflect change to /conf file structure

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* update docs, folder structure diagrams

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

---------

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Change CONTRIBUTING.md file based on PR #2894 (#2914)

Signed-off-by: lrcouto <laurarccouto@gmail.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Move contribution guidelines from CONTRIBUTING.md to the Wiki (#2894)

* Add drafts for new contributing.md and wiki page

Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Add contribution details to wiki page draft

Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Remove contributing-to-Kedro.md draft from the branch

Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Fix broken link to Prefect website in deploy guide (#2885)

Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Clarify the <micropkg_name> argument to kedro micropkg package (#2835)

Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Allow registering of custom resolvers to `OmegaConfigLoader` (#2869)

* Allow registering of custom resolvers to OCL

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Complete doc string

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Add test for overwritten resolvers

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update test for overwritten resolvers

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Remove replace=True by default

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update release notes

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update release notes

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Add debug level log for registering new resolver

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

---------

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Some adjustments to ordering of text

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Document the use of custom resolvers with `OmegaConfigLoader` (#2896)

* Update docs for custom resolvers for OCL

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update FAQ with correct link

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Fix typo

Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>

* Add note for oc.env

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Suggestions from code review

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

---------

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Update kedro pipeline create to use new /conf file structure (#2856)

* add parameters_pipelineNane.yml template in the conf folder

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* del yml template from parameters folder

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* modify delete pipeline command to use correct path to yml files

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* modify test_pipeline move parameters foldername to the yml filename

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* collect configs to package not only from parameters folder, but from core conf folder also

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* move parameters foldername to the yml filename, remove nesting

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* move parameters foldername to the yml filename, remove nesting

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* Add changes to RELEASE.md

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* Update RELEASE.md

Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>

* Update RELEASE.md, change to release 0.18.13

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* save old structure for 'pipeline delete' command for backward compatibility

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* modify test_pipeline for 'pipeline delete' backward compability

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

---------

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>
Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Update CONTRIBUTING.md

Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Fix broken link

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Update kedro catalog create to use new /conf structure (#2884)

* modify test_catalog, remove nesting in the catalog folder

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* move catalog foldername to the yml filename, remove nesting

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* Update RELEASE.md

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* Modify docs, fix broken link

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

---------

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>
Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Add migration steps for `ConfigLoader` to `OmegaConfigLoader` (#2887)

* Add migration steps for CL to OCL

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Fix #2498 Adding logging issue 2498 (#2842)

* Fixing typo n on line 114 file /docs/source/nodes_and_pipelines/run_a_pipeline.md from issue #2607

Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com>

* Removed redundant packages at lines 66, 67 and 68 on file kedro/docs/source/nodes_and_pipelines/run_a_pipeline.md, packages:  from kedro.io import AbstractDataSet, DataCatalog, MemoryDataSet,  from kedro.pipeline import Pipeline and  from kedro.runner.runner import AbstractRunner

Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com>

* Add logger object for logging when computing micropackage version

Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com>

* Added new format to the logging

Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com>

---------

Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com>
Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Try only trigger docs build when release notes updated (#2907)

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Add Python 3.11 support to Kedro (#2851)

* add 3.11

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update setup.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* limit number of python version tests

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* fix setup.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* delta-spark version

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* try original setup

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* remove 3.11 from circleci

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update setup.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update setup.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* pin matplotlib to 3.7.2 for python 3.11

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update test_import.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* add noks test adjustments

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update setup.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update setup.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* replace eq with is for multi_catalog

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* minor fixes

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* remove matplotlib 3.11 pin

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* matplotlib pin for 3.11

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* run original matplotlib pin with ubuntu only

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* run circleci

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* pin spark < 3.4 as its breaking

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* spark datasets revert earlier changes back to original

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* delta

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* remove all windows tests from GHA

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* skip coverage as not needed

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Checking only WIN tests (will revert)

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update unit-tests.yml

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* undo previous commit

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* add msbuild

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* add ubuntu tests back in and remove 3.11 tests from circleci add in msbuild for the other tests

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* set hdf5 disable check and lint

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* pin tables for 3.8

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* pin 3.8 tables

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* lint

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* remove 3.11 completely from circleci and tidy up

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* add 3.11 to pyarrow pin for docs

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* undo previous commit

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* update rtd to python 3.11

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* try 3.10 on rtd

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* fix outdated pylint to ruff

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* changes based on review

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* add mock for pyarrow and deltalake for docs

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* undo previous commit

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update setup.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update .readthedocs.yml

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update conf.py

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* changes based on review

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* fix comments

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* test running 3.10 parallel

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Add to release notes and update badges on docs

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Add migration tips

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

---------

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Revise FAQs and README (#2909)

* Revise FAQs and README

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Add back the data layers FAQ as I've no idea where else it fits

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* minor changes from review

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

---------

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Update Generator example (#2858)

* update release note

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* update to binary mode  when opening file

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>

* Simplify the implementation of Dataset

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>

* simplify the example - wip

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>

* Update the nodes

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>

* update example

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>

* Update example

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* Apply suggestions from code review

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Apply suggestions from code review

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Apply suggestions from code review

Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Apply suggestions from code review

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* add more instruction to guide user to create the custom dataset

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* update the reference from `pipeline.py` -> `nodes.py`

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* Collapse the nodes.py function into a expandable section

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* Update nodes.md

* Update nodes.md

---------

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Update CONTRIBUTING.md

---------

Signed-off-by: lrcouto <laurarccouto@gmail.com>
Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com>
Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Co-authored-by: Jonathan Cohen <jon@joncohen.dev>
Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com>
Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Co-authored-by: Dmitry Sorokin <40151847+DimedS@users.noreply.github.com>
Co-authored-by: Laíza Milena Scheid Parizotto <41296740+laizaparizotto@users.noreply.github.com>
Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com>
Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Remove redundant pages and direct users to wiki (#2917)

* Remove redundant pages and direct users to wiki

Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Lint docs changes

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Few tweaks to index page

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

---------

Signed-off-by: lrcouto <laurarccouto@gmail.com>
Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Deprecate abstract "DataSet" in favor of "Dataset" (#2746)

* Deprecate abstract "DataSet" in favor of "Dataset"

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Update docs references to abstract "DataSet" class

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Fix `AbstractDataSet` reference to `AbstractDataset`

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Update docs/source/conf.py

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Change remaining `_DEPRECATED_ERROR_CLASSES` to `_DEPRECATED_CLASSES`, update type hints

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Update RELEASE.md

* Reformat kedro/io/__init__.py with Black and isort

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Remove duplicate imports

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Move imports

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

---------

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Reorganise and improve the data catalog documentation (#2888)

* First drop of newly organised data catalog docs

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* linter

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Added to-do notes

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Afternoon's work in rewriting/reorganising content

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* More changes

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Further changes

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Another chunk of changes

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Final changes

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Revise ordering of pages

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Add new CLI commands to dataset factory docs (#2935)

* Add changes from #2930

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Lint

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Apply suggestions from code review

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Make code snippets collapsable

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

---------

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Bunch of changes from feedback

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* A few more tweaks

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Update h1,h2,h3 font sizes

Signed-off-by: Tynan DeBold <thdebold@gmail.com>

* Add code snippet for using DataCatalog with Kedro config

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Few more tweaks

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Update docs/source/data/data_catalog.md

* Upgrade kedro-datasets for docs

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Improve prose

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

---------

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Signed-off-by: Tynan DeBold <thdebold@gmail.com>
Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Co-authored-by: Ahdra Merali <90615669+AhdraMeraliQB@users.noreply.github.com>
Co-authored-by: Tynan DeBold <thdebold@gmail.com>
Co-authored-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Add line about Viz to PR template (#2929)

Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Add architecture graphic back to docs with revisions (#2916)

* Revise FAQs and README

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Add back the data layers FAQ as I've no idea where else it fits

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* minor changes from review

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Re-add kedro arch diagram, with revised graphic

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* fix broken anchor

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* fix broken anchor

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Update docs/source/extend_kedro/architecture_overview.md

Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com>

* Update docs/source/extend_kedro/architecture_overview.md

Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com>

* Update docs/source/extend_kedro/architecture_overview.md

Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com>

* Update docs/source/extend_kedro/architecture_overview.md

Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com>

* Changes to architecture page following review

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Change diagram following reivew

* Add links to API docs

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Added in type of users

* Fix linting error

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

---------

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com>
Co-authored-by: Yetunde Dada <43755008+yetudada@users.noreply.github.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Add kedro catalog resolve command (#2891)

* Implement working draft

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Remove comments

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Remove raw_config

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Use config loader to access catalog config

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Use dict comprehension

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Remove pipeline filtering

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Prevent overwrite or param inclusion

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Trim filepath

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Appease linter

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Add test for resolve

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Add test to exclude params

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Add test for overwrite (not working)

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Fix test

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Remove print

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Add changes to RELEASE.md

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Trim unreachable code

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Make helper function private

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Refactor

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Add changes to the docs

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Refactor 2

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Fix docs build

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Add suggestions from code review (docs)

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Add suggestions from code review (code)

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Rename catalog_config variable to explicit_datasets

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Apply suggestions from code review

Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>

* Fix mocking in test

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Change test fixture name

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

* Lint

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>

---------

Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com>
Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Replace "DataSet" with "Dataset" in Markdown files (#2735)

* LambdaDataSet->LambdaDataset in .md files

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* MemoryDataSet->MemoryDataset in .md files

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* PartitionedDataSet->PartitionedDataset in .md files

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* IncrementalDataSet->IncrementalDataset in .md files

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* CachedDataSet->CachedDataset in .md files

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* DataSetError->DatasetError in .md files

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* DataSetNotFoundError->DatasetNotFoundError in .md files

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Replace "DataSet" with "Dataset" in Markdown files

* Update RELEASE.md

* Fix remaining instance of "*DataSet*"->"*Dataset*"

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* `find . -name '*.md' -print0 | xargs -0 sed -i "" "s/\([^A-Za-z]\)DataSet/\1Dataset/g"`

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Change non-class instances of Dataset to dataset

* Replace any remaining instances of DataSet in docs

* Fix a broken link to docs for `PartitionedDataset`

---------

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Update h1,h2,h3 font sizes in the docs pages (#2938)

* Update h1,h2,h3 font sizes

Signed-off-by: Tynan DeBold <thdebold@gmail.com>

* Update h2 font size

Signed-off-by: Tynan DeBold <thdebold@gmail.com>

---------

Signed-off-by: Tynan DeBold <thdebold@gmail.com>
Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Automatically trigger `kedro-starters` release on the release of `kedro` (#2889)

* Add GA for triggering starters release

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update token name

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

---------

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Create issues_metrics.yml (#2814)

* Create issues_metrics.yml

* Update issues_metrics.yml

Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Clean up setuptools and wheel requirements to align with PEP-518 (#2927)

Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Clean up `kedro pipeline create` outdated docs (#2945)

* remove the README template generate by kedro pipeline create

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* fix docstring for kedro catalog create

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* Fix broken link

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* fix broken link

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

---------

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Add globals feature for `OmegaConfigLoader` using a globals resolver (#2921)

* Refactor load_and_merge_dir()

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Try adding globals resolver

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Minor change

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Add globals resolver

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Revert refactoring

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Add test + remove self.globals

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Allow for nested variables in globals

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Add documentation

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Typo

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Add error message + test

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Apply suggestions from code review

Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>

* Split test into multiple tests

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Restrict the globals config_patterns

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Release notes

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update docs/source/configuration/advanced_configuration.md

Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>

* Add helpful error message for keys starting with _

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Enable setting default value for globals resolver

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Typo

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Move test for keys starting with _ to the top

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Fix cross ref link in docs

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

---------

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Consolidate two `ruff-pre-commit` entries into one (#2881)

* Clean up Ruff configuration in base pyproject.toml

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Consolidate two `ruff-pre-commit` entries into one

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Move Ruff hook to the top, because it's aggressive

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Add `kedro` as `known-first-party` for `isort` run

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Add noqa in docs/source/conf.py

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* Fix ruff linter precommit config

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

---------

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
Co-authored-by: Nok <nok.lam.chan@quantumblack.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Fix typos across the documentation (#2956)

* Fix typos across docs

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Capitalisation stuff

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

---------

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Setup Vale linter as a GHA workflow (#2953)

* Add vale GHA

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update condition

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update .github/styles/Kedro/ignore.txt

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* add some styles

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Remove styles for now

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

---------

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Fix README to show graphics on PyPI (#2961)

* Fix for PyPI

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Few more tweaks

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Few more tweaks

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Few more tweaks

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Few more tweaks

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Update README.md

* Update README.md

---------

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Add some Vale styles (#2963)

* add some styles

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Tweaks to styles

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* more tweaks

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Revise some styles

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Revise severity of some styles

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* Update .github/styles/Kedro/words.yml

Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

---------

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Minor changes to test + release notes + docs

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Update anyconfig requirement from ~=0.10.0 to >=0.10,<0.14 (#2876)

Updates the requirements on [anyconfig](https://github.com/ssato/python-anyconfig) to permit the latest version.
- [Changelog](https://github.com/ssato/python-anyconfig/blob/next/NEWS)
- [Commits](https://github.com/ssato/python-anyconfig/compare/RELEASE_0.10.0...RELEASE_0.13.0)

Signed-off-by: Jason Hite <jasonmhite@gmail.com>
---
updated-dependencies:
- dependency-name: anyconfig
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Move default template to static `pyproject.toml`, take 2 (#2853)

* Remove leftover reference to micropkg setup.py

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Replace outdated extras requirement with kedro-datasets

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Remove outdated mention to environment.yml

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Move flake8 configuration to dedicated file

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Replace setup.py with pyproject.toml in default project template

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Adapt link to moving file in docs

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Revert "Replace outdated extras requirement with kedro-datasets"

This reverts commit dfa8a9e8377ca60dcc3026e93087c5538791a0b5.

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

* Add release notes for pyproject.toml change

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

---------

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Add deprecation warnings to top-level use of layer in catalog definition (#2964)

* add deprecation

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* lint

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Use FutureWarning

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* changes based on review

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* add link to docs

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>

* Update RELEASE.md

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

---------

Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Update on credentials.md (#2787)

* Update on credentials.md

Updating example code in credentials.md to make it usable when copy/pasting. Since as of now it will generate a TypeError

Signed-off-by: Jose <jmnunezd123@gmail.com>

* adding a more explicit code example, now we directly show the user that project_path is a pathlib.Path object

Signed-off-by: Jose <jmnunezd123@gmail.com>

* Update docs/source/configuration/credentials.md

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>

---------

Signed-off-by: Jose <jmnunezd123@gmail.com>
Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Cap pluggy 1.3 release (#2981)

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Make vale linter only run when PR opened or reopened (#2982)

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Update merge-gatekeeper.yml (#2960)

Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Configure starters to use OmegaConfigLoader (#2974)

* Change starters to use OmegaConfigLoader

Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Fix linter error

Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Suppress import outside toplevel linting for starters template

Signed-off-by: L. R. Couto <laurarccouto@gmail.com>

* Fix linter error

Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Add changes to release notes

Signed-off-by: lrcouto <laurarccouto@gmail.com>

---------

Signed-off-by: lrcouto <laurarccouto@gmail.com>
Signed-off-by: L. R. Couto <laurarccouto@gmail.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Stop OmegaConfigLoader from reading config from hidden directory like `ipynb_checkpoints` (#2977)

* Check plugins implement valid hooks

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>

* Add release note

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>

* Staging work - add custom functions to check hidden folder and files. Tests still failing

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>

* Fix test - checkpoints should use the same environment

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>

* Revert "Check plugins implement valid hooks"

This reverts commit f10bede0401b80e3385e93458995835a2db721c9.

* Update RELEASE.md

Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com>

* fix lint

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

---------

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Add migration steps for `TemplatedConfigLoader` to `OmegaConfigLoader` (#2904)

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Introduce a sentinel value _NO_VALUE to improve Global resolvers to support defaults `0` or `None` (#2976)

* Add None support to globals

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Add warning when default value is used

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Check keys

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Nok's suggestions

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Create the test to check the non-existing keys

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* add more tests to catch case when global key is not a dict

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* Fix the null test

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* Introduce sentinel value _NO_VALUE

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* rename test

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* Improve error mesasge and raise InterpolationResolutionError when key does not exist and no default

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* Fix non exist default test

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* Fix test

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* Use omegaconf to replace the custom resolving logic

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* uncommented some tests

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

* Remove dead code

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update error message

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

---------

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
Co-authored-by: Nok <nok.lam.chan@quantumblack.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Release 0.18.13 (#2988)

Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Fix docstrings on kedro/extras/datasets (#2995)

* Fix docstrings on kedro/extras/datasets

Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Fix formatting error

Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Explicitly define code block

Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Fix empty line under code block

Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Fix formatting error

Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Fix broken link

Signed-off-by: lrcouto <laurarccouto@gmail.com>

* Bump kedro-datasets version

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update links in partitioned and incremental datasets

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update links in partitioned and incremental datasets

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update links in partitioned dataset

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update links in partitioned dataset

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Add polars.GenericDataSet to .rst

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

---------

Signed-off-by: lrcouto <laurarccouto@gmail.com>
Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Co-authored-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Minor docs changes on data section to create a PR and test Vale styles (#2966)

* Minor changes to create a PR and test Vale styles

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

* fix some vale warnings

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

---------

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* PiP/pyproject.toml and Conda/Meta.yaml Sync (#2922)

* Resynced

Signed-off-by: rxm7706 <95496360+rxm7706@users.noreply.github.com>

* restore lower bound for pluggy

Signed-off-by: rxm7706 <95496360+rxm7706@users.noreply.github.com>

---------

Signed-off-by: rxm7706 <95496360+rxm7706@users.noreply.github.com>
Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* fix typo for build (#3001)

Signed-off-by: rxm7706 <95496360+rxm7706@users.noreply.github.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Add hook example to access `metadata`  (#2998)

* Check plugins implement valid hooks

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>

* add Metadtahook

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>

* Fix docs according to comments

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

---------

Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Bump release notes version

Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Expand docs for customized pipeline templates.

Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Update release notes and docs

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Update language linter to also run when PR converted ready for review

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Fix ci - Use `--resolver=backtracking` with `kedro build-reqs` in e2e tests (#3010)

* Bump build version

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Remove telemetry from test default starter

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Add package_name back

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Pin build only for 3.7

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Try upgrade pip

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Add constraint

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Update for windows

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Run build-reqs with backtracking resolver

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Run build-reqs with backtracking resolver

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

---------

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Update docs

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Update style to catch some more US spellings

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Make dataset factory resolve nested dict properly (#2993)

Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Apply suggestions from code review

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Resolve suggestions from code review

Signed-off-by: Jason Hite <jasonmhite@gmail.com>

* Revert changes ukspelling.yml

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

---------

Signed-off-by: Jason Hite <jasonmhite@gmail.com>
Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: laizaparizotto <laizamsparizotto@gmail.com>
Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: lrcouto <laurarccouto@gmail.com>
Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: Ahdra Merali <ahdra.merali…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Document datasets' metadata attribute in catalog definition
3 participants