Update kedro catalog list command to account for dataset factories

[AhdraMeraliQB]

Description

With the introduction of dataset factories to the data catalog (#2635) our catalog CLI commands need to be updated to reflect the changes. This PR updates kedro catalog list to include datasets that make use of the dataset factories.

Development notes

I have tested this manually with a modified version of spaceflights. In the catalog, both entries for preprocessed_shuttles and preprocessed_companies have been replaced with the following:

preprocessed_{name}:
  type: pandas.ParquetDataSet
  filepath: data/02_intermediate/preprocessed_{name}.pq

Before the changes made in this PR, this project setup would yield the following output using kedro catalog list -p data_processing:

Datasets in 'data_processing' pipeline:
  Datasets mentioned in pipeline:
    CSVDataSet:
    - companies
    - reviews
    DefaultDataset:
    - preprocessed_shuttles
    - preprocessed_companies
    ExcelDataSet:
    - shuttles
    ParquetDataSet:
    - model_input_table
  Datasets not mentioned in pipeline:
    PickleDataSet:
    - regressor

With the changes this now looks like:

Datasets in 'data_processing' pipeline:
  Datasets generated from factories:
    pandas.ParquetDataSet:
    - preprocessed_shuttles
    - preprocessed_companies
  Datasets mentioned in pipeline:
    CSVDataSet:
    - companies
    - reviews
    ExcelDataSet:
    - shuttles
    ParquetDataSet:
    - model_input_table
  Datasets not mentioned in pipeline:
    PickleDataSet:
    - regressor

Questions for reviewers

This is more of a nit but I'm unsure "Datasets generated from factories" is the best way to describe this section, any other ideas/comments?

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes

[AhdraMeraliQB]

It's worth noting that PyYaml will automatically sort what it dumps to terminal. To configure this we would need to update to PyYAML 5.1.

Currently, generated datasets will be listed first, then those mentioned in the pipeline, and lastly those that aren't. A similar alphabetical ordering is applied to the pipelines themselves (as opposed to in order of input). I'm not the biggest fan of this, but I'm unsure it's worth bumping PyYAML for. In any case, I have re-ordered them, so that if/when we do update the PyYAML lower bound we can make the changes easily.

[ankatiyar]

QQ: any reason we're distinguishing between datasets explicitly in the catalog vs generated from factories?

[AhdraMeraliQB]

QQ: any reason we're distinguishing between datasets explicitly in the catalog vs generated from factories?

@ankatiyar

The idea spawned from a conversation with @merelcht - it is probably useful to distinguish these not just for clarity but also debugging purposes.

[ankatiyar]

The idea spawned from a conversation with @merelcht - it is probably useful to distinguish these not just for clarity but also debugging purposes.

This sounds good but I think then the heading for the "non factories" datasets could be changed to be consistent. Something like "Datasets generated from catalog files:"? What do you think'?

Also another thing, I was trying it out with my dummy project. But I noticed that this wasn't working for "Datasets not mentioned in pipeline" when these datasets are factories -

When all the datasets are explicitly mentioned :

Datasets in '__default__' pipeline:
  Datasets mentioned in pipeline:
    CSVDataSet:
    - france_companies
    - germany_companies
    - france_preprocessed_companies
    - germany_preprocessed_companies
    - switzerland_companies
    - switzerland_preprocessed_companies
    ParquetDataSet:
    - germany_final
    - switzerland_final
    - france_final
Datasets in 'pipe1' pipeline:
  Datasets mentioned in pipeline:
    CSVDataSet:
    - france_companies
    - germany_companies
    - france_preprocessed_companies
    - germany_preprocessed_companies
    - switzerland_companies
    - switzerland_preprocessed_companies
  Datasets not mentioned in pipeline:
    ParquetDataSet:
    - germany_final
    - switzerland_final
    - france_final

vs When all datasets are factories

Datasets in '__default__' pipeline:
  Datasets generated from factories:
    pandas.CSVDataSet:
    - france_preprocessed_companies
    - switzerland_companies
    - germany_companies
    - germany_preprocessed_companies
    - france_companies
    - switzerland_preprocessed_companies
    pandas.ParquetDataSet:
    - switzerland_final
    - france_final
    - germany_final
Datasets in 'pipe1' pipeline:
  Datasets generated from factories:
    pandas.CSVDataSet:
    - france_preprocessed_companies
    - switzerland_companies
    - germany_companies
    - germany_preprocessed_companies
    - france_companies
    - switzerland_preprocessed_companies

[AhdraMeraliQB]

@ankatiyar

The "Datasets not mentioned in pipeline" looks at entries included in the catalog that aren't included in the pipeline. Because the factory datasets aren't included in the catalog, they wouldn't show up here - for them to be resolved they would have to be explicitly defined in the pipeline, which would then mean they show up in the "Datasets mentioned in pipeline" section instead.

This sounds good but I think then the heading for the "non factories" datasets could be changed to be consistent. Something like "Datasets generated from catalog files:"? What do you think'?

I'm unsure the distinction here would be necessary as it is the default case, but it's worth considering 🤔

[ankatiyar]

The "Datasets not mentioned in pipeline" looks at entries included in the catalog that aren't included in the pipeline. Because the factory datasets aren't included in the catalog, they wouldn't show up here - for them to be resolved they would have to be explicitly defined in the pipeline, which would then mean they show up in the "Datasets mentioned in pipeline" section instead.

Could this be resolved like we do in runner.py -

kedro/kedro/runner/runner.py

Line 79 in c1822a2

registered_ds = [ds for ds in pipeline.data_sets() if ds in catalog]

Like -> catalog_ds = Union(set(catalog.list()), set(registered_ds))

[AhdraMeraliQB]

@ankatiyar I see, you're suggesting any datasets that make use of factories within the default pipeline should be considered in the "Datasets not mentioned in pipeline" section?

To be completely honest, I feel like it might be overkill - to me "Datasets generated from factories" are generated by the pipeline therefore there would be no need to list the ones generated by a different pipeline, but it would be good to see if @merelcht / any one else have any thoughts on this.

[merelcht]

@ankatiyar I see, you're suggesting any datasets that make use of factories within the default pipeline should be considered in the "Datasets not mentioned in pipeline" section?

To be completely honest, I feel like it might be overkill - to me "Datasets generated from factories" are generated by the pipeline therefore there would be no need to list the ones generated by a different pipeline, but it would be good to see if @merelcht / any one else have any thoughts on this.

I don't really understand this. If a dataset isn't mentioned in the pipeline it wouldn't be picked up by a dataset factory, so how could a dataset generated by a factory every belong to the "not mentioned in pipeline" section?

[merelcht]

I've left a couple of suggestions and questions, but overall it looks good. I think "datasets generated from factories" is a good way to describe this section.

[SajidAlamQB]

This looks great! 🌟 This change makes a lot of sense and improves the kedro catalog list to add visibility into datasets generated from factories well.

[merelcht]

Nice work 👍

[merelcht]

LGTM! ⭐