[Parent] - Add-ons Project Creation flow

[noklam]

Milestone: https://github.com/kedro-org/kedro/milestone/33 (To see what's left)

Blocking (Top priority):

• Merge pyproject.toml files for starters in line with python src-layout #2975 @DimedS
• Move tests folder to be aligned with develop template kedro-starters#170 @SajidAlamQB
• Update add-on test to parse pyproject.toml #3230 @noklam
• Strip pyproject.toml and requirements.txt in starters so it can be used by add-ons kedro-starters#171
• Move logging.yml in spaceflights starters kedro-starters#177 @SajidAlamQB

This issue is currently blocking all new add-ons Pr (i.e. spark, viz) because we have different template structure in kedro/templates and kedro-starters.

In Progress:

• Add addon flags to kedro new command  #2873 @lrcouto
• Add add_ons metadata to pyproject.toml for project creation #3188 @noklam (merge this first as refactor affect most add-on flow)
• Add add_ons to pyproject.toml in spaceflight starters kedro-starters#175 @merelcht
• Move logging.yml in spaceflights starters kedro-starters#177 @SajidAlamQB
• Add e2e tests for add-ons flow #3268 @SajidAlamQB

Blocked:

• Add pyspark to add-ons options #3073 @SajidAlamQB
• Add kedro-viz to add-ons options #3074 @noklam @SajidAlamQB
• Make it possible for users to select both pyspark and viz in add-ons #3075 @SajidAlamQB @AhdraMeraliQB (This is being done in Add kedro-viz to add-ons options #3074)
• Add an option to get an example pipeline in the project creation flow #3076 @DimedS

Design Ticket:

• Iterate on feedback on Project Creation Flow  #3054

Final:

• Let's do a final spike at the end to try our best to break this workflow
  Checking:
• kedro new - Should trigger add-ons flow
• kedro new --add-ons --starter should trigger an error, invalid combination of options
• kedro new + select pyspark will pull from kedro-starters
• Do kedro run to test that the created projects can be run successfully.
• "Data Structure" - do we have a better name?
• fill in the correct URLs to the relevant docs for each step of the flow

Potential Improvement:

• Refactor the logic to a separate add_on.py? It started to feel it overcrowd the starters.py. Starters still exist as a separate feature but kedro new can use either starter flow or add_on

[amandakys]

I've gone for Data Folder in the prototype that i've updated in #3054

[noklam]

I notice that in this PR. @amandakys

kedro new --addons="test, docs, log"

Should pyproject.toml stores this or the other mappings? Should we even have two?

ADD_ONS_DICT = {
    "1": "Linting",
    "2": "Testing",
    "3": "Custom Logging",
    "4": "Documentation",
    "5": "Data Structure",
}

[SajidAlamQB]

I notice that in this PR. @amandakys

I noticed it to, the ADD_ONS_DICT is used in the backend for post_gen_project.py while the other mappings are just for the cli. Is it worth aligning the naming between the backend and cli?

[amandakys]

I'm not sure i fully understand the problem but I think we agreed that the --addons flag would have the shortnames 'lint, docs, test' etc. While in the copy, they are called Linting, Testing, Documentation etc. For the user all they see is the copy, and in the --help they are told what the corresponding shortnames are.

I think it makes more sense to pick one set of names for the implementation, and the shortnames are probably fine. We can think of the long names as "display/presentation" names and the shortnames as like the "code" names

[noklam]

👍🏼, So pyproject.toml will keep the long name and CLI will keep the short. I can take the rest from here as it's just implementation details.

[noklam]

Just watch the video of the add-ons tech design.

I started some refactoring before watching the video, I share a lot of opinion with @DimedS.

Pure Function - @DimedS has a good point. IMO we shouldn't mutate function inputs unless it helps a lot on performance, which I think 99% of the case we shouldn't do it. Even if this has to be done, we should adopt some pattern. For example, in Pytorch there is an user interface add function, internally they have a add_ which actually mutate the tensor.

#3265 is a first step and it can be merged without interfering with existing works, the other refactor will affect PR that are still in progress so I prefer to do it a bit later.

[merelcht]

All major tasks listed on this issue have been completed and we did the bug bash to discover small issues that will be addressed as lower priority.