Revise the modular pipelines documentation to improve clarity

[stichbury]

Description

The docs page on modular pipelines could use some ❤️ as it's hard to understand and doesn't tie back very well to what we've already discussed in the spaceflights tutorial.

It needs a solid review and update.

[merelcht]

@stichbury What kind of changes do you have in mind? Let's add them here, so anyone in the team could work on this ticket.

[stichbury]

@merelcht Sorry it's taken me so long to spot your question!

So I think the page at present doesn't really explain to the user why they may want to create a modular pipeline, nor what/how they do it.

I'd like to see it more like a tutorial that is based on the spaceflights starter (extends it). So we title the page something like "How to manage complex pipelines" because we can't be sure that everyone knows what a modular pipeline is.

How to manage complex pipelines

Introducing modular pipelines

Benefits of modular pipelines

Not marketing, but list each of the potential user problems and explain how the modular pipeline may solve them

Tutorial to create a modular pipeline

Detail goes here -- explain namespaces

How to share a pipeline

How to use a shared pipeline

FAQs

[stichbury]

This is just a rough draft of how the page could look but gives an idea of how to rearrange what is there/add new materials

[astrojuanlu]

Could we merge this with #2723 ?

[stichbury]

Tempting! Ideally they need to be done together but the modular pipelines terminology discussion is a deeper one than the "fix the docs so they read better" task. That is; I'm pretty sure I could do the latter but I'm no way qualified to tackle the former #2723 without some further "training".

[astrojuanlu]

Since badly-named "modular pipelines" (for most practical purposes, a "pipeline" in Kedro framework is a modular pipeline) are still confusing (echoing @idanov comments on @ankatiyar's Tech Design session on kedro-airflow) I think it's time we take on this issue.

I have a very specific proposal:

• We rename "Pipelines" to "Pipeline objects" in https://docs.kedro.org/en/0.19.5/nodes_and_pipelines/pipeline_introduction.html
• We rewrite the intro of https://docs.kedro.org/en/0.19.5/nodes_and_pipelines/modular_pipelines.html

kedro/docs/source/nodes_and_pipelines/modular_pipelines.md

Line 5 in 8da1f91

In many typical Kedro projects, a single (“main”) pipeline increases in complexity as the project evolves. To keep your project fit for purpose, we recommend that you create modular pipelines, which are logically isolated and can be reused. Modular pipelines are easier to develop, test and maintain, and are portable so they can be copied and reused between projects.

to make it clear that "Modular pipelines" is the Kedro Framework take on how Pipeline objects should be wrapped and structured.

• We remove the sentence "The kedro.pipeline.modular_pipeline.pipeline wrapper method unlocks the real power of modular pipelines" because it's a lie (see Rectify "modular pipelines" terminology #2723 (comment))
• We remove any explicit mention of kedro.pipeline.modular_pipeline.pipeline from anywhere in our docs (users should always use kedro.pipeline.pipeline), including from this section https://docs.kedro.org/en/0.19.5/nodes_and_pipelines/modular_pipelines.html#using-the-modular-pipeline-wrapper-to-provide-overrides
• We sweep all our docs and make sure that, when we talk about namespaces, we call them Namespaces or even Namespaced pipelines
• We unhide the folder structure in https://docs.kedro.org/en/0.19.5/nodes_and_pipelines/modular_pipelines.html#what-does-kedro-pipeline-create-do and explain that pipeline.py contains a create_pipeline function that returns a Pipeline object
• We move all the examples from https://docs.kedro.org/en/0.19.5/nodes_and_pipelines/modular_pipelines.html#using-the-modular-pipeline-wrapper-to-provide-overrides onwards to "Pipeline objects" (current pipeline_introduction.md)

How does this sound?

[astrojuanlu]

Reprioritising this as High.

[stichbury]

I'm going to unsubscribe from this issue as I can't help much, but do tag me in later if you want a review of the finished docs. I would say, just as you've noted that you're reprioritising this, that you need to be sure it is an issue that's impacting sufficient people to warrant the time it'll take to unpick and detangle this. I know it's frustrating for the team, but is it confusing sufficient numbers of users at this point to be something you take on? If so, godspeed, and let me know when you're ready for a review!