Usage of Kedro pipeline with web services & Deployment

[antonymilne]

@noklam I turned this into an issue just so I could add to it more easily. It's fine to have an issue just to bounce ideas around even if it's not ready for the backlog yet :)

Note. This issue is created for documenting how people are exposing Kedro pipeline to Web API (mainly RESTful), which helps us to understand the usage and think about the design in advance. Please share your use cases if you have done that!

Written by @noklam:

Nothing solid here, but I have seen a few GH issues and #kedro-user question. I think it is useful to understand how Kedro is being used with web API so I start writing down something.

• How common Kedro pipeline being exposed as a web endpoint?
• Parametrized session_id #1571 - User need to have custom session_id for logging purpose
• Creation of a KedroSession is quite expensive, it adds overhead particular we only allow 1 run per session, do we need KedroSession.reset() that simply reset the flag & timestamp? Currently I haven't seen anyone complain about it, but it's something that we can improve.
• How to distribute and extend kedro pipelines #795 - How can we make Kedro pipeline work better with a RESTful API? Is there an easy way that user can pass extra data (common in a RESTful call with JSON) and trigger a Kedro pipeline?

[antonymilne]

This discussion is very relevant also: #933 (comment)

[noklam]

Adding examples:

• Allow injecting data into a KedroSession run #2169 (comment)

[astrojuanlu]

Related: #143

[astrojuanlu]

This issue is almost void of content. Closing it so we can centralise the discussion.