Kedro role in the end-to-end ML lifecycle

[DimedS]

MLOps Insights from AWS Training

After attending the AWS MLOps training course, I appreciated how cloud providers explain their solutions to address the end-to-end MLOps model lifecycle. I'd like to share the core components, as I believe it will be useful for us to provide a comprehensive Kedro proposal in the MLOps architecture.

Key Points about MLOps:

1. Beyond DevOps: MLOps extends beyond traditional DevOps by encompassing not only code, CI/CD, and monitoring but also datasets, features, and model objects.
2. Collaborative Effort: MLOps involves a diverse set of roles, including data engineers, data scientists, DevOps engineers, and MLOps engineers. In some cases, models must be approved before deployment, potentially involving regulatory officers. Therefore, role-based access control is crucial.
3. Maturity Levels: Companies exhibit different MLOps maturity levels, ranging from the experimentation stage to repeatable (using repositories), reliable (automated testing, monitoring, multiple accounts), and finally scalable. Our solution should account for the company’s current stage.

Typical End-to-End ML Lifecycle:

Model Production Stages:

1. Model Building: This stage involves creating a model object and possibly selecting the best one among several models. While this can be done manually, it should also be automated at mature stages.
2. Model Deployment: This stage involves deploying one or more models from the previous stage to produce outputs (model predictions).

Detailed Stages:

Model Deployment Methods:

1. Batch: Offline inference with large datasets.
2. Asynchronous: Near-online execution with longer processing times, like a task queue.
3. Serverless: Online response during periods of no traffic, typically supported by cloud providers with pay-per-response models.
4. Real-time: Live predictions with low latency and sustained traffic.

Deployment decisions should consider resource management, whether using a single host with multiple models or a multiple-container architecture, and load balancing in real-time systems.

Deployment Strategies for Traffic Shifting:

1. Shadow Mode: The new model runs alongside the current model, but users receive results from the current model. Once validated, traffic can shift to the new model.
2. A/B Testing: Some users continue with the current model, while others use the new model(s), allowing data scientists to compare performance.

Monitoring:

1. Batch or Scheduled: Regular monitoring as part of a batch process or pipeline.
2. Online: Continuous monitoring.
3. Feedback Integration: Determine how monitoring feedback is integrated into the process, such as stopping a model if necessary.

Competitive Advantage of Cloud Providers:

Cloud providers excel because their solutions are integrated and offer resource management by code, allowing for seamless resource allocation within the ML pipeline.

For example, in AWS SageMaker Pipelines:

1. You can create an ML pipeline encompassing all the above steps, from data processing to model deployment and monitoring, including traffic shifting and A/B tests.
2. All code is notebook-friendly, allowing you to create and visualize the entire pipeline within a notebook.

Recommendations for Kedro:

Despite their advantages, reliance on cloud providers can lead to dependency on their solutions. For Kedro, we should consider two directions:

1. Platform-Agnostic ML Lifecycle: Provide an end-to-end ML lifecycle using Kedro alongside other open-source tools.
2. Cloud Integration: Enhance Kedro's value within cloud provider architectures.

Additionally, integration with core MLOps components that have become standalone solutions is crucial:

1. Feature Store
2. Model Registry
3. Model Monitoring:

Conclusion:

Since Kedro aims to simplify ML model production, I believe one of our goals should be to clearly explain its role at different stages of the end-to-end model cycle and how it integrates with other solutions. This architecture should consider the company’s maturity level and role model. Seamless integration with these solutions is essential for Kedro's development.

This is the first part of my learnings from the AWS course. I plan to write more about my understanding of Kedro's current proposal within the MLOps ecosystem. I welcome any thoughts on this topic.

[ElenaKhaustova]

Thank you, @DimedS, for this comprehensive summary and the valuable starting points!

Here are a few things I would take into consideration:

1. Integration vs. Development: The first main question is whether we want to provide the ML lifecycle via integration with platforms, develop it on our end, or pursue both avenues. This sounds like a complex task that requires significantly more resources than those used to develop Kedro so far. Therefore, I would lean towards focusing on integration rather than providing this feature within the framework itself.
2. Unified Approach: Based on item 1, I would not divide Platform-Agnostic ML Lifecycle and Cloud Integration. While I see the value of Kedro being platform-agnostic, we need cloud platforms to run our pipelines, as there is no easy way to replace them.
3. Competitive Positioning: Can we really compete with cloud platforms, and should we? Given the disparity in resources, it might be more beneficial to find our niche by offering an easy entry-level approach to implementing pipelines and the flexibility to switch between platforms. We should explore how Kedro can complement strengths of cloud platforms rather than competing directly.
4. Other framework Integration: Considering other frameworks used for model implementation, such as TensorFlow and PyTorch, cloud platforms are already integrated with them. Do we want to integrate with these frameworks directly, and how do we envision Kedro fitting into this setup involving Kedro, ML frameworks, and cloud platforms?
5. Operational Level: We need to determine at what level Kedro should operate:
   • Inside the Containers: As a package, this might be the easiest solution, but the drawback is that Kedro would operate with separate pipeline parts internally, losing some framework features.
   • Platform SDK Integration: This would require significant effort and extensive adaptation to various platforms.
   • Container Level: The challenge here lies in splitting the pipeline into parts, containerizing them, and transitioning from Kedro’s data flow to the platform's workflow.

This discussion serves as a critical starting point for defining our strategy, which can further lead to alignment on what we want to achieve in the short and long term and balancing it with what we can afford.