OSL Internship ‐ 2024 ‐ 2nd Cycle

Makim

• Documentation: https://osl-incubator.github.io/makim
• License: BSD 3 Clause: https://github.com/osl-incubator/makim/blob/main/LICENSE
• Code of Conduct: https://github.com/osl-incubator/makim/blob/main/CODE_OF_CONDUCT.md

---

Project Idea 1: Adding Pipeline Support to Makim and Change from dependencies to hooks

Abstract

Makim is a versatile and extensible automation tool designed to simplify complex workflows and tasks in software development. While it excels at managing individual targets, it currently lacks native support for defining and executing pipelines, a critical feature for orchestrating sequences of tasks efficiently. This proposal aims to extend Makim's capabilities by introducing support for defining, running, and visualizing pipelines within Makim configuration files.

The core objectives of this project are as follows:

1. Change from dependencies to hooks: Instead of dependencies that just define pre-run targets, hooks would allow pre-run (setup) and post-run (teardown).

2. Pipeline Definition: Extend Makim configuration file (YAML format) to include a dedicated section for defining pipelines. Pipelines will consist of a sequence of steps, where each step can be associated with any existing Makim target.

3. Pipeline Execution: Implement a pipeline execution mechanism within Makim, allowing users to run defined pipelines using a simple command-line interface. Pipelines should support both linear and branching flows, enabling complex task orchestration.

4. Pipeline Visualization: Integrate a graph visualization tool, such as asciinet, to allow users to view the structure and dependencies of defined pipelines. This feature will enhance transparency and aid in debugging complex workflows.

5. Documentation: Update Makim's documentation to include comprehensive guidance on defining and executing pipelines. Provide examples and best practices for creating efficient and maintainable pipeline configurations.

Current State

Makim is very well structured in order to allow the inclusion of the pipelines support.

Tasks

• https://github.com/osl-incubator/makim/issues/75
• https://github.com/osl-incubator/makim/issues/26

Expected Outcomes

• Support for pipelines with Makim
• The package on conda-forge should be updated
• The creation of a blog post about pipelines with Makim
• Documentation should be updated
• Add the correspondent tests (unit test, and smoke tests) on CI

Details

• Prerequisites:
  • Python
  • Object-oriented programming (OOP)
  • YAML
  • basic concepts about pipelines between commands
• Expected Time: 350 hours
• Potential Mentor(s): Ivan Ogasawara, Sandro Loch

References

• https://www.gnu.org/software/bash/manual/html_node/Pipelines.html
• https://airflow.apache.org/docs/apache-airflow/stable/index.html
• https://www.mkdocs.org/
• https://squidfunk.github.io/mkdocs-material/