Skip to content

dotflow-io/dotflow

Repository files navigation

Website Β β€’Β  Documentation Β β€’Β  Pypi

GitHub Org's stars GitHub last commit PyPI PyPI - Python Version PyPI - Downloads

Welcome to dotflow

With Dotflow, you get a powerful and easy-to-use library designed to create execution pipelines without complication. Add tasks intuitively and control the entire process with just a few commands.

Our goal is to make task management faster and more secure, without overwhelming you with complexity. Simply instantiate the DotFlow class, add your tasks with the add method, and start execution with the start method.

Start with the basics here.

Table of Contents

Click to expand

Getting Help

We use GitHub issues for tracking bugs and feature requests and have limited bandwidth to address them. If you need anything, I ask you to please follow our templates for opening issues or discussions.

Getting Started

Install

To install Dotflow, run the following command from the command line:

With Pip

pip install dotflow

With Poetry

poetry add dotflow

A Simple Example

The simplest file could look like this:

from dotflow import DotFlow, action

def my_callback(*args, **kwargs):
    print(args, kwargs)

@action
def my_task_x():
    print("task")

@action
def my_task_y():
    print("task")

workflow = DotFlow()

workflow.task.add(step=my_task_x, callback=my_callback)
workflow.task.add(step=my_task_y, callback=my_callback)

workflow.start()

First Steps

1. Import

Start with the basics, which is importing the necessary classes and methods. (DotFlow, action)

from dotflow import DotFlow, action

2. Callback function

Create a my_callback function to receive execution information of a task. It is not necessary to include this function, as you will still have a report at the end of the execution in the instantiated object of the DotFlow class. This my_callback function is only needed if you need to do something after the execution of the task, for example: sending a message to someone, making a phone call, or sending a letter. More details

def my_callback(*args, **kwargs):
    print(args, kwargs)

3. Task function

Now, create the function responsible for executing your task. It's very simple; just use the action decorator above the function, and that's itβ€”you've created a task.

@action
def my_task_x():
    print("task")

4. DotFlow Class

Instantiate the DotFlow class in a workflow variable to be used in the following steps. More details.

workflow = DotFlow()

5. Add Task

Now, simply add the my_task_x and my_callback functions you created earlier to the workflow using the code below. This process is necessary to define which tasks will be executed and the order in which they will run. The execution order follows the sequence in which they were added to the workflow. More details

  • Adding one step at a time:
workflow.task.add(step=my_task_x, callback=my_callback)
workflow.task.add(step=my_task_y, callback=my_callback)
  • Adding multiple steps at the same time:
workflow.task.add(step=[my_task_x, my_task_y], callback=my_callback)
  • Adding a step with the module path:
workflow.task.add(step="module.task.my_task_x", callback=my_callback)

6. Start

Finally, just execute the workflow with the following code snippet. More details

workflow.start()

CLI

Simple Start

dotflow start --step examples.cli_with_mode.simple_step

With Initial Context

dotflow start --step examples.cli_with_initial_context.simple_step --initial-context abc

With Callback

dotflow start --step examples.cli_with_callback.simple_step --callback examples.cli_with_callback.callback

With Mode

dotflow start --step examples.cli_with_mode.simple_step --mode sequential
dotflow start --step examples.cli_with_mode.simple_step --mode background
dotflow start --step examples.cli_with_mode.simple_step --mode parallel

Process Mode

Sequential

workflow.task.add(step=task_foo)
workflow.task.add(step=task_bar)

workflow.start()
Click to see diagram
flowchart TD
A[Start] -->|run| B
B[task_foo] -->|response to| C
C[task_bar] -->|response| D
D[Finish]
Loading

Sequential with Groups

workflow.task.add(step=task_foo, group_name="foo")
workflow.task.add(step=task_bar, group_name="bar")

workflow.start()
Click to see diagram
flowchart TD
    A[Start] -->|run| C(Parallel Groups)
    C -->|run| D[task_a]
    C -->|run| E[task_c]
    D -->|response| X[task_b]
    X --> H[Finish]
    E -->|response| Y[task_d]
    Y --> H[Finish]
Loading

Background

workflow.task.add(step=task_foo)
workflow.task.add(step=task_bar)

workflow.start(mode="background")
Click to see diagram
flowchart TD
A[Start] -->|run| B
B[task_foo] -->|response to| C
C[task_bar] -->|response| D
D[Finish]
Loading

Parallel

workflow.task.add(step=task_a)
workflow.task.add(step=task_b)
workflow.task.add(step=task_c)
workflow.task.add(step=task_d)

workflow.start(mode="parallel")
Click to see diagram
flowchart TD
    S[Start] -->|run| A[task_a]
    S[Start] -->|run| B[task_b]
    S[Start] -->|run| C[task_c]
    S[Start] -->|run| D[task_d]
    A --> H[Finish]
    B --> H[Finish]
    C --> H[Finish]
    D --> H[Finish]
Loading

More Examples

Example Command
cli_with_callback dotflow start --step examples.cli_with_callback.simple_step --callback examples.cli_with_callback.callback
cli_with_initial_context dotflow start --step examples.cli_with_initial_context.simple_step --initial-context abc
cli_with_mode dotflow start --step examples.cli_with_mode.simple_step --mode sequential
cli_with_output_context dotflow start --step examples.cli_with_output_context.simple_step --storage file
cli_with_path dotflow start --step examples.cli_with_path.simple_step --path .storage --storage file
flow python examples/flow.py
simple_class_workflow python examples/simple_class_workflow.py
simple_cli dotflow start --step examples.simple_cli.simple_step
simple_function_workflow python examples/simple_function_workflow.py
simple_function_workflow_with_error python examples/simple_function_workflow_with_error.py
step_class_result_context python examples/step_class_result_context.py
step_class_result_storage python examples/step_class_result_storage.py
step_class_result_task python examples/step_class_result_task.py
step_function_result_context python examples/step_function_result_context.py
step_function_result_storage python examples/step_function_result_storage.py
step_function_result_task python examples/step_function_result_task.py
step_with_groups python examples/step_with_groups.py
step_with_initial_context python examples/step_with_initial_context.py
step_with_many_contexts python examples/step_with_many_contexts.py
step_with_notify_telegram python examples/step_with_notify_telegram.py
step_with_previous_context python examples/step_with_previous_context.py
step_with_storage_file python examples/step_with_storage_file.py
step_with_storage_mongodb python examples/step_with_storage_mongodb.py
workflow_background_mode python examples/workflow_background_mode.py
workflow_keep_going_true python examples/workflow_keep_going_true.py
workflow_parallel_mode python examples/workflow_parallel_mode.py
workflow_sequential_group_mode python examples/workflow_sequential_group_mode.py
workflow_sequential_mode python examples/workflow_sequential_mode.py
workflow_step_callback python examples/workflow_step_callback.py
workflow_with_backoff python examples/workflow_with_backoff.py
workflow_with_callback_failure python examples/workflow_with_callback_failure.py
workflow_with_callback_success python examples/workflow_with_callback_success.py
workflow_with_retry python examples/workflow_with_retry.py
workflow_with_retry_delay python examples/workflow_with_retry_delay.py
workflow_with_timeout python examples/workflow_with_timeout.py

Commit Style

  • βš™οΈ FEATURE
  • πŸ“ PEP8
  • πŸ“Œ ISSUE
  • πŸͺ² BUG
  • πŸ“˜ DOCS
  • πŸ“¦ PyPI
  • ❀️️ TEST
  • ⬆️ CI/CD
  • ⚠️ SECURITY

License

GitHub License

This project is licensed under the terms of the MIT License.