Docker Task Manager

Overview

Docker Task Manager is a Python-based system that manages and distributes tasks across multiple Docker containers asynchronously via HTTP. The system ensures efficient workload distribution, handles failures, retries failed tasks, enforces execution timeouts, and supports logging and health checks.

Features

• Concurrent Execution: Run multiple Docker containers in parallel and distribute configurations asynchronously.
• JSON Communication: All configuration inputs and simulation outputs use JSON format.
• Timeout & Retry Mechanisms: Configurable container timeouts and error handling that distinguishes between configuration issues and system failures.
• Priority-Based Distribution: Uses a load manager that prioritizes configuration assignment based on a FIFO or custom priority logic.
• Dynamic Container Management: Manages container workers using the Docker Python SDK for launching and monitoring without Docker Compose.
• Health Monitoring: Provides streaming logs and health check endpoints to monitor container status in real time.
• Persistent Storage: Uses a JSON file with file locking to safely record simulation results and worker statuses.

How to use

1. run make install to install necessary requirements.

import container_management as cm
cm.build()
supervisor = cm.Supervisor()
supervisor.run()