Skip to content

Codabench Architecture

ObadaS edited this page Sep 4, 2024 · 6 revisions

image

Source:

graph TD
A((User))
B[Caddy]
C{{Django}}
D[Postgres]
F{{Rabbit}}
G[Minio]
H[Create Buckets]
I{{Builder}}
K[Flower]
L[Compute Worker]
M[Site Worker]
A --> B
B --> C
C --> D
C --> F
F --> L
F --> M
C -->|Storage|G
G -->|Initialization Helper| H
F --> K
C -->|JS Asset Helper| I

Overview

Codabench consists of many docker containers that are connected and organized through docker compose. Below is an overview of each container and their function:

Django

The main container that runs and contains the python code (A Django project). This is the container that is mainly used for utility functions like creating admins, creating backups, and manually making changes through the python django shell. It has a gunicorn webserver that serves internally on port 8000.

Caddy

The HTTP/HTTPs web server. It acts as a reverse proxy for the Django container. This container controls SSL/HTTPs functionality and other web server configuration options. Serves on port 80/443.

Postgres (Labeled DB in docker-compose)

The default database container. It will contain the default Postgres database used by codabench. (Name/user/pass,etc determined by .env.) If you need to make manual DB changes, this is the container to look into.

Here is the DB schema codabench_db

Compute Worker

The container(s) (can be external) that runs submissions for the codabench instance on their associated queues. The default workers are tied to the default queue. Commands to re-build the compute worker Docker image can be found here.

Site Worker

The container that runs various tasks for the Django container such as unpacking and processing a competition bundle.

Minio

The default storage solution container. Runs a MinIO instance that serves on the port defined by settings in your .env.

Create Buckets

A helper container for the Minio container that initially creates the buckets defined by settings in your .env if they don't already exist. Usually this container exits after that.

Builder

A container to build RiotJS tags into a single unified tag that can then be mounted. Uses NPM.

Rabbit

Task/Message management container. Organizes queues for Celery Tasks and compute workers. Any queues that get created are accessible through rabbit's own command line interface.

Flower

Administrative utility container for Celery tasks and queues.

Competition docker image

The services of Codabench run inside Docker environments. This should not be confused with the Docker environment used to run the submissions, which may vary for each benchmark. The docker running submissions is discussed here: https://github.com/codalab/codabench/wiki/Submission-Docker-Container-Layout.

Clone this wiki locally