Walkthrough:
API request [1] arrives. The request is passed to one of the servers [2]. Assuming the request is a job submission, the job is registered with the redis db [3], and the inputs are copied to external storage [4][*], and the job is then put on the queue. A worker [5] will then take the job off the queue, create a docker container to run the job, copy inputs[*], execute the container. When the job is finished, outputs[*] are copied to external storage [4], and the job is finished and removed from the queue.
In the background, lambdas [6] monitor the workers and the queue, and will scale up/down as appropriate.
The CCC stack does not have any access control itself, so it is the responsibility of the person deploying CCC to control access.
By default, there is no difference between servers that process API requests, and workers that process jobs, the same server process handles both.
However, if the env var DISABLE_WORKER=true is passed to the server process, then processing jobs is disabled. This reduces the attack surface of the stack as the docker daemon is not mounted by servers. Servers and workers do not interact directly, they both communicate via redis.
This would mean you need auto-scaling groups: one for the servers, and one for the workers:
There are two types of jobs
- Standard jobs, where inputs and outputs are copied to external storage. These jobs can be long running. A submission request is returned immediately with the job id, this is used to query the job status via HTTP requests or a websocket.
- Fast jobs. External storage is ignored, inputs are copied into Redis (so cannot be huge), and the jobs are ahead of standard jobs in the queue. The request response is returned when the job is complete. Since there is no durability of inputs/outputs, if a job fails, the querying application should retry.
Stores:
- queues
- jobs
- job inputs/outputs
- worker health status
- job statistics
It contains the state of the application, and if it goes down, the stack will not work.
E.g. S3. Job inputs and outputs are stored here.
A worker consists of simply a machine with docker installed. The worker process runs as a privileged docker container that listens to the redis queue. When a job is taken off the queue, a container is spun up, inputs copied, docker container is executed, outputs copied, and the queue is notified the job is complete.
Because the Autoscaling group control of scaling lacks the ability to look at the redis queue, AWS lambdas periodically check the redis queue, and adjust workers up or down depending on various factors (queue size/state, time workers have been up, worker health).
The lambdas also check the worker health, by checking the redis db for a key that matches the worker. Workers periodically update this key with their health status. If the key is missing, or the value is not 'HEALTHY' then the worker is terminated. If the autoscaling group (ASG) minimum is not fulfilled, the ASG will create a fresh worker.
Both scale-up and scale-down lambdas perform this health check. Scale up checks occur frequently (1/minute) so the stack is responsive, scale down occurs les frequently (every 15/30m).