kubernetes support v0

[DominicBurkart]

✨ 🛰️ 🤖 ☸️ kubernetes support ☸️ 🤖 🛰️ ✨

Aside from implementing k8s support, this PR changes some of the main abstractions and better defines roles in the test suite. Full description below.

k8s support example

#[macro_use]
extern crate lazy_static;
#[macro_use(c)]
extern crate cute;
use futures::future::try_join_all;
use rand::{thread_rng, Rng};
use tokio::sync::Mutex;
use tracing;
use tracing_subscriber;

use turbolift::kubernetes::K8s;
use turbolift::on;

/// instantiate the global cluster manager
lazy_static! {
    static ref K8S: Mutex<K8s> = Mutex::new(K8s::with_deploy_function_and_max_replicas(
        Box::new(load_container_into_kind),
        2
    ));
}

/// The application writer is responsible for placing
/// images where your cluster can access them. The
/// K8s constructor has a parameter which takes
/// a function that is called after the container is
/// built, so that the container may be added to a
/// specific registry or otherwise be made available.
fn load_container_into_kind(tag: &str) -> anyhow::Result<&str> {
    std::process::Command::new("kind")
        .args(format!("load docker-image {}", tag).as_str().split(' '))
        .status()?;
    Ok(tag)
}

/// This is the function we want to distribute!
#[on(K8S)]
fn square(u: u64) -> u64 {
    u * u
}

fn random_numbers() -> Vec<u64> {
    let mut pseud = thread_rng();
    c![pseud.gen_range(0, 1000), for _i in 1..10]
}

fn main() {
    // use tracing.rs to print info about the program to stdout
    tracing_subscriber::fmt()
        .with_max_level(tracing::Level::TRACE)
        .with_writer(std::io::stdout)
        .init();

    let input = random_numbers();
    let futures = c![square(*int), for int in &input];
    let mut rt = tokio::runtime::Runtime::new().unwrap();
    let output = rt.block_on(try_join_all(futures)).unwrap();
    println!(
        "\n\ncomputation complete.\ninput: {:?}\noutput: {:?}",
        input, output
    );
    if output != c![x*x, for x in input] {
        std::process::exit(1)
    }
}

k8s support technical overview

• With the distributed feature flag, a docker file for each microservice is generated at compile time on the host machine. This file contains the extracted function source code embedded as a tar. The file is then embedded directly into the output binary. FEATURE SHIFT: because the tar can be really big, it doesn't make sense to try to pipe it directly to docker build via echo '[docker file contents]' | docker build -t [function name] - (there are pipe errors and weird shell errors about the maximum allowed size of literals / vars). To avoid these issues, I just make a regular directory that contains the tar and dockerfile, and then delete it later.
• At runtime, when declare is run (once), the main program connects to the cluster. The main program starts a local registry from the docker registry image, and tags each microservice image in the registry (todo: how do all nodes in the cluster get access to this registry?). The main program submits a pod with the relevant image to the cluster and records its address and port.
• At runtime, when dispatch is run (each time the target function is called), the main program makes an http request to the relevant pod and awaits the response.

testing

k8s support is tested using a new example project that runs in CI using a minimal microk8s setup.

other changes

• makes declare on DistributionPlatform trait async. This is nice because the network request to connect to a cluster may take some time, so having a non-blocking API here will be useful for any clients with an async environment. This will also be useful for any other distributed platforms. Since declare was already async and the on macro already handles the async environment for declare, this doesn't change much for end users. 5b1f407
• test scheme change. The docker suite is now responsible for running example tests, and the rust test suite is responsible for lints & static analysis (clippy, fmt, cargo check). Before, the example tests were duplicated in the rust test suite. feacdd8
• fixed bug where --features "distributed" silently built local version. 0682c58 (secondary bugs: 40fcbd2)
• removed incorrect feature-gating of actix-web in turbolift. 0682c58
• removed unimplemented "service" feature since it was muddying codebase and doesn't work. 35d81b2
• don't try to memoize platform declaration; force each platform implementation to define how to keep track of whether a function has been declared, so that we can choose when to declare. f550673
• use microk8s in the CI, since it makes setting up a local registry easy and is a good example k8s implementation.
• use whether project is being built with --release or not to determine if microservices should be built with --release or not. caa1a09
• add tracing support and use tracing in the examples c48813e. This was necessary while debugging where the program was erroring.
• made the interface for instantiating K8s more complicated. We need the user to be in charge of getting the images from their local docker to a place accessible by their cluster, since there isn't a free way to do this without doing fancy https stuff for an internal registry. 71d3245

next steps

• add support for custom configs via kube's config model (with a K8s::from_config function or similar)
• microservice containers should use the same operating system as the compilation target.
• when creating an environment to build the docker image for each microservice, we should use an in-memory temp filesystem, instead of creating a directory in the normal filesystem and then deleting it later.
• globally, we need a smarter pattern for selecting available ports.
• the local docker registry used by k8s should be ephemeral and only tied to the current run of the application. any already active registries should be ignored and the registry should be removed when the application completes.
• pods & services should be monitored for error and validated before declare is allowed to return.
• currently, autoscale is constrained to a max of 4 pods. This should be user-defined and declared either when a specific function is flagged or when the k8s object is initialized.
• currently, turbolift's k8s support only works with the en0 or eth0 interface. For devices with multiple network interfaces (en1, en2, ..., eth1, eth2, ..., and other kinds of network interfaces), we choose the first en0 or eth0 interface and ignore all other interfaces while sharing information from the across the local network. Instead, we should allow users to choose the relevant network, or infer it based on what the k8s instance is using.
• figure out why we need to use a new runtime when expending futures in async traits: 39fffe1 (discussion here: kubernetes support v0 #2 (comment))
• roadmap an observability solution. Right now we don't have a way to transmit the tracing information from the microservices to the main application, which means all of the distributed logic is invisible. There are some potential solutions, but I need to spend time to figure out the best pattern (maybe return some kind of object that magically inserts the trace into the trace for the main application?)

[DominicBurkart]

This is the worst PR I have ever written, but I'm happy with where the code is now vs when it was opened! May [git] history be my judge.