Immutable platform - all as a code approach

Purpose

Yet another story about kubernetes and declarative approach to infrastructure, deployments, immutability and development.

... being more verbose ... fully automated kubernetes environment based on FaaS to be run on local environment, virtual machines or in cloud based leveraging nixos and nixpkgs ecosystem. This is an example architecture how things can be modeled in fully reproducible manner, be language agnostic and provide full testing ability of infrastructure as well as on application level following gitops way.

Preparation

• get docker for minikube.
• get nix - creating isolated local environment and package management

Quick start

• if you did all steps from preparatioin, navigate to development/minikube folder ... after that direnv should take control and spawn local environment just for you

pro tip: in very first time it would be better if you will leave your computer for a while ... it takes time

Inspiration part

• brigade js in action
• knative
• knative comprehensive feature overview
• brigade & virtual-kubelet
• gitops
• argo cd
• nix - ecosystem, features overview, kubernetes
• nix - sales pitch
• docker images without docker - to get general idea of immutability

What is super hot!

• development with skaffold
• gitops - infrastructure and applications described as generated from nix yamls and stored in git
• full determinism of results
• monitoring tools with predefined dashboards
• scale pods to 0 with knative & istio, scale based on concurrency level or resources level
• fully declarative descriptor of environment to provision local env, virtual machine as well as clouds based on nixpkgs, nixops and nixOS
• building docker without daemon with nix
• distributed storage with rook-ceph and backups with restic and velero
• private nix binary store and cache - to speed up spawning local env and speed up kubernetes docker image building
• great experience development with lorri and direnv
• private binary channel

Tutorials

• knative
• nix

... and more

• pure nix solution - there is no any yaml file related to descriptor docker, kubernetes or helm
• nix in charge of building and pushing docker images to docker repository
• full composability of components and configs
• all parts of project are sharable - nix is everywhere, in local env, ci worker or at system level - all scripts and libraries can be used in every context
• incremental builds! - if there were no change, artifact, docker or any other thing won't be builded
• helm charts without helm and tiller
• diverged targeted builds - darwin and linux in the same time within nested closures - required for local docker provisioning
• distributed build cache and sharing intermediate states between builds - remote stores to speed up provisioning and ci results - work in progress
• nixops is provisioning ec2 or virtualbox instances based upon declarative nix file
• custom tool to manage remote state for deployments called remote-state (check infra/shell.nix for usage or it's docs)
• terraform provisioning of infra and deployment of nixos configurations with 4 simple steps
• follows and assumes https://12factor.net/
• conftest & opa for resource validation, access validation with istio

Build status

• hydra instance

How to start

Start from start guide first.

How to connect the dots

• interactive mode

People are doing it already ...

• step towards future configuration infrastructure management nix
• how we use nix at iohk

Tools comparison

• ansible vs nix
• docker vs nix

Good to familiar with

• knative with knctl
• docker-containers
• nixos container
• distributed builds
• nix & docker

Some articles which were helpful down the road

• nix & concourse
• nix & kubernetes

Docs

• Stack - tools and such
• How gitops work
• How brigade work
• How cache is handled
• How to debug
• How to setup local development
• What kind of errors you can expect
• How secrets are handled
• What is the technology stack
• Some tips and tricks
• Where I'm and where I want to be
• Some alternative approaches
• Cluster monitoring
• Some good reads
• nix-darwin and remote-builders
• nix-channels
• Build go package
• What I have learnt down the road

Monitoring

• grafana
• weavescope
• zipkin

Gitops

• ifra repo
• argo-cd