Skip to content

operator‐1901

Jump to bottom
Allan Roger Reid edited this page Dec 4, 2024 · 4 revisions

Steps

Create a kubernetes cluster such as in https://github.com/allanrogerr/public/wiki/vm-broker-vanilla-k8s-cluster-on-multiple-instances,-same-node

SSH into control plane

ssh -p 20014 ubuntu@1.2.3.4 -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"

Persist sessions after logout

loginctl enable-linger ubuntu

Clone operator

cd ~
git clone https://github.com/allanrogerr/operator.git
cd operator && git checkout master

Install essential packages

sudo apt-get update && sudo apt-get upgrade -y
sudo apt-get install vim -y
sudo apt-get install zip -y
sudo apt-get install make -y
sudo apt install build-essential -y

Install go

cd ~
wget https://go.dev/dl/go1.21.3.linux-amd64.tar.gz
sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.21.3.linux-amd64.tar.gz
cat <<EOF >> $HOME/.profile 
export PATH=$PATH:/usr/local/go/bin:~/go/bin
EOF
cat $HOME/.profile 
source $HOME/.profile
go version

Install nvm

sudo apt-get install nodejs -y
sudo apt-get install npm -y
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash
source ~/.bashrc
nvm install && nvm use && sudo npm install -g yarn

Install podman on each node. Push newly compile operator to docker

sudo apt-get install podman -y
sudo apt-get install podman-docker -y

podman run -d -p 5000:5000 --restart always --name registry registry:2

Note: make local registry insecure e.g. in /etc/containers/registries.conf on each node set

[[registry]]
insecure=true
location="10.62.75.170:5000"

Restart podman

podman restart registry

Edit Makefile

VERSION ?= $(shell git describe --tags --always)
VERSIONV ?= $(shell git describe --tags --always | sed 's,v,,g')

binary:
        @CGO_ENABLED=0 GOOS=linux go build -trimpath -o minio-operator ./cmd/operator

docker: operator
        @docker buildx build --no-cache --platform linux/amd64 -t $(TAG) .

Install https://github.com/mvdan/gofumpt then,

export CI=true
cd ~/operator && make assets && make build
podman tag localhost/minio/operator:dev 10.62.75.170:5000/minio/operator:0.1
podman images
podman push 10.62.75.170:5000/minio/operator:0.1 --tls-verify=false
podman search --list-tags 10.62.75.170:5000/minio/operator --tls-verify=false
or
curl -X GET http://10.62.75.170:5000/v2/_catalog

Add on each node e.g. podman pull 10.62.75.170:5000/minio/operator:0.1

Then

sudo vi /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."10.62.75.170:5000".tls] # edited line
          ca_file = "" # edited line
          cert_file = "" # edited line
          insecure_skip_verify = true # edited line
          key_file = "" # edited line

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."10.62.75.170:5000"] # edited line
          endpoint = ["http://10.62.75.170:5000"] # edited line

On all nodes

sudo systemctl restart containerd

Install mc

mkdir -p ~/mc && cd ~/mc && rm -rf mc* && wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc && cd ~

Build & install kubectl-minio plugin

cd ~/operator/kubectl-minio
CGO_ENABLED=1 go build --ldflags "-s -w" -trimpath -o ../kubectl-minio

Deploy minio

~/operator/kubectl-minio/kubectl-minio init --image 10.62.75.170:5000/minio/operator:0.1 --console-image 10.62.75.170:5000/minio/operator:0.1
SA_TOKEN=$(kubectl -n minio-operator get secret console-sa-secret -o jsonpath="{.data.token}" | base64 --decode)
echo $SA_TOKEN
kubectl patch service -n minio-operator console -p '{"spec":{"ports":[{"name": "http","port": 9090,"protocol": "TCP","nodePort":31090}],"type": "NodePort"}}'

Set default storage class

kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
kubectl get storageclass

Login

http://k8s-master.minio.training:31090/login

Test cases

FAIL EXPECTED

Test 0

Summary: As reported by user, attempt to create a tenant with 1 node and 1 drive with 1GiB storage Observe no valid parity EC:0 available image

Also observe that if an invalid Resource is specified, then the user navigates away then back to the main create page, the Create button is enabled incorrectly. image

Also observe that if multiple parities were selectable using a particular drive/server configuration e.g. 4 nodes, 8 drives (giving EC:8,7,6,5,4,3,2, THEN a 1 node 1 drive configuration were selected that those same parities EC:8,7,6,5,4,3,2 are selectable still, even though they cannot be applied to a 1 node 1 drive configuration. image

PASS EXPECTED

Implement fix

cd ~/operator && git pull && git checkout recomment-trivial-parity
cd ~/operator/kubectl-minio
CGO_ENABLED=1 go build --ldflags "-s -w" -trimpath -o ../kubectl-minio

Compile and Push newly compile operator to docker. See above instructions for building

Deploy minio

export CI=true
cd ~/operator && make assets && make build
podman tag localhost/minio/operator:dev 10.62.75.170:5000/minio/operator:0.4
podman images
podman push 10.62.75.170:5000/minio/operator:0.4 --tls-verify=false
podman search --list-tags 10.62.75.170:5000/minio/operator --tls-verify=false

Add on each node e.g. podman pull 10.62.75.170:5000/minio/operator:0.4

Deploy minio

~/operator/kubectl-minio/kubectl-minio init --image 10.62.75.170:5000/minio/operator:0.4 --console-image 10.62.75.170:5000/minio/operator:0.4
SA_TOKEN=$(kubectl -n minio-operator get secret console-sa-secret -o jsonpath="{.data.token}" | base64 --decode)
echo $SA_TOKEN
kubectl patch service -n minio-operator console -p '{"spec":{"ports":[{"name": "http","port": 9090,"protocol": "TCP","nodePort":31090}],"type": "NodePort"}}'

Test 1

Summary: As reported by user, attempt to create a tenant with 1 node and 1 drive with 1GiB storage Observe valid parity EC:0 is available image

Also observe that if an invalid Resource is specified, then the user navigates away then back to the main create page, the Create button is not enabled. image

Also observe that if multiple parities were selectable using a particular drive/server configuration e.g. 4 nodes, 8 drives (giving EC:8,7,6,5,4,3,2, THEN a 1 node 1 drive configuration were selected that those same parities EC:8,7,6,5,4,3,2 are not selectable anymore. image

Test 2

Summary: Actually create the tenant and validate its storage Port forward

kubectl --namespace myminio port-forward svc/test-hl 9000:9000 --address 0.0.0.0 &
~/mc/mc alias set operator-1901 http://127.0.0.1:9000 minioadmin minioadmin
~/mc/mc admin info --json operator-1901 | jq '.info.backend'

{
  "backendType": "Erasure",
  "onlineDisks": 1,
  "offlineDisks": 0,
  "standardSCParity": 0,
  "rrSCParity": 0,
  "totalSets": [
    1
  ],
  "totalDrivesPerSet": [
    1
  ]
}
image

Cleanup

kubectl delete ns myminio
Clone this wiki locally