Loki Canary should have a better way to be suspended

[shokada]

For details, please refer to #1695

[slim-bean]

Currently we have a script that iterates over the kubernetes contexts on your machine and sends a SIGINT to all the canaries, this is both slow and error prone depending on who runs it and what contexts they have access to:

#!/bin/bash

function sigint_all_canaries {
    for POD in $(kubectl --context=${1} --namespace=default get pod -l "name=loki-canary" -o name | sed 's/pod\///')
    do
        echo Suspending $POD in ${1}
        kubectl --context=${1} --namespace=default exec $POD -- kill -2 1
    done
}


function usage() {
    echo
    echo "$0: sends a SIGINT to all loki-canaries in a cluster causing them to suspend operations until the pod is restarted"
    echo
    echo "arguments:"
    echo "all              iterate through all locally defined CONTEXTS using the command: kubectl config get-contexts -o name"
    echo "showall          print the output of: kubectl config get-contexts -o name and exit"
    echo "[context_name]   specify an individual context"
}

[ -z "$1" ] && usage && exit


case ${1} in
all)
  for CONTEXT in $(kubectl config get-contexts -o name)
  do
	echo Suspending canaries in context: ${CONTEXT}
	sigint_all_canaries ${CONTEXT}
  done
  ;;
showall)
  kubectl config get-contexts -o name
  ;;
*)
  echo Suspending canaries in context: ${1}
  sigint_all_canaries ${1}
  ;;
esac

I'm not sure the best way to handle this, it would be nice to be able to do it from a centralized manner, even from Loki, but the challenge here is usually that when you want to shut down the canaries it's because Loki or its supporting infrastructure is in trouble and it would be hard to guarantee you could make this work.

I still wonder though if maybe we should have both promtail and the canaries have behaviors to backoff connections when receiving specific http response codes, or even have the canaries shutdown on a specific http response code?

[shokada]

In a kubernetes environment there is no easy way to deschedule a daemonset and killing a pod will restart it so that's why this exisits.

There seems to be a way to scale down the daemonset using the nodeSelector with any non-existing label.
https://stackoverflow.com/a/57533340

What do you think about this way?

[joe-elliott]

I'm proposing adding an http endpoint /suspend that indefinitely stops all canary processes.