Standalone cli tool able to transform SLO definitions into prom rules

[mmazur]

I'd like to be able to use pyrra's functionality in an environment that does not match how pyrra is supposed to be deployed. To that end I'm interested in the following feature: a build of (parts of) pyrra as a standalone CLI binary capable (at least initially) of transforming files containing SLO definitions into files containing prom rules.

Example usage:
pyrratool slo2prom -i slo.yaml -o promrules.yaml

[metalmatze]

Hey. Excellent idea.

In the end this tool doesn't even need to be standalone. We can put that functionality into the pyrra binary itself.
pyrra generate my-slo.yaml > my-slo-output.yaml for example.

I'd start by adding a new sub command next to the existing ones called generate:

pyrra/main.go

Lines 113 to 114 in 3a58b4e

	case "kubernetes":
	code = cmdKubernetes(logger, CLI.Kubernetes.MetricsAddr, CLI.Kubernetes.ConfigMapMode)

The first argument would be a relative file path that needs to be loaded and unmarshalled:

pyrra/filesystem.go

Lines 158 to 203 in 3a58b4e

	bytes, err := ioutil.ReadFile(f)
	if err != nil {
	reconcilesErrors.Inc()
	return fmt.Errorf("failed to read file %q: %w", f, err)
	}
	var config v1alpha1.ServiceLevelObjective
	if err := yaml.UnmarshalStrict(bytes, &config); err != nil {
	reconcilesErrors.Inc()
	return fmt.Errorf("failed to unmarshal objective %q: %w", f, err)
	}
	objective, err := config.Internal()
	if err != nil {
	reconcilesErrors.Inc()
	return fmt.Errorf("failed to get objective: %w", err)
	}
	increases, err := objective.IncreaseRules()
	if err != nil {
	reconcilesErrors.Inc()
	return fmt.Errorf("failed to get increase rules: %w", err)
	}
	burnrates, err := objective.Burnrates()
	if err != nil {
	reconcilesErrors.Inc()
	return fmt.Errorf("failed to get burn rate rules: %w", err)
	}
	rule := monitoringv1.PrometheusRuleSpec{
	Groups: []monitoringv1.RuleGroup{increases, burnrates},
	}
	bytes, err = yaml.Marshal(rule)
	if err != nil {
	reconcilesErrors.Inc()
	return fmt.Errorf("failed to marshal recording rules: %w", err)
	}
	_, file := filepath.Split(f)
	path := filepath.Join(prometheusFolder, file)
	if err := ioutil.WriteFile(path, bytes, 0o644); err != nil {
	reconcilesErrors.Inc()
	return fmt.Errorf("failed to write file %q: %w", path, err)
	}

This should be a lot of similar code that can be reused, probably even be refactored into a function such that both filesystem and generate can use the same code.

PS: For a year now, I wanted to finally rewrite promtools.dev to use the same Pyrra generator rather than the previous slo-libsonnet... Hopefully I get to that one day, so that the website is equally helpful.

[Gaikanomer9]

Hey @ArthurSens , sorry didn't notice you've already opened the PR for that because that was in my drafts for about a week. I can close mine because our implementations are quite similar.

[ArthurSens]

Hey @ArthurSens , sorry didn't notice you've already opened the PR for that because that was in my drafts for about a week. I can close mine because our implementations are quite similar.

Heeey, funny how we came up with the same solution 🤣. Since we both did the same thing, I don't mind which PR is merged...

[hsolberg]

This looks very interesting for a similar use-case we have with a dedicated prometheus server. Would #361 result in a dedicated binary that we could use in Ansible playbooks?

Just to clarify, this is our use-case. We have OpenShift with a Prometheus server outside of the cluster looking in. We used the example in #219 to depoly Pyrra in OpenShift with some minor tweaks to namespace and resource specs. Prometheus is unable to pick up the recording and alerting rules since they exist inside the cluster and are expected to be picked up by a prometheus-operator. The alternative of creating a configmap doesn't change this as we need to get the configmaps onto the Prometheus server under the expected rules folder and trigger a configuration reload.

If #361 results in a dedicated binary then we could create a playbook with ansible to generate the files based of the slo.yaml and send the files to the Prometheus server and trigger a configuration reload. Is there a better way of solving this that I'm overlooking?

[ArthurSens]

If #361 results in a dedicated binary then we could create a playbook with ansible to generate the files based of the slo.yaml and send the files to the Prometheus server and trigger a configuration reload. Is there a better way of solving this that I'm overlooking?

Hey, it is not a whole new CLI. It is the same pyrra CLI, but with a new command 🙂.

It has been years since I tried Ansible playbooks so I can't tell if it works with Ansible or not, but you can test the binary yourself by running:

git clone -b ArthurSens:arthursens/standalone-cli-tool-314 https://github.com/ArthurSens/pyrra
cd pyrra
make ui/node_modules ui/build  build

You can run ./pyrra generate --help to get more information about the command then

[hsolberg]

If #361 results in a dedicated binary then we could create a playbook with ansible to generate the files based of the slo.yaml and send the files to the Prometheus server and trigger a configuration reload. Is there a better way of solving this that I'm overlooking?

Hey, it is not a whole new CLI. It is the same pyrra CLI, but with a new command slightly_smiling_face.

It has been years since I tried Ansible playbooks so I can't tell if it works with Ansible or not, but you can test the binary yourself by running:

git clone -b ArthurSens:arthursens/standalone-cli-tool-314 https://github.com/ArthurSens/pyrra
cd pyrra
make ui/node_modules ui/build  build

You can run ./pyrra generate --help to get more information about the command then

Thanks, worked great! Only one issue, doesn't seem to handle more than one entry per yaml-file. Tried putting 3 SLO-entries in the same yaml-file and only the first entry was generated. If I split it up into 3 separate files and run the command with a wildcard to target all 3 files then it works fine.

Example with all 3 entries in one file below:

---
apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: prometheus-operator-http
  namespace: pyrra
  labels:
    prometheus: k8s
    role: alert-rules
spec:
  target: '95.0'
  window: 7d
  indicator:
    ratio:
      errors:
        metric: prometheus_operator_kubernetes_client_http_requests_total{status_code=~"5.."}
      total:
        metric: prometheus_operator_kubernetes_client_http_requests_total
---
apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  labels:
    prometheus: k8s
    role: alert-rules
  name: apiserver-read-response-errors
  namespace: pyrra
spec:
  description: ""
  indicator:
    ratio:
      errors:
        metric: apiserver_request_total{job="apiserver",verb=~"LIST|GET",code=~"5.."}
      total:
        metric: apiserver_request_total{job="apiserver",verb=~"LIST|GET"}
  target: "99.99"
  window: 2w
---
apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  labels:
    prometheus: k8s
    role: alert-rules
  name: apiserver-write-response-errors
  namespace: pyrra
spec:
  description: ""
  indicator:
    ratio:
      errors:
        metric: apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE",code=~"5.."}
      total:
        metric: apiserver_request_total{job="apiserver",verb=~"POST|PUT|PATCH|DELETE"}
  target: "99"
  window: 1w

[ArthurSens]

We're evaluating using a SaaS service for prometheus and just noticed how this issue would be useful for this scenario. Using a SaaS service for metrics storage and rule evaluation makes us unable to use prometheus-operator's PromRule CRD.

If pyrra can be used as a CLI to transform SLO definitions into rule files, we can use CI to upload the rule definitions into the SaaS offering. Pyrra UI would query this same SaaS, where the recording rules are being evaluated 🙂

[metalmatze]

Sounds like a plan, yes!
We should rebase and update #361 and I'm happy to still have that component 👍