Detectors for Prometheus varnish exporter (#475)

* Bootstrap module and README

* Add Backend Failed detector

* Add Thread Number detector

* Add Dropped Sessions detector

* Add Hit Rate detector

* Add Memory Usage detector

* Update readme

* Update docs

---------

Co-authored-by: R.Sicart <roger.sicart@fr.clara.net>

docs/severity.md

@@ -78,6 +78,7 @@
 - [prometheus-exporter_kong](#prometheus-exporter_kong)
 - [prometheus-exporter_oracledb](#prometheus-exporter_oracledb)
 - [prometheus-exporter_squid](#prometheus-exporter_squid)
+- [prometheus-exporter_varnish](#prometheus-exporter_varnish)
 - [prometheus-exporter_wallix-bastion](#prometheus-exporter_wallix-bastion)
 - [smart-agent_apache](#smart-agent_apache)
 - [smart-agent_cassandra-nodetool](#smart-agent_cassandra-nodetool)
@@ -858,6 +859,18 @@
 |Squid total amount of requests|X|-|-|-|-|
 
 
+## prometheus-exporter_varnish
+
+|Detector|Critical|Major|Minor|Warning|Info|
+|---|---|---|---|---|---|
+|Varnish heartbeat|X|-|-|-|-|
+|Varnish backend failed|X|-|-|-|-|
+|Varnish thread number|X|-|-|-|-|
+|Varnish dropped sessions|X|-|-|-|-|
+|Varnish hit rate|-|X|X|-|-|
+|Varnish memory usage|X|X|-|-|-|
+
+
 ## prometheus-exporter_wallix-bastion
 
 |Detector|Critical|Major|Minor|Warning|Info|

modules/prometheus-exporter_varnish/README.md

@@ -0,0 +1,146 @@
+# VARNISH SignalFx detectors
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+:link: **Contents**
+
+- [How to use this module?](#how-to-use-this-module)
+- [What are the available detectors in this module?](#what-are-the-available-detectors-in-this-module)
+- [How to collect required metrics?](#how-to-collect-required-metrics)
+ - [Examples](#examples)
+ - [Metrics](#metrics)
+- [Related documentation](#related-documentation)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+## How to use this module?
+
+This directory defines a [Terraform](https://www.terraform.io/)
+[module](https://www.terraform.io/language/modules/syntax) you can use in your
+existing [stack](https://github.com/claranet/terraform-signalfx-detectors/wiki/Getting-started#stack) by adding a
+`module` configuration and setting its `source` parameter to URL of this folder:
+
+```hcl
+module "signalfx-detectors-prometheus-exporter-varnish" {
+ source = "github.com/claranet/terraform-signalfx-detectors.git//modules/prometheus-exporter_varnish?ref={revision}"
+
+ environment = var.environment
+ notifications = local.notifications
+}
+```
+
+Note the following parameters:
+
+* `source`: Use this parameter to specify the URL of the module. The double slash (`//`) is intentional and required.
+ Terraform uses it to specify subfolders within a Git repo (see [module
+ sources](https://www.terraform.io/language/modules/sources)). The `ref` parameter specifies a specific Git tag in
+ this repository. It is recommended to use the latest "pinned" version in place of `{revision}`. Avoid using a branch
+ like `master` except for testing purpose. Note that every modules in this repository are available on the Terraform
+ [registry](https://registry.terraform.io/modules/claranet/detectors/signalfx) and we recommend using it as source
+ instead of `git` which is more flexible but less future-proof.
+
+* `environment`: Use this parameter to specify the
+ [environment](https://github.com/claranet/terraform-signalfx-detectors/wiki/Getting-started#environment) used by this
+ instance of the module.
+ Its value will be added to the `prefixes` list at the start of the [detector
+ name](https://github.com/claranet/terraform-signalfx-detectors/wiki/Templating#example).
+ In general, it will also be used in the `filtering` internal sub-module to [apply
+ filters](https://github.com/claranet/terraform-signalfx-detectors/wiki/Guidance#filtering) based on our default
+ [tagging convention](https://github.com/claranet/terraform-signalfx-detectors/wiki/Tagging-convention) by default.
+
+* `notifications`: Use this parameter to define where alerts should be sent depending on their severity. It consists
+ of a Terraform [object](https://www.terraform.io/language/expressions/type-constraints#object) where each key represents an available
+ [detector rule severity](https://docs.splunk.com/observability/alerts-detectors-notifications/create-detectors-for-alerts.html#severity)
+ and its value is a list of recipients. Every recipients must respect the [detector notification
+ format](https://registry.terraform.io/providers/splunk-terraform/signalfx/latest/docs/resources/detector#notification-format).
+ Check the [notification binding](https://github.com/claranet/terraform-signalfx-detectors/wiki/Notifications-binding)
+ documentation to understand the recommended role of each severity.
+
+These 3 parameters alongs with all variables defined in [common-variables.tf](common-variables.tf) are common to all
+[modules](../) in this repository. Other variables, specific to this module, are available in
+[variables-gen.tf](variables-gen.tf).
+In general, the default configuration "works" but all of these Terraform
+[variables](https://www.terraform.io/language/values/variables) make it possible to
+customize the detectors behavior to better fit your needs.
+
+Most of them represent usual tips and rules detailled in the
+[guidance](https://github.com/claranet/terraform-signalfx-detectors/wiki/Guidance) documentation and listed in the
+common [variables](https://github.com/claranet/terraform-signalfx-detectors/wiki/Variables) dedicated documentation.
+
+Feel free to explore the [wiki](https://github.com/claranet/terraform-signalfx-detectors/wiki) for more information about
+general usage of this repository.
+
+## What are the available detectors in this module?
+
+This module creates the following SignalFx detectors which could contain one or multiple alerting rules:
+
+|Detector|Critical|Major|Minor|Warning|Info|
+|---|---|---|---|---|---|
+|Varnish heartbeat|X|-|-|-|-|
+|Varnish backend failed|X|-|-|-|-|
+|Varnish thread number|X|-|-|-|-|
+|Varnish dropped sessions|X|-|-|-|-|
+|Varnish hit rate|-|X|X|-|-|
+|Varnish memory usage|X|X|-|-|-|
+
+## How to collect required metrics?
+
+This module deploys detectors using metrics reported by the
+scraping of a server following the [OpenMetrics convention](https://openmetrics.io/) based on and compatible with [the Prometheus
+exposition format](https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#openmetrics-text-format).
+
+They are generally called `Prometheus Exporters` which can be fetched by both the [SignalFx Smart Agent](https://github.com/signalfx/signalfx-agent)
+thanks to its [prometheus exporter monitor](https://github.com/signalfx/signalfx-agent/blob/main/docs/monitors/prometheus-exporter.md) and the
+[OpenTelemetry Collector](https://github.com/signalfx/splunk-otel-collector) using its [prometheus
+receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver) or its derivates.
+
+These exporters could be embedded directly in the tool you want to monitor (e.g. nginx ingress) or must be installed next to it as
+a separate program configured to connect, create metrics and expose them as server.
+
+
+Check the [Related documentation](#related-documentation) section for more detailed and specific information about this module dependencies.
+
+The detectors of this module uses metrics from the [prometheus varnish exporter](https://github.com/jonnenauha/prometheus_varnish_exporter) plugin for Prometheus.
+
+This exporter is not an official Prometheus exporter.
+
+### Examples
+
+Sample OTEL Agent configuration snippet:
+
+```yaml
+receivers:
+ smartagent/prometheus-exporter:
+ type: prometheus-exporter
+ host: localhost
+ port: 9131
+service:
+ pipelines:
+ metrics:
+ receivers:
+ - smartagent/prometheus-exporter
+```
+
+
+### Metrics
+
+
+Here is the list of required metrics for detectors in this module.
+
+* `varnish_backend_fail`
+* `varnish_main_cache_hit`
+* `varnish_main_cache_miss`
+* `varnish_main_sessions`
+* `varnish_main_threads`
+* `varnish_sma_g_bytes`
+* `varnish_sma_g_space`
+
+
+
+
+## Related documentation
+
+* [Terraform SignalFx provider](https://registry.terraform.io/providers/splunk-terraform/signalfx/latest/docs)
+* [Terraform SignalFx detector](https://registry.terraform.io/providers/splunk-terraform/signalfx/latest/docs/resources/detector)
+* [Splunk Observability integrations](https://docs.splunk.com/Observability/gdi/get-data-in/integrations.html)
+* [Prometheus Exporter for Varnish](https://github.com/jonnenauha/prometheus_varnish_exporter)

modules/prometheus-exporter_varnish/common-filters.tf

@@ -0,0 +1 @@
+../../common/module/filters-prometheus-exporter.tf

modules/prometheus-exporter_varnish/common-locals.tf

@@ -0,0 +1 @@
+../../common/module/locals.tf

modules/prometheus-exporter_varnish/common-modules.tf

@@ -0,0 +1 @@
+../../common/module/modules.tf

modules/prometheus-exporter_varnish/common-variables.tf

@@ -0,0 +1 @@
+../../common/module/variables.tf

modules/prometheus-exporter_varnish/common-versions.tf

@@ -0,0 +1 @@
+../../common/module/versions.tf

modules/prometheus-exporter_varnish/conf/00-heartbeat.yaml

@@ -0,0 +1,14 @@
+module: varnish
+name: heartbeat
+
+transformation: false
+aggregation: true
+exclude_not_running_vm: true
+
+filtering: "filter('system.type', 'prometheus-exporter')"
+
+signals:
+ signal:
+ metric: "varnish_main_threads"
+rules:
+ critical:

modules/prometheus-exporter_varnish/conf/01-backend-failed.yaml

@@ -0,0 +1,16 @@
+module: varnish
+name: Backend Failed
+
+transformation: ".min(over='5m')"
+aggregation: true
+
+filtering: "filter('system.type', 'prometheus-exporter')"
+
+signals:
+ signal:
+ metric: varnish_backend_fail
+ rollup: delta
+rules:
+ critical:
+ threshold: 0
+ comparator: ">"

modules/prometheus-exporter_varnish/conf/02-thread-number.yaml

@@ -0,0 +1,15 @@
+module: varnish
+name: Thread number
+
+transformation: ".min(over='5m')"
+aggregation: true
+
+filtering: "filter('system.type', 'prometheus-exporter')"
+
+signals:
+ signal:
+ metric: varnish_main_threads
+rules:
+ critical:
+ threshold: 1
+ comparator: "<"

modules/prometheus-exporter_varnish/conf/03-dropped-sessions.yaml

@@ -0,0 +1,17 @@
+module: varnish
+name: Dropped Sessions
+
+transformation: ".min(over='5m')"
+aggregation: true
+
+filtering: "filter('system.type', 'prometheus-exporter')"
+
+signals:
+ signal:
+ metric: varnish_main_sessions
+ rollup: delta
+ filter: "filter('type', 'dropped')"
+rules:
+ critical:
+ threshold: 0
+ comparator: ">"

modules/prometheus-exporter_varnish/conf/04-hit-rate.yaml

@@ -0,0 +1,25 @@
+module: varnish
+name: Hit Rate
+
+transformation: ".min(over='5m')"
+aggregation: true
+
+filtering: "filter('system.type', 'prometheus-exporter')"
+
+signals:
+ A:
+ metric: varnish_main_cache_hit
+ B:
+ metric: varnish_main_cache_miss
+ signal:
+ formula: (A/(A+B)).fill(0).scale(100)
+rules:
+ minor:
+ threshold: 90
+ comparator: "<"
+ lasting_duration: "5m"
+ major:
+ threshold: 80
+ comparator: "<="
+ lasting_duration: "5m"
+ dependency: minor

modules/prometheus-exporter_varnish/conf/05-memory-usage.yaml

@@ -0,0 +1,27 @@
+module: varnish
+name: Memory Usage
+
+transformation: ".min(over='5m')"
+aggregation: true
+
+filtering: "filter('system.type', 'prometheus-exporter')"
+
+signals:
+ A:
+ metric: varnish_sma_g_bytes
+ filter: "filter('type', 's0')"
+ B:
+ metric: varnish_sma_g_space
+ filter: "filter('type', 's0')"
+ signal:
+ formula: (A / (A+B)).scale(100).fill(0)
+rules:
+ critical:
+ threshold: 90
+ comparator: ">"
+ lasting_duration: "5m"
+ major:
+ threshold: 80
+ comparator: ">"
+ lasting_duration: "5m"
+ dependency: critical

modules/prometheus-exporter_varnish/conf/readme.yaml

@@ -0,0 +1,25 @@
+documentations:
+ - name: Prometheus Exporter for Varnish
+ url: https://github.com/jonnenauha/prometheus_varnish_exporter
+
+source_doc: |
+ The detectors of this module uses metrics from the [prometheus varnish exporter](https://github.com/jonnenauha/prometheus_varnish_exporter) plugin for Prometheus.
+
+ This exporter is not an official Prometheus exporter.
+
+ ### Examples
+
+ Sample OTEL Agent configuration snippet:
+
+ ```yaml
+ receivers:
+ smartagent/prometheus-exporter:
+ type: prometheus-exporter
+ host: localhost
+ port: 9131
+ service:
+ pipelines:
+ metrics:
+ receivers:
+ - smartagent/prometheus-exporter
+ ```