Skip to content

CSCfi/grafana-prometheus-template

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
Jul 12, 2021
Jun 24, 2021
Feb 28, 2023

Repository files navigation

grafana-prometheus-template

This template deploys Prometheus and Grafana for monitoring pods running in the same Rahti namespace.

Prometheus is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts when specified conditions are observed. Grafana is open source visualization and analytics software. It allows you to query, visualize, alert on, and explore your metrics no matter where they are stored. It provides tools to turn time-series database (TSDB) data into graphs and visualizations.

You can read more about Prometheus here and about Grafana here.

Using the template with oc CLI tool

Login to Rahti

$ oc login https://rahti.csc.fi:8443 --token=<token from Rahti>

Create a new project

$ oc new-project <your Rahti project name> --description="csc_project:<your CSC project name>"

Deploy Prometheus and Grafana using the template

$ oc new-app -f <path/to/template> -p PARAM1=Value1 -p PARAM2=Value2

Parameters to be supplied

(Note: memory requests and limits should only be changed after checking project quota to avoid errors)

Parameter Description Default value
NAMESPACE The namespace Prometheus and Grafana are being deployed to (no default)
PROMETHEUS_IMAGE The location of the Prometheus image prom/prometheus:v2.27.1
PROMETHEUS_RETENTION_TIME Storage retention time for Prometheus 15d
PROMETHEUS_VOLUMESIZE Size of the persistent volume for prometheus 10Gi
PROMETHEUS_LIMITMEMORY Memory limit for Prometheus 4G
PROMETHEUS_REQMEMORY Requested memory for Prometheus 4G
BASIC_AUTH_USERNAME Username for prometheus basic auth admin
BASIC_AUTH_PASSWORD Password for prometheus basic auth (no default, generated automatically if left empty)
GRAFANA_IMAGE The location of the Grafana image grafana/grafana:7.5.7
GRAFANA_ADMIN_USERNAME Username for the Grafana admin user admin
GRAFANA_ADMIN_PASSWORD Password for the Grafana admin user (no default, generated automatically if left empty)
GRAFANA_VOLUMESIZE Size of the persistent volume for Grafana 100Mi
GRAFANA_LIMITMEMORY Memory limit for Grafana 1Gi
GRAFANA_REQMEMORY Requested memory for Grafana 512Mi

Monitoring applications

Pods running in the same namespace

Prometheus is configured by default to be able to monitor pods running in the same namespace.

You need to add the following to metadata.annotations in the deployment configs of the pods you want to monitor:

  • prometheus.io/scrape: 'true'
  • prometheus.io/path: <path> if you need to scrape metrics from a path other than /metrics
  • prometheus.io/port: <port> if you need to use a port other than the pod's declared ports

Apps outside of the namespace

Add new monitoring targets to the ConfigMap prometheus-config under scrape_configs for each application you want to monitor. After editing the ConfigMap, redeploy Prometheus to see the changes. More about configuring monitoring targets can be found here.

No authentication

If your application uses no authentication, you can add a new monitoring target with:

- job_name: '<job name>'
  static_configs:
  - targets: ['<URL>']
    labels:
      <label name>: '<label value>'
  metrics_path: '<path/to/metrics (default: /metrics)>'
  scheme: '<scheme (default: http)>'

Basic auth

If your application is using basic auth, you can use:

- job_name: '<job name>'
  static_configs:
  - targets: ['<URL>']
    labels:
      <label name>: '<label value>'
  metrics_path: '<path/to/metrics (default: /metrics)>'
  scheme: '<scheme (default: http)>'
  basic_auth:
    username: '<username>'
    password: '<password>'

It is also possible to store the password in a file. Use password_file: '<path/to/file>' and mount a secret to the Prometheus deployment with the file as data.

Token authorization

If your application uses token authorization with HTTP headers, you can use:

- job_name: '<job name>'
  static_configs:
  - targets: ['<URL>']
    labels:
      <label name>: '<label value>'
  metrics_path: '<path/to/metrics (default: /metrics)>'
  scheme: '<scheme (default: http)>'
  authorization:
    type: '<type (default: Bearer)>'
    credentials: '<secret>'

It is also possible to store the credentials in a file. Use credentials_file: '<path/to/file>' and mount a secret to the Prometheus deployment with the file as data.

IP whitelisting on Rahti

If your applications are running in Rahti, it is advisable to use IP whitelisting to allow connections to your monitoring targets from Rahti only.

On Rahti, you can do this with:

$ oc annotate route <route_name> haproxy.router.openshift.io/ip_whitelist='193.167.189.25'

On Rahti-int you can do this with:

$ oc annotate route <route_name> haproxy.router.openshift.io/ip_whitelist='193.166.25.134'

Note: Doing this will allow connections from all Rahti namespaces. Contact Rahti admins if you want a specific egress IP for your namespace.

About

This OpenShift template deploys Grafana and Prometheus

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published