- Horizontal Pod Autoscaler using Prometheus custom metrics
- Deploy the application example
- Prometheus and Grafana deployment
- Annotations for scraping application metrics
- Kubernetes HPA metrics
kubectl apply -f k8s/configmap.yaml
kubectl apply -f k8s/deploy.yaml
kubectl apply -f k8s/service.yaml
kubectl apply -f k8s/mongodb.yaml
kubectl apply -f k8s/hpa_custom_metrics.yaml
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/prometheus
Check if the service is available:
export NAMESPACE=<YOUR_NAMESPACE>
export POD_NAME=$(kubectl get pods -n $NAMESPACE -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace $NAMESPACE port-forward $POD_NAME 9090:9090
helm install prometheus-adapter --set prometheus.url=http://prometheus-server.default.svc --set prometheus.port=80 prometheus-community/prometheus-adapter
Deploy the new prometheus adapter configuration to scrape custom metrics configmap
Create a new prometheus adapter configuration configmap to scrape the new active_connections
metric from go-demo-3 application:
rules:
- seriesQuery: '{__name__= "active_connections"}'
resources:
overrides:
kubernetes_namespace:
resource: namespace
kubernetes_pod_name:
resource: pod
name:
matches: "active_connections"
as: ""
metricsQuery: <<.Series>>{<<.LabelMatchers>>,container_name!="POD"}
kubectl apply -f k8s/prometheus-adapter/prometheus-adapter-cm.yaml
helm install grafana grafana/grafana
Connect to grafana:
export NAMESPACE=<YOUR_NAMESPACE>
export POD_NAME=$(kubectl get pods -n $NAMESPACE -l "app.kubernetes.io/name=grafana" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace $NAMESPACE port-forward $POD_NAME 3000:3000
Get admin password:
export NAMESPACE=<YOUR_NAMESPACE>
kubectl get secret -n $NAMESPACE grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
Open Grafana in your browser:
open http://localhost:3000
Your browser with open automatically Grafana.
To automatically scrape metrics from pods, add to the application with the following annotations:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "8080"
prometheus.io/scrape: "true"
Check which custom metrics are avaible:
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq .
Get active connections:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/active_connections" | jq