Running jiralert on k8s #82

JatinSomarajan · 2021-03-09T11:37:59Z

Hi,

I am trying to setup a k8s cluster which includes Prometheus+AlertManager+Grafana+Jiralert. The end goal is for AlertManager to create jira tickets as and when alerts are fired.

In my current setup, jiralert is running as a process and everything is working as expected. But I would like to containerise jiralert and run it as a pod. Is there an official docker image available for jiralert? Also, how do I go about setting it up? How will the image handle jiralert.yml? Do we have to mount the file onto the container?

Thank You,
Jatin

charlspjohn · 2021-03-09T14:57:30Z

Hi jatin, I had the same requirements 6 months ago. Then i build the jiralert container inhouse from the source code available at github. Then created a simple helmchart with three templates a deployment, service and a configmap. And yes the configmap is where jiralert.yaml resides. Try to make use of helm templating so that you can customize jiralert.yaml from values file..

…

On Tue, Mar 9, 2021, 17:08 JatinSomarajan ***@***.***> wrote: Hi, I am trying to setup a k8s cluster which includes Prometheus+AlertManager+Grafana+Jiralert. The end goal is for AlertManager to create jira tickets as and when alerts are fired. In my current setup, jiralert is running as a process and everything is working as expected. But I would like to containerise jiralert and run it as a pod. Is there an official docker image available for jiralert? Also, how do I go about setting it up? How will the image handle jiralert.yml? Do we have to mount the file onto the container? Thank You, Jatin — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#82>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADRJVMDO354ZDTX6M25TWODTCYCCPANCNFSM4Y3M4PBA> .

erdii · 2021-03-09T14:57:30Z

Hey and welcome :)

I don't think that we currently build 'official' docker images, but we definitely should!
You can already create it on your own by checking out and running make docker and then afterwards relabeling the created image so that you can push it to your docker registry (or public+free docker hub / quay.io account).

How will the image handle jiralert.yml?

you'll need to supply it as a file, yes. I'd propably either use a configmap or secret and mount it then.
there is a pitfall though: if you just change the configmap, jiralert will not restart and pick up the config file changes.

JatinSomarajan · 2021-03-09T15:25:51Z

Hi jatin, I had the same requirements 6 months ago. Then i build the jiralert container inhouse from the source code available at github. Then created a simple helmchart with three templates a deployment, service and a configmap. And yes the configmap is where jiralert.yaml resides. Try to make use of helm templating so that you can customize jiralert.yaml from values file..
…

Hi @charlspjohn,

Thank you for your suggestion.. When it comes to helm I am total noob and will need a lot of handholding. Have you documented your steps by chance or have you published your helm chart? It would be of great help. Thank a lot in advance!!

JatinSomarajan · 2021-03-09T15:33:52Z

Hey and welcome :)

I don't think that we currently build 'official' docker images, but we definitely should!
You can already create it on your own by checking out and running make docker and then afterwards relabeling the created image so that you can push it to your docker registry (or public+free docker hub / quay.io account).

How will the image handle jiralert.yml?

you'll need to supply it as a file, yes. I'd propably either use a configmap or secret and mount it then.
there is a pitfall though: if you just change the configmap, jiralert will not restart and pick up the config file changes.

Hi @erdii,

I do see a couple of docker images at docker hub, but they seem to be quite old. I might as well create one and upload to docker hub as you suggested.

In addition to configmap or Secret, I will also need a deployment right? If yes, what options should I start the deployment with?

Thank You,
Jatin

erdii · 2021-03-12T16:36:54Z

FYI there is a PR pending for docker images: #86

erdii · 2021-03-12T16:56:01Z

@JatinSomarajan This is a starting point for deploying on kubernetes: https://gist.github.com/erdii/8db7b5a6356c3d5e023b985b6febd57c

JatinSomarajan · 2021-03-15T14:37:15Z

@JatinSomarajan This is a starting point for deploying on kubernetes: https://gist.github.com/erdii/8db7b5a6356c3d5e023b985b6febd57c

This is great..Thank you @erdii .. Really appreciate it!!

Takshika · 2021-10-05T11:33:19Z

@erdii @JatinSomarajan I have a similar requirement to deploy jiralert in k8s cluster. I referenced the code provided in https://gist.github.com/erdii/8db7b5a6356c3d5e023b985b6febd57c. however, the container fails with a "Back-off restarting failed container" Error. Can you suggest to me what needs to be done in order to make it running?
`
Events:
Type Reason Age From Message

Normal Scheduled 52m default-scheduler Successfully assigned default/jiralert-848797799b-98gl6 to aks-nodepool1-18523433-vmss000000
Normal Pulled 52m kubelet Successfully pulled image "quay.io/app-sre/jiralert:latest" in 142.086737ms
Normal Pulled 52m kubelet Successfully pulled image "quay.io/app-sre/jiralert:latest" in 139.965279ms
Normal Pulled 52m kubelet Successfully pulled image "quay.io/app-sre/jiralert:latest" in 131.040599ms
Normal Created 51m (x4 over 52m) kubelet Created container jiralert
Normal Started 51m (x4 over 52m) kubelet Started container jiralert
Normal Pulling 51m (x4 over 52m) kubelet Pulling image "quay.io/app-sre/jiralert:latest"
Normal Pulled 51m kubelet Successfully pulled image "quay.io/app-sre/jiralert:latest" in 146.765274ms
Warning BackOff 2m27s (x240 over 52m) kubelet Back-off restarting failed container
`

beneso · 2022-11-02T10:09:26Z

We use this Dockerfile:

FROM golang:1.15 AS builder
WORKDIR /go/src/github.com/prometheus-community/jiralert
COPY . /go/src/github.com/prometheus-community/jiralert
RUN GO111MODULE=on GOBIN=/tmp/bin make

FROM alpine

COPY --from=builder /go/src/github.com/prometheus-community/jiralert/jiralert /bin/jiralert
RUN apk add --update ca-certificates

ENTRYPOINT [ "/bin/jiralert" ]

(we override the command in deployment):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: alertmanager-jiralert-webhook-deployment
  labels:
    app: alertmanager-jiralert-webhook-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: alertmanager-jiralert-webhook-deployment
  template:
    metadata:
      labels:
        app: alertmanager-jiralert-webhook-deployment
    spec:
      containers:
        - name: alertmanager-jiralert-webhook-deployment
          image: "our/image/alertmanager-jiralert:v1.2-alpine-0.1"
          command: 
          - /bin/sh
          - -c 
          - /usr/sbin/update-ca-certificates && /bin/jiralert -config /config/jiralert.yml -log.level debug
          ports:
            - containerPort: 9097
          resources:
            limits:
              cpu: 1
              memory: "256Mi"
            requests:
              cpu: "500m"
              memory: "128Mi"
          volumeMounts:
          - name: config
            mountPath: /config
          - name: certs
            mountPath: /usr/local/share/ca-certificates/
      volumes:
      - name: config
        configMap: 
          name: alertmanager-jiralert-webhook-configmap
      - name: certs
        secret:
          secretName: alertmanager-jiralert-webhook-tls

The configmap:

apiVersion: v1
data:
  jiralert.tmpl: |
    {{`{{ define "jira.summary" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}

    {{ define "jira.description" }}{{ range .Alerts.Firing }}Labels:
    {{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}
    {{ end }}
    Annotations:
    {{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}
    {{ end }}
    Source: {{ .GeneratorURL }}
    {{ end }}{{ end }}`}}
  jiralert.yml: |
    {{`
    defaults:
      api_url: https://jira.example.com
      user: user
      password: 'password'
      issue_type: Bug
      priority: Critical
      summary: '{{ template "jira.summary" . }}'
      description: '{{ template "jira.description" . }}'
      reopen_state: "To Do"
      wont_fix_resolution: "Won't Fix"
      reopen_duration: 0h

    receivers:
      - name: 'kubernetes_jiralert'
        project: PROJECT
        add_group_labels: false

    template: jiralert.tmpl`}}
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: alertmanager-jiralert-webhook-configmap

The service:

apiVersion: v1
kind: Service
metadata:
  name: alertmanager-jiralert-webhook-service
spec:
  selector:
    app: alertmanager-jiralert-webhook-deployment
  ports:
    - protocol: TCP
      port: 9097
      targetPort: 9097

Prometheus Alertmanager config (the kube-prometheus stack):

        routes:
          - matchers:
            - env_type = "dev"
            receiver: 'kubernetes_jiralert'

      receivers:
        - name: kubernetes_jiralert
          webhook_configs:
          - url: http://alertmanager-jiralert-webhook-service:9097/alert

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Running jiralert on k8s #82

Running jiralert on k8s #82

JatinSomarajan commented Mar 9, 2021

charlspjohn commented Mar 9, 2021 via email

erdii commented Mar 9, 2021

JatinSomarajan commented Mar 9, 2021

JatinSomarajan commented Mar 9, 2021

erdii commented Mar 12, 2021

erdii commented Mar 12, 2021

JatinSomarajan commented Mar 15, 2021

Takshika commented Oct 5, 2021 •

edited

Loading

beneso commented Nov 2, 2022

Running jiralert on k8s #82

Running jiralert on k8s #82

Comments

JatinSomarajan commented Mar 9, 2021

charlspjohn commented Mar 9, 2021 via email

erdii commented Mar 9, 2021

JatinSomarajan commented Mar 9, 2021

JatinSomarajan commented Mar 9, 2021

erdii commented Mar 12, 2021

erdii commented Mar 12, 2021

JatinSomarajan commented Mar 15, 2021

Takshika commented Oct 5, 2021 • edited Loading

beneso commented Nov 2, 2022

Takshika commented Oct 5, 2021 •

edited

Loading