Skip to content

Alerting Routing Model

Jump to bottom
Alexandre Lamarre edited this page Feb 17, 2023 · 6 revisions

Alerting Routing Model

Summary:

The alerting routing model is concrete AlertManager configuration files that opni uses to create routing relationships between Endpoints and Conditions.

It aims to manages the following customer usescases :

  • routing relationships for Opni conditions & Opni endpoints, without the user having an familiarity with AlertManager

Table of contents

Description

WIP: link to godoc once code is in main

Opni routing uses an interface to delegate implementation details of constructing AlertManager configurations.

The Opni routing interface itself uses a ProductionConfigSyncer interface to index and sync any configuration file to the router implementation.

type ProductionConfigSyncer interface {
	// Walks the tree of routes in the config, calling the given function
	Walk(map[string]string, func(depth int, r *config.Route) error) error
	// Returns the routes that match the given labels
	Search(labels map[string]string) []*config.Route
	// Merges two OpniRouting objects (also includes merging plain AlertManager configs for users)
	Merge(other OpniRouting) (OpniRouting, error)
	// Converts a valid AlertManager config to OpniRouting
	// Returns an FailedPrecondition error if the config cannot be unmarshalled,
	// Returns an InternalServerError if the config is invalid
	SyncExternalConfig(content []byte) error
}

type OpniRouting interface {
	ProductionConfigSyncer
	yaml.Unmarshaler
	MarshalYAML() ([]byte, error)

	SetDefaultNamespaceConfig(endps []*alertingv1.AlertEndpoint) error
	SetNamespaceSpec(namespace string, routeId string, specs *alertingv1.FullAttachedEndpoints) error
	// When an already attached endpoint is updated, propagate updates to the routing tree
	UpdateEndpoint(id string, spec *alertingv1.AlertEndpoint) error
	// When an already attached endpoint is delete, propagate all deletions to the routing tree
	DeleteEndpoint(endpointId string) error
	// Converts OpniRouting to a valid AlertManager config
	// Returns a NotFound error if the a route to update or delete is not found
	// Returns a Conflict error if we try to insert a duplicate config, unique up to its keys
	BuildConfig() (*config.Config, error)
	Clone() OpniRouting
}

Note : Namespace identifiers are opaque identifiers meant to help the implementation of the OpniRouting interface to determine how to group and optimize the AlertManager routing configuration file.

Opni RouterV1 Implementation

  • The namespaces are constructed into explicit subtrees
  • Incoming alerts that come from Opni-Alerting are matched into namespaces
  • Incoming alerts that come from Opni-Monitoring are matched to an embedded config
  • The current build also adds finalizer hooks (that point to the Opni Embedded server in each AlertManager pod) to each namespaced subtree
  • The default namespace corresponds to the OpniSeverity enum since it always defined on all Opni-Alerting sources
    • Decreasing severity increases rate limiting -- Info is the most heavily rate limited & Critical is the least heavily rate limited

Overview

EntryPoint
|
|__ Is this generated by Opni conditions?
|   |
|   |__ match to correct routing namespace
|       |
|       |__ match to correct conditionId ---> dispatch alert to endpoints attached to that conditionId 
|
|
|__ Matches no known namespaces --> Send to default endpoints   
|
|__ Is this generated by the Opni metrics backend?
    | 
    |__ <embed a routing configuration of our choosing here>

Example construction of namespaced tree

// --- start of Opni-Alerting datasource
routes:
// --- start of namespace "test"
    - matchers:
      - test!=""
      continue: false
      routes:
      - receiver: opni.test.144848a2-4b89-4a99-81a1-e09872ae5e63
// --- alarm of id 144848a2-4b89-4a99-81a1-e09872ae5e63
        matchers:
        - test="144848a2-4b89-4a99-81a1-e09872ae5e63"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
// --- explicit finalizer hook
      - receiver: opni.default.hook
        continue: false

Default namespace construction

  • Automatically constructed from grpc enum in the form:
    routes:
// --- begin default namespace
    - matchers:
      - OpniSeverity!=""
      continue: false
// --- For each ENUM_VALUE expand
      routes:
      - receiver: opni.OpniSeverity.<ENUM_VALUE>
        matchers:
        - OpniSeverity="<ENUM_VALUE>"
        continue: true
        group_wait: 10s
        group_interval: 10m
        repeat_interval: 10m
// --- explicit finalizer hook
    - receiver: opni.default.hook
      continue: false
  group_wait: 1m
  repeat_interval: 5h
receivers:
// ...
// --- For each ENUM_VALUE expand
- name: opni.OpniSeverity.<ENUM_VALUE>
templates: []

Example Auto-Generated config:

Paste into AlertManager visualization tool.

Generated files in production configs are easier to read -- this is only used to gain some intuition.

Auto-generated config file 

global:
  resolve_timeout: 5m
  http_config:
    follow_redirects: true
    enable_http2: true
  smtp_hello: localhost
  smtp_require_tls: true
  pagerduty_url: https://events.pagerduty.com/v2/enqueue
  opsgenie_api_url: https://api.opsgenie.com/
  wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
  victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
  telegram_api_url: https://api.telegram.org
route:
  receiver: opni.default.hook
  continue: false
  routes:
  - matchers:
    - OpniDatasource=""
    continue: true
    routes:
    - matchers:
      - aiops!=""
      continue: false
      routes:
      - receiver: opni.aiops.iv3cgdKsTzxXaDtAHmkxq6
        matchers:
        - aiops="iv3cgdKsTzxXaDtAHmkxq6"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.aiops.PY6QbhhN4TfxwybNAiuCFm
        matchers:
        - aiops="PY6QbhhN4TfxwybNAiuCFm"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.default.hook
        continue: false
      group_wait: 1m
      repeat_interval: 5h
    - matchers:
      - backend!=""
      continue: false
      routes:
      - receiver: opni.backend.g5mgnXt6gUrb6XwcKNJAuJ
        matchers:
        - backend="g5mgnXt6gUrb6XwcKNJAuJ"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.backend.dMnFLP2ayEbQvCBgcpz83X
        matchers:
        - backend="dMnFLP2ayEbQvCBgcpz83X"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.backend.ZtvEZ6cUFg72Yhc8aLviqD
        matchers:
        - backend="ZtvEZ6cUFg72Yhc8aLviqD"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.backend.YvmUV7PdSAUXKjXX7Yjyud
        matchers:
        - backend="YvmUV7PdSAUXKjXX7Yjyud"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.backend.LKJobzptsRVPdM25cu9tam
        matchers:
        - backend="LKJobzptsRVPdM25cu9tam"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.backend.Cndvkz4ckUeuBik4ehkjyM
        matchers:
        - backend="Cndvkz4ckUeuBik4ehkjyM"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.backend.Bzw6ig6W3hGV9zN9S3wcMi
        matchers:
        - backend="Bzw6ig6W3hGV9zN9S3wcMi"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.backend.8jMFBqazKeUfRTrbzEBZ59
        matchers:
        - backend="8jMFBqazKeUfRTrbzEBZ59"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.backend.2FhnYsiya8z5TocmKRheNP
        matchers:
        - backend="2FhnYsiya8z5TocmKRheNP"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.default.hook
        continue: false
      group_wait: 1m
      repeat_interval: 5h
    - matchers:
      - capability!=""
      continue: false
      routes:
      - receiver: opni.capability.yJcchjmWdiigQiei64wEWJ
        matchers:
        - capability="yJcchjmWdiigQiei64wEWJ"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.capability.qb2sQQGH8qnFdWJyvhKg8F
        matchers:
        - capability="qb2sQQGH8qnFdWJyvhKg8F"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.capability.kRriGcitDKUTW87GSEGcSo
        matchers:
        - capability="kRriGcitDKUTW87GSEGcSo"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.capability.JhHiaepenGTWqwmwV7CLac
        matchers:
        - capability="JhHiaepenGTWqwmwV7CLac"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.default.hook
        continue: false
      group_wait: 1m
      repeat_interval: 5h
    - matchers:
      - cpu!=""
      continue: false
      routes:
      - receiver: opni.cpu.NvEpKD6J6NiB8GyJRwCrtK
        matchers:
        - cpu="NvEpKD6J6NiB8GyJRwCrtK"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.cpu.BfadEokVjMAfEyxSB6Ys7T
        matchers:
        - cpu="BfadEokVjMAfEyxSB6Ys7T"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.cpu.5b5RAopxAVyuMMz4jSpq84
        matchers:
        - cpu="5b5RAopxAVyuMMz4jSpq84"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.default.hook
        continue: false
      group_wait: 1m
      repeat_interval: 5h
    - matchers:
      - disconnect!=""
      continue: false
      routes:
      - receiver: opni.disconnect.ChbcwZs8UFptwcpyGDxyzH
        matchers:
        - disconnect="ChbcwZs8UFptwcpyGDxyzH"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.default.hook
        continue: false
      group_wait: 1m
      repeat_interval: 5h
    - matchers:
      - memory!=""
      continue: false
      routes:
      - receiver: opni.memory.k2pUZAGL4MAeEe98UjKDf6
        matchers:
        - memory="k2pUZAGL4MAeEe98UjKDf6"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.memory.DJUgDdGyHPLPxbP6YvfoHF
        matchers:
        - memory="DJUgDdGyHPLPxbP6YvfoHF"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.memory.BbRnAdfGESVx3zFzKX9NfD
        matchers:
        - memory="BbRnAdfGESVx3zFzKX9NfD"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.memory.5qHohrmEg9SkCQcrnx9Q
        matchers:
        - memory="5qHohrmEg9SkCQcrnx9Q"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.default.hook
        continue: false
      group_wait: 1m
      repeat_interval: 5h
    - matchers:
      - prometheusQuery!=""
      continue: false
      routes:
      - receiver: opni.prometheusQuery.iLBBaQ5LaGAhLi7vSj9C8F
        matchers:
        - prometheusQuery="iLBBaQ5LaGAhLi7vSj9C8F"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.prometheusQuery.coKWKPD5mpjyZtsVFgcMsc
        matchers:
        - prometheusQuery="coKWKPD5mpjyZtsVFgcMsc"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.prometheusQuery.WbCdYxccA5fev4bUkXLV7h
        matchers:
        - prometheusQuery="WbCdYxccA5fev4bUkXLV7h"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.prometheusQuery.MG6LkNmDEobUq85a9ZJoFm
        matchers:
        - prometheusQuery="MG6LkNmDEobUq85a9ZJoFm"
        continue: true
        group_wait: 1m
        group_interval: 30s
        repeat_interval: 10m
      - receiver: opni.default.hook
        continue: false
      group_wait: 1m
      repeat_interval: 5h
    - matchers:
      - OpniSeverity!=""
      continue: false
      routes:
      - receiver: opni.OpniSeverity.Critical
        matchers:
        - OpniSeverity="Critical"
        continue: true
        group_wait: 10s
        group_interval: 10m
        repeat_interval: 10m
      - receiver: opni.OpniSeverity.Error
        matchers:
        - OpniSeverity="Error"
        continue: true
        group_wait: 10s
        group_interval: 10m
        repeat_interval: 10m
      - receiver: opni.OpniSeverity.Info
        matchers:
        - OpniSeverity="Info"
        continue: true
        group_wait: 10s
        group_interval: 10m
        repeat_interval: 10m
      - receiver: opni.OpniSeverity.Warning
        matchers:
        - OpniSeverity="Warning"
        continue: true
        group_wait: 10s
        group_interval: 10m
        repeat_interval: 10m
      - receiver: opni.default.hook
        continue: false
      group_wait: 1m
      repeat_interval: 5h
    - receiver: opni.default.hook
      continue: false
  - matchers:
    - OpniDatasource="metrics"
    continue: true
    group_wait: 10s
    group_interval: 10m
    repeat_interval: 10m
  group_wait: 1m
  repeat_interval: 5h
receivers:
- name: opni.aiops.PY6QbhhN4TfxwybNAiuCFm
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '[UPDATED] testcase-PY6QbhhN4TfxwybNAiuCFm to Endpoints [endp-7LfZWCsAgrYCT4bPZATihA
      endp-iv7S4DTZvjUg8jrxfcTaSB]'
    text: '[UPDATED] testcase-PY6QbhhN4TfxwybNAiuCFm body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
- name: opni.aiops.iv3cgdKsTzxXaDtAHmkxq6
  email_configs:
  - send_resolved: false
    to: endp-LJZdtF9PLPkinChrmgTm8F-to@gmail.com
    from: endp-LJZdtF9PLPkinChrmgTm8F-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-iv3cgdKsTzxXaDtAHmkxq6 body'
    html: '[UPDATED] testcase-iv3cgdKsTzxXaDtAHmkxq6 body'
  - send_resolved: false
    to: endp-jE8GmTFEUgAbUYoFkEzN8Z-to@gmail.com
    from: endp-jE8GmTFEUgAbUYoFkEzN8Z-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-iv3cgdKsTzxXaDtAHmkxq6 body'
    html: '[UPDATED] testcase-iv3cgdKsTzxXaDtAHmkxq6 body'
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '[UPDATED] testcase-iv3cgdKsTzxXaDtAHmkxq6 body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-iv3cgdKsTzxXaDtAHmkxq6 to Endpoints [endp-LJZdtF9PLPkinChrmgTm8F
        endp-oezwYn49Q7U3MBZF8LqgUi endp-GXvVZRRWU74XLYiTDjYHgN endp-jE8GmTFEUgAbUYoFkEzN8Z
        endp-7LfZWCsAgrYCT4bPZATihA endp-6khsAZ8VVWoXzCx5niJiJo]'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '[UPDATED] testcase-iv3cgdKsTzxXaDtAHmkxq6 to Endpoints [endp-LJZdtF9PLPkinChrmgTm8F
      endp-oezwYn49Q7U3MBZF8LqgUi endp-GXvVZRRWU74XLYiTDjYHgN endp-jE8GmTFEUgAbUYoFkEzN8Z
      endp-7LfZWCsAgrYCT4bPZATihA endp-6khsAZ8VVWoXzCx5niJiJo]'
    text: '[UPDATED] testcase-iv3cgdKsTzxXaDtAHmkxq6 body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
  - send_resolved: false
    url: https://webhook.com/endp-oezwYn49Q7U3MBZF8LqgUi
    max_alerts: 0
- name: opni.backend.2FhnYsiya8z5TocmKRheNP
  email_configs:
  - send_resolved: false
    to: endp-LJZdtF9PLPkinChrmgTm8F-to@gmail.com
    from: endp-LJZdtF9PLPkinChrmgTm8F-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-2FhnYsiya8z5TocmKRheNP body'
    html: '[UPDATED] testcase-2FhnYsiya8z5TocmKRheNP body'
  - send_resolved: false
    to: endp-jE8GmTFEUgAbUYoFkEzN8Z-to@gmail.com
    from: endp-jE8GmTFEUgAbUYoFkEzN8Z-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-2FhnYsiya8z5TocmKRheNP body'
    html: '[UPDATED] testcase-2FhnYsiya8z5TocmKRheNP body'
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '[UPDATED] testcase-2FhnYsiya8z5TocmKRheNP body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-2FhnYsiya8z5TocmKRheNP to Endpoints [endp-LJZdtF9PLPkinChrmgTm8F
        endp-7LfZWCsAgrYCT4bPZATihA endp-GXvVZRRWU74XLYiTDjYHgN endp-6khsAZ8VVWoXzCx5niJiJo
        endp-oezwYn49Q7U3MBZF8LqgUi endp-jE8GmTFEUgAbUYoFkEzN8Z endp-H5rtQnGvav69kCJfgPWTuJ
        endp-iv7S4DTZvjUg8jrxfcTaSB]'
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '[UPDATED] testcase-2FhnYsiya8z5TocmKRheNP body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-2FhnYsiya8z5TocmKRheNP to Endpoints [endp-LJZdtF9PLPkinChrmgTm8F
        endp-7LfZWCsAgrYCT4bPZATihA endp-GXvVZRRWU74XLYiTDjYHgN endp-6khsAZ8VVWoXzCx5niJiJo
        endp-oezwYn49Q7U3MBZF8LqgUi endp-jE8GmTFEUgAbUYoFkEzN8Z endp-H5rtQnGvav69kCJfgPWTuJ
        endp-iv7S4DTZvjUg8jrxfcTaSB]'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '[UPDATED] testcase-2FhnYsiya8z5TocmKRheNP to Endpoints [endp-LJZdtF9PLPkinChrmgTm8F
      endp-7LfZWCsAgrYCT4bPZATihA endp-GXvVZRRWU74XLYiTDjYHgN endp-6khsAZ8VVWoXzCx5niJiJo
      endp-oezwYn49Q7U3MBZF8LqgUi endp-jE8GmTFEUgAbUYoFkEzN8Z endp-H5rtQnGvav69kCJfgPWTuJ
      endp-iv7S4DTZvjUg8jrxfcTaSB]'
    text: '[UPDATED] testcase-2FhnYsiya8z5TocmKRheNP body'
    short_fields: false
    link_names: false
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '[UPDATED] testcase-2FhnYsiya8z5TocmKRheNP to Endpoints [endp-LJZdtF9PLPkinChrmgTm8F
      endp-7LfZWCsAgrYCT4bPZATihA endp-GXvVZRRWU74XLYiTDjYHgN endp-6khsAZ8VVWoXzCx5niJiJo
      endp-oezwYn49Q7U3MBZF8LqgUi endp-jE8GmTFEUgAbUYoFkEzN8Z endp-H5rtQnGvav69kCJfgPWTuJ
      endp-iv7S4DTZvjUg8jrxfcTaSB]'
    text: '[UPDATED] testcase-2FhnYsiya8z5TocmKRheNP body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
  - send_resolved: false
    url: https://webhook.com/endp-oezwYn49Q7U3MBZF8LqgUi
    max_alerts: 0
- name: opni.backend.8jMFBqazKeUfRTrbzEBZ59
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
- name: opni.backend.Bzw6ig6W3hGV9zN9S3wcMi
  email_configs:
  - send_resolved: false
    to: endp-LJZdtF9PLPkinChrmgTm8F-to@gmail.com
    from: endp-LJZdtF9PLPkinChrmgTm8F-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-Bzw6ig6W3hGV9zN9S3wcMi body'
    html: '[UPDATED] testcase-Bzw6ig6W3hGV9zN9S3wcMi body'
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '[UPDATED] testcase-Bzw6ig6W3hGV9zN9S3wcMi body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-Bzw6ig6W3hGV9zN9S3wcMi to Endpoints [endp-H5rtQnGvav69kCJfgPWTuJ
        endp-LJZdtF9PLPkinChrmgTm8F]'
- name: opni.backend.Cndvkz4ckUeuBik4ehkjyM
  email_configs:
  - send_resolved: false
    to: endp-LJZdtF9PLPkinChrmgTm8F-to@gmail.com
    from: endp-LJZdtF9PLPkinChrmgTm8F-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-Cndvkz4ckUeuBik4ehkjyM body'
    html: '[UPDATED] testcase-Cndvkz4ckUeuBik4ehkjyM body'
  - send_resolved: false
    to: endp-jE8GmTFEUgAbUYoFkEzN8Z-to@gmail.com
    from: endp-jE8GmTFEUgAbUYoFkEzN8Z-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-Cndvkz4ckUeuBik4ehkjyM body'
    html: '[UPDATED] testcase-Cndvkz4ckUeuBik4ehkjyM body'
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '[UPDATED] testcase-Cndvkz4ckUeuBik4ehkjyM body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-Cndvkz4ckUeuBik4ehkjyM to Endpoints [endp-LJZdtF9PLPkinChrmgTm8F
        endp-GXvVZRRWU74XLYiTDjYHgN endp-7LfZWCsAgrYCT4bPZATihA endp-oezwYn49Q7U3MBZF8LqgUi
        endp-6khsAZ8VVWoXzCx5niJiJo endp-iv7S4DTZvjUg8jrxfcTaSB endp-jE8GmTFEUgAbUYoFkEzN8Z
        endp-H5rtQnGvav69kCJfgPWTuJ]'
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '[UPDATED] testcase-Cndvkz4ckUeuBik4ehkjyM body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-Cndvkz4ckUeuBik4ehkjyM to Endpoints [endp-LJZdtF9PLPkinChrmgTm8F
        endp-GXvVZRRWU74XLYiTDjYHgN endp-7LfZWCsAgrYCT4bPZATihA endp-oezwYn49Q7U3MBZF8LqgUi
        endp-6khsAZ8VVWoXzCx5niJiJo endp-iv7S4DTZvjUg8jrxfcTaSB endp-jE8GmTFEUgAbUYoFkEzN8Z
        endp-H5rtQnGvav69kCJfgPWTuJ]'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '[UPDATED] testcase-Cndvkz4ckUeuBik4ehkjyM to Endpoints [endp-LJZdtF9PLPkinChrmgTm8F
      endp-GXvVZRRWU74XLYiTDjYHgN endp-7LfZWCsAgrYCT4bPZATihA endp-oezwYn49Q7U3MBZF8LqgUi
      endp-6khsAZ8VVWoXzCx5niJiJo endp-iv7S4DTZvjUg8jrxfcTaSB endp-jE8GmTFEUgAbUYoFkEzN8Z
      endp-H5rtQnGvav69kCJfgPWTuJ]'
    text: '[UPDATED] testcase-Cndvkz4ckUeuBik4ehkjyM body'
    short_fields: false
    link_names: false
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '[UPDATED] testcase-Cndvkz4ckUeuBik4ehkjyM to Endpoints [endp-LJZdtF9PLPkinChrmgTm8F
      endp-GXvVZRRWU74XLYiTDjYHgN endp-7LfZWCsAgrYCT4bPZATihA endp-oezwYn49Q7U3MBZF8LqgUi
      endp-6khsAZ8VVWoXzCx5niJiJo endp-iv7S4DTZvjUg8jrxfcTaSB endp-jE8GmTFEUgAbUYoFkEzN8Z
      endp-H5rtQnGvav69kCJfgPWTuJ]'
    text: '[UPDATED] testcase-Cndvkz4ckUeuBik4ehkjyM body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
  - send_resolved: false
    url: https://webhook.com/endp-oezwYn49Q7U3MBZF8LqgUi
    max_alerts: 0
- name: opni.backend.LKJobzptsRVPdM25cu9tam
  email_configs:
  - send_resolved: false
    to: endp-LJZdtF9PLPkinChrmgTm8F-to@gmail.com
    from: endp-LJZdtF9PLPkinChrmgTm8F-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-LKJobzptsRVPdM25cu9tam body'
    html: '[UPDATED] testcase-LKJobzptsRVPdM25cu9tam body'
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '[UPDATED] testcase-LKJobzptsRVPdM25cu9tam body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-LKJobzptsRVPdM25cu9tam to Endpoints [endp-iv7S4DTZvjUg8jrxfcTaSB
        endp-6khsAZ8VVWoXzCx5niJiJo endp-7LfZWCsAgrYCT4bPZATihA endp-LJZdtF9PLPkinChrmgTm8F
        endp-oezwYn49Q7U3MBZF8LqgUi]'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '[UPDATED] testcase-LKJobzptsRVPdM25cu9tam to Endpoints [endp-iv7S4DTZvjUg8jrxfcTaSB
      endp-6khsAZ8VVWoXzCx5niJiJo endp-7LfZWCsAgrYCT4bPZATihA endp-LJZdtF9PLPkinChrmgTm8F
      endp-oezwYn49Q7U3MBZF8LqgUi]'
    text: '[UPDATED] testcase-LKJobzptsRVPdM25cu9tam body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
  - send_resolved: false
    url: https://webhook.com/endp-oezwYn49Q7U3MBZF8LqgUi
    max_alerts: 0
- name: opni.backend.YvmUV7PdSAUXKjXX7Yjyud
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '[UPDATED] testcase-YvmUV7PdSAUXKjXX7Yjyud body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-YvmUV7PdSAUXKjXX7Yjyud to Endpoints [endp-iv7S4DTZvjUg8jrxfcTaSB
        endp-oezwYn49Q7U3MBZF8LqgUi endp-6khsAZ8VVWoXzCx5niJiJo endp-GXvVZRRWU74XLYiTDjYHgN
        endp-H5rtQnGvav69kCJfgPWTuJ]'
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '[UPDATED] testcase-YvmUV7PdSAUXKjXX7Yjyud body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-YvmUV7PdSAUXKjXX7Yjyud to Endpoints [endp-iv7S4DTZvjUg8jrxfcTaSB
        endp-oezwYn49Q7U3MBZF8LqgUi endp-6khsAZ8VVWoXzCx5niJiJo endp-GXvVZRRWU74XLYiTDjYHgN
        endp-H5rtQnGvav69kCJfgPWTuJ]'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '[UPDATED] testcase-YvmUV7PdSAUXKjXX7Yjyud to Endpoints [endp-iv7S4DTZvjUg8jrxfcTaSB
      endp-oezwYn49Q7U3MBZF8LqgUi endp-6khsAZ8VVWoXzCx5niJiJo endp-GXvVZRRWU74XLYiTDjYHgN
      endp-H5rtQnGvav69kCJfgPWTuJ]'
    text: '[UPDATED] testcase-YvmUV7PdSAUXKjXX7Yjyud body'
    short_fields: false
    link_names: false
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '[UPDATED] testcase-YvmUV7PdSAUXKjXX7Yjyud to Endpoints [endp-iv7S4DTZvjUg8jrxfcTaSB
      endp-oezwYn49Q7U3MBZF8LqgUi endp-6khsAZ8VVWoXzCx5niJiJo endp-GXvVZRRWU74XLYiTDjYHgN
      endp-H5rtQnGvav69kCJfgPWTuJ]'
    text: '[UPDATED] testcase-YvmUV7PdSAUXKjXX7Yjyud body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-oezwYn49Q7U3MBZF8LqgUi
    max_alerts: 0
- name: opni.backend.ZtvEZ6cUFg72Yhc8aLviqD
  email_configs:
  - send_resolved: false
    to: endp-jE8GmTFEUgAbUYoFkEzN8Z-to@gmail.com
    from: endp-jE8GmTFEUgAbUYoFkEzN8Z-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-ZtvEZ6cUFg72Yhc8aLviqD body'
    html: '[UPDATED] testcase-ZtvEZ6cUFg72Yhc8aLviqD body'
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '[UPDATED] testcase-ZtvEZ6cUFg72Yhc8aLviqD body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-ZtvEZ6cUFg72Yhc8aLviqD to Endpoints [endp-jE8GmTFEUgAbUYoFkEzN8Z
        endp-H5rtQnGvav69kCJfgPWTuJ]'
- name: opni.backend.dMnFLP2ayEbQvCBgcpz83X
  email_configs:
  - send_resolved: false
    to: endp-LJZdtF9PLPkinChrmgTm8F-to@gmail.com
    from: endp-LJZdtF9PLPkinChrmgTm8F-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-dMnFLP2ayEbQvCBgcpz83X body'
    html: '[UPDATED] testcase-dMnFLP2ayEbQvCBgcpz83X body'
  - send_resolved: false
    to: endp-jE8GmTFEUgAbUYoFkEzN8Z-to@gmail.com
    from: endp-jE8GmTFEUgAbUYoFkEzN8Z-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-dMnFLP2ayEbQvCBgcpz83X body'
    html: '[UPDATED] testcase-dMnFLP2ayEbQvCBgcpz83X body'
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '[UPDATED] testcase-dMnFLP2ayEbQvCBgcpz83X body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-dMnFLP2ayEbQvCBgcpz83X to Endpoints [endp-6khsAZ8VVWoXzCx5niJiJo
        endp-LJZdtF9PLPkinChrmgTm8F endp-oezwYn49Q7U3MBZF8LqgUi endp-jE8GmTFEUgAbUYoFkEzN8Z
        endp-7LfZWCsAgrYCT4bPZATihA endp-GXvVZRRWU74XLYiTDjYHgN]'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '[UPDATED] testcase-dMnFLP2ayEbQvCBgcpz83X to Endpoints [endp-6khsAZ8VVWoXzCx5niJiJo
      endp-LJZdtF9PLPkinChrmgTm8F endp-oezwYn49Q7U3MBZF8LqgUi endp-jE8GmTFEUgAbUYoFkEzN8Z
      endp-7LfZWCsAgrYCT4bPZATihA endp-GXvVZRRWU74XLYiTDjYHgN]'
    text: '[UPDATED] testcase-dMnFLP2ayEbQvCBgcpz83X body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
  - send_resolved: false
    url: https://webhook.com/endp-oezwYn49Q7U3MBZF8LqgUi
    max_alerts: 0
- name: opni.backend.g5mgnXt6gUrb6XwcKNJAuJ
  email_configs:
  - send_resolved: false
    to: endp-LJZdtF9PLPkinChrmgTm8F-to@gmail.com
    from: endp-LJZdtF9PLPkinChrmgTm8F-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-g5mgnXt6gUrb6XwcKNJAuJ body'
    html: '[UPDATED] testcase-g5mgnXt6gUrb6XwcKNJAuJ body'
  - send_resolved: false
    to: endp-jE8GmTFEUgAbUYoFkEzN8Z-to@gmail.com
    from: endp-jE8GmTFEUgAbUYoFkEzN8Z-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-g5mgnXt6gUrb6XwcKNJAuJ body'
    html: '[UPDATED] testcase-g5mgnXt6gUrb6XwcKNJAuJ body'
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '[UPDATED] testcase-g5mgnXt6gUrb6XwcKNJAuJ body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-g5mgnXt6gUrb6XwcKNJAuJ to Endpoints [endp-oezwYn49Q7U3MBZF8LqgUi
        endp-H5rtQnGvav69kCJfgPWTuJ endp-jE8GmTFEUgAbUYoFkEzN8Z endp-GXvVZRRWU74XLYiTDjYHgN
        endp-7LfZWCsAgrYCT4bPZATihA endp-6khsAZ8VVWoXzCx5niJiJo endp-iv7S4DTZvjUg8jrxfcTaSB
        endp-LJZdtF9PLPkinChrmgTm8F]'
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '[UPDATED] testcase-g5mgnXt6gUrb6XwcKNJAuJ body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-g5mgnXt6gUrb6XwcKNJAuJ to Endpoints [endp-oezwYn49Q7U3MBZF8LqgUi
        endp-H5rtQnGvav69kCJfgPWTuJ endp-jE8GmTFEUgAbUYoFkEzN8Z endp-GXvVZRRWU74XLYiTDjYHgN
        endp-7LfZWCsAgrYCT4bPZATihA endp-6khsAZ8VVWoXzCx5niJiJo endp-iv7S4DTZvjUg8jrxfcTaSB
        endp-LJZdtF9PLPkinChrmgTm8F]'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '[UPDATED] testcase-g5mgnXt6gUrb6XwcKNJAuJ to Endpoints [endp-oezwYn49Q7U3MBZF8LqgUi
      endp-H5rtQnGvav69kCJfgPWTuJ endp-jE8GmTFEUgAbUYoFkEzN8Z endp-GXvVZRRWU74XLYiTDjYHgN
      endp-7LfZWCsAgrYCT4bPZATihA endp-6khsAZ8VVWoXzCx5niJiJo endp-iv7S4DTZvjUg8jrxfcTaSB
      endp-LJZdtF9PLPkinChrmgTm8F]'
    text: '[UPDATED] testcase-g5mgnXt6gUrb6XwcKNJAuJ body'
    short_fields: false
    link_names: false
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '[UPDATED] testcase-g5mgnXt6gUrb6XwcKNJAuJ to Endpoints [endp-oezwYn49Q7U3MBZF8LqgUi
      endp-H5rtQnGvav69kCJfgPWTuJ endp-jE8GmTFEUgAbUYoFkEzN8Z endp-GXvVZRRWU74XLYiTDjYHgN
      endp-7LfZWCsAgrYCT4bPZATihA endp-6khsAZ8VVWoXzCx5niJiJo endp-iv7S4DTZvjUg8jrxfcTaSB
      endp-LJZdtF9PLPkinChrmgTm8F]'
    text: '[UPDATED] testcase-g5mgnXt6gUrb6XwcKNJAuJ body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
  - send_resolved: false
    url: https://webhook.com/endp-oezwYn49Q7U3MBZF8LqgUi
    max_alerts: 0
- name: opni.capability.JhHiaepenGTWqwmwV7CLac
  email_configs:
  - send_resolved: false
    to: endp-LJZdtF9PLPkinChrmgTm8F-to@gmail.com
    from: endp-LJZdtF9PLPkinChrmgTm8F-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-JhHiaepenGTWqwmwV7CLac body'
    html: '[UPDATED] testcase-JhHiaepenGTWqwmwV7CLac body'
  - send_resolved: false
    to: endp-jE8GmTFEUgAbUYoFkEzN8Z-to@gmail.com
    from: endp-jE8GmTFEUgAbUYoFkEzN8Z-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-JhHiaepenGTWqwmwV7CLac body'
    html: '[UPDATED] testcase-JhHiaepenGTWqwmwV7CLac body'
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '[UPDATED] testcase-JhHiaepenGTWqwmwV7CLac body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-JhHiaepenGTWqwmwV7CLac to Endpoints [endp-6khsAZ8VVWoXzCx5niJiJo
        endp-H5rtQnGvav69kCJfgPWTuJ endp-GXvVZRRWU74XLYiTDjYHgN endp-LJZdtF9PLPkinChrmgTm8F
        endp-oezwYn49Q7U3MBZF8LqgUi endp-iv7S4DTZvjUg8jrxfcTaSB endp-jE8GmTFEUgAbUYoFkEzN8Z
        endp-7LfZWCsAgrYCT4bPZATihA]'
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '[UPDATED] testcase-JhHiaepenGTWqwmwV7CLac body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-JhHiaepenGTWqwmwV7CLac to Endpoints [endp-6khsAZ8VVWoXzCx5niJiJo
        endp-H5rtQnGvav69kCJfgPWTuJ endp-GXvVZRRWU74XLYiTDjYHgN endp-LJZdtF9PLPkinChrmgTm8F
        endp-oezwYn49Q7U3MBZF8LqgUi endp-iv7S4DTZvjUg8jrxfcTaSB endp-jE8GmTFEUgAbUYoFkEzN8Z
        endp-7LfZWCsAgrYCT4bPZATihA]'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '[UPDATED] testcase-JhHiaepenGTWqwmwV7CLac to Endpoints [endp-6khsAZ8VVWoXzCx5niJiJo
      endp-H5rtQnGvav69kCJfgPWTuJ endp-GXvVZRRWU74XLYiTDjYHgN endp-LJZdtF9PLPkinChrmgTm8F
      endp-oezwYn49Q7U3MBZF8LqgUi endp-iv7S4DTZvjUg8jrxfcTaSB endp-jE8GmTFEUgAbUYoFkEzN8Z
      endp-7LfZWCsAgrYCT4bPZATihA]'
    text: '[UPDATED] testcase-JhHiaepenGTWqwmwV7CLac body'
    short_fields: false
    link_names: false
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '[UPDATED] testcase-JhHiaepenGTWqwmwV7CLac to Endpoints [endp-6khsAZ8VVWoXzCx5niJiJo
      endp-H5rtQnGvav69kCJfgPWTuJ endp-GXvVZRRWU74XLYiTDjYHgN endp-LJZdtF9PLPkinChrmgTm8F
      endp-oezwYn49Q7U3MBZF8LqgUi endp-iv7S4DTZvjUg8jrxfcTaSB endp-jE8GmTFEUgAbUYoFkEzN8Z
      endp-7LfZWCsAgrYCT4bPZATihA]'
    text: '[UPDATED] testcase-JhHiaepenGTWqwmwV7CLac body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
  - send_resolved: false
    url: https://webhook.com/endp-oezwYn49Q7U3MBZF8LqgUi
    max_alerts: 0
- name: opni.capability.kRriGcitDKUTW87GSEGcSo
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '[UPDATED] testcase-kRriGcitDKUTW87GSEGcSo body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-kRriGcitDKUTW87GSEGcSo to Endpoints [endp-7LfZWCsAgrYCT4bPZATihA
        endp-iv7S4DTZvjUg8jrxfcTaSB endp-H5rtQnGvav69kCJfgPWTuJ]'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '[UPDATED] testcase-kRriGcitDKUTW87GSEGcSo to Endpoints [endp-7LfZWCsAgrYCT4bPZATihA
      endp-iv7S4DTZvjUg8jrxfcTaSB endp-H5rtQnGvav69kCJfgPWTuJ]'
    text: '[UPDATED] testcase-kRriGcitDKUTW87GSEGcSo body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
- name: opni.capability.qb2sQQGH8qnFdWJyvhKg8F
  email_configs:
  - send_resolved: false
    to: endp-jE8GmTFEUgAbUYoFkEzN8Z-to@gmail.com
    from: endp-jE8GmTFEUgAbUYoFkEzN8Z-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-qb2sQQGH8qnFdWJyvhKg8F body'
    html: '[UPDATED] testcase-qb2sQQGH8qnFdWJyvhKg8F body'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '[UPDATED] testcase-qb2sQQGH8qnFdWJyvhKg8F to Endpoints [endp-jE8GmTFEUgAbUYoFkEzN8Z
      endp-GXvVZRRWU74XLYiTDjYHgN endp-7LfZWCsAgrYCT4bPZATihA]'
    text: '[UPDATED] testcase-qb2sQQGH8qnFdWJyvhKg8F body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
- name: opni.capability.yJcchjmWdiigQiei64wEWJ
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '[UPDATED] testcase-yJcchjmWdiigQiei64wEWJ body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-yJcchjmWdiigQiei64wEWJ to Endpoints [endp-7LfZWCsAgrYCT4bPZATihA
        endp-6khsAZ8VVWoXzCx5niJiJo]'
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
- name: opni.cpu.5b5RAopxAVyuMMz4jSpq84
  email_configs:
  - send_resolved: false
    to: endp-LJZdtF9PLPkinChrmgTm8F-to@gmail.com
    from: endp-LJZdtF9PLPkinChrmgTm8F-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-5b5RAopxAVyuMMz4jSpq84 body'
    html: '[UPDATED] testcase-5b5RAopxAVyuMMz4jSpq84 body'
  - send_resolved: false
    to: endp-jE8GmTFEUgAbUYoFkEzN8Z-to@gmail.com
    from: endp-jE8GmTFEUgAbUYoFkEzN8Z-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-5b5RAopxAVyuMMz4jSpq84 body'
    html: '[UPDATED] testcase-5b5RAopxAVyuMMz4jSpq84 body'
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '[UPDATED] testcase-5b5RAopxAVyuMMz4jSpq84 body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-5b5RAopxAVyuMMz4jSpq84 to Endpoints [endp-H5rtQnGvav69kCJfgPWTuJ
        endp-oezwYn49Q7U3MBZF8LqgUi endp-LJZdtF9PLPkinChrmgTm8F endp-7LfZWCsAgrYCT4bPZATihA
        endp-GXvVZRRWU74XLYiTDjYHgN endp-jE8GmTFEUgAbUYoFkEzN8Z endp-iv7S4DTZvjUg8jrxfcTaSB
        endp-6khsAZ8VVWoXzCx5niJiJo]'
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '[UPDATED] testcase-5b5RAopxAVyuMMz4jSpq84 body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-5b5RAopxAVyuMMz4jSpq84 to Endpoints [endp-H5rtQnGvav69kCJfgPWTuJ
        endp-oezwYn49Q7U3MBZF8LqgUi endp-LJZdtF9PLPkinChrmgTm8F endp-7LfZWCsAgrYCT4bPZATihA
        endp-GXvVZRRWU74XLYiTDjYHgN endp-jE8GmTFEUgAbUYoFkEzN8Z endp-iv7S4DTZvjUg8jrxfcTaSB
        endp-6khsAZ8VVWoXzCx5niJiJo]'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '[UPDATED] testcase-5b5RAopxAVyuMMz4jSpq84 to Endpoints [endp-H5rtQnGvav69kCJfgPWTuJ
      endp-oezwYn49Q7U3MBZF8LqgUi endp-LJZdtF9PLPkinChrmgTm8F endp-7LfZWCsAgrYCT4bPZATihA
      endp-GXvVZRRWU74XLYiTDjYHgN endp-jE8GmTFEUgAbUYoFkEzN8Z endp-iv7S4DTZvjUg8jrxfcTaSB
      endp-6khsAZ8VVWoXzCx5niJiJo]'
    text: '[UPDATED] testcase-5b5RAopxAVyuMMz4jSpq84 body'
    short_fields: false
    link_names: false
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '[UPDATED] testcase-5b5RAopxAVyuMMz4jSpq84 to Endpoints [endp-H5rtQnGvav69kCJfgPWTuJ
      endp-oezwYn49Q7U3MBZF8LqgUi endp-LJZdtF9PLPkinChrmgTm8F endp-7LfZWCsAgrYCT4bPZATihA
      endp-GXvVZRRWU74XLYiTDjYHgN endp-jE8GmTFEUgAbUYoFkEzN8Z endp-iv7S4DTZvjUg8jrxfcTaSB
      endp-6khsAZ8VVWoXzCx5niJiJo]'
    text: '[UPDATED] testcase-5b5RAopxAVyuMMz4jSpq84 body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
  - send_resolved: false
    url: https://webhook.com/endp-oezwYn49Q7U3MBZF8LqgUi
    max_alerts: 0
- name: opni.cpu.BfadEokVjMAfEyxSB6Ys7T
  email_configs:
  - send_resolved: false
    to: endp-LJZdtF9PLPkinChrmgTm8F-to@gmail.com
    from: endp-LJZdtF9PLPkinChrmgTm8F-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-BfadEokVjMAfEyxSB6Ys7T body'
    html: '[UPDATED] testcase-BfadEokVjMAfEyxSB6Ys7T body'
  - send_resolved: false
    to: endp-jE8GmTFEUgAbUYoFkEzN8Z-to@gmail.com
    from: endp-jE8GmTFEUgAbUYoFkEzN8Z-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-BfadEokVjMAfEyxSB6Ys7T body'
    html: '[UPDATED] testcase-BfadEokVjMAfEyxSB6Ys7T body'
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '[UPDATED] testcase-BfadEokVjMAfEyxSB6Ys7T body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-BfadEokVjMAfEyxSB6Ys7T to Endpoints [endp-6khsAZ8VVWoXzCx5niJiJo
        endp-iv7S4DTZvjUg8jrxfcTaSB endp-oezwYn49Q7U3MBZF8LqgUi endp-7LfZWCsAgrYCT4bPZATihA
        endp-GXvVZRRWU74XLYiTDjYHgN endp-LJZdtF9PLPkinChrmgTm8F endp-jE8GmTFEUgAbUYoFkEzN8Z]'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '[UPDATED] testcase-BfadEokVjMAfEyxSB6Ys7T to Endpoints [endp-6khsAZ8VVWoXzCx5niJiJo
      endp-iv7S4DTZvjUg8jrxfcTaSB endp-oezwYn49Q7U3MBZF8LqgUi endp-7LfZWCsAgrYCT4bPZATihA
      endp-GXvVZRRWU74XLYiTDjYHgN endp-LJZdtF9PLPkinChrmgTm8F endp-jE8GmTFEUgAbUYoFkEzN8Z]'
    text: '[UPDATED] testcase-BfadEokVjMAfEyxSB6Ys7T body'
    short_fields: false
    link_names: false
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '[UPDATED] testcase-BfadEokVjMAfEyxSB6Ys7T to Endpoints [endp-6khsAZ8VVWoXzCx5niJiJo
      endp-iv7S4DTZvjUg8jrxfcTaSB endp-oezwYn49Q7U3MBZF8LqgUi endp-7LfZWCsAgrYCT4bPZATihA
      endp-GXvVZRRWU74XLYiTDjYHgN endp-LJZdtF9PLPkinChrmgTm8F endp-jE8GmTFEUgAbUYoFkEzN8Z]'
    text: '[UPDATED] testcase-BfadEokVjMAfEyxSB6Ys7T body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
  - send_resolved: false
    url: https://webhook.com/endp-oezwYn49Q7U3MBZF8LqgUi
    max_alerts: 0
- name: opni.cpu.NvEpKD6J6NiB8GyJRwCrtK
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '[UPDATED] testcase-NvEpKD6J6NiB8GyJRwCrtK to Endpoints [endp-7LfZWCsAgrYCT4bPZATihA
      endp-iv7S4DTZvjUg8jrxfcTaSB]'
    text: '[UPDATED] testcase-NvEpKD6J6NiB8GyJRwCrtK body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
- name: opni.disconnect.ChbcwZs8UFptwcpyGDxyzH
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '[UPDATED] testcase-ChbcwZs8UFptwcpyGDxyzH to Endpoints [endp-GXvVZRRWU74XLYiTDjYHgN
      endp-oezwYn49Q7U3MBZF8LqgUi endp-7LfZWCsAgrYCT4bPZATihA]'
    text: '[UPDATED] testcase-ChbcwZs8UFptwcpyGDxyzH body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
  - send_resolved: false
    url: https://webhook.com/endp-oezwYn49Q7U3MBZF8LqgUi
    max_alerts: 0
- name: opni.memory.5qHohrmEg9SkCQcrnx9Q
  email_configs:
  - send_resolved: false
    to: endp-LJZdtF9PLPkinChrmgTm8F-to@gmail.com
    from: endp-LJZdtF9PLPkinChrmgTm8F-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-5qHohrmEg9SkCQcrnx9Q body'
    html: '[UPDATED] testcase-5qHohrmEg9SkCQcrnx9Q body'
  - send_resolved: false
    to: endp-jE8GmTFEUgAbUYoFkEzN8Z-to@gmail.com
    from: endp-jE8GmTFEUgAbUYoFkEzN8Z-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-5qHohrmEg9SkCQcrnx9Q body'
    html: '[UPDATED] testcase-5qHohrmEg9SkCQcrnx9Q body'
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '[UPDATED] testcase-5qHohrmEg9SkCQcrnx9Q body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-5qHohrmEg9SkCQcrnx9Q to Endpoints [endp-LJZdtF9PLPkinChrmgTm8F
        endp-H5rtQnGvav69kCJfgPWTuJ endp-jE8GmTFEUgAbUYoFkEzN8Z]'
- name: opni.memory.BbRnAdfGESVx3zFzKX9NfD
  email_configs:
  - send_resolved: false
    to: endp-LJZdtF9PLPkinChrmgTm8F-to@gmail.com
    from: endp-LJZdtF9PLPkinChrmgTm8F-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-BbRnAdfGESVx3zFzKX9NfD body'
    html: '[UPDATED] testcase-BbRnAdfGESVx3zFzKX9NfD body'
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '[UPDATED] testcase-BbRnAdfGESVx3zFzKX9NfD body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-BbRnAdfGESVx3zFzKX9NfD to Endpoints [endp-6khsAZ8VVWoXzCx5niJiJo
        endp-LJZdtF9PLPkinChrmgTm8F endp-GXvVZRRWU74XLYiTDjYHgN endp-H5rtQnGvav69kCJfgPWTuJ
        endp-iv7S4DTZvjUg8jrxfcTaSB]'
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '[UPDATED] testcase-BbRnAdfGESVx3zFzKX9NfD body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-BbRnAdfGESVx3zFzKX9NfD to Endpoints [endp-6khsAZ8VVWoXzCx5niJiJo
        endp-LJZdtF9PLPkinChrmgTm8F endp-GXvVZRRWU74XLYiTDjYHgN endp-H5rtQnGvav69kCJfgPWTuJ
        endp-iv7S4DTZvjUg8jrxfcTaSB]'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '[UPDATED] testcase-BbRnAdfGESVx3zFzKX9NfD to Endpoints [endp-6khsAZ8VVWoXzCx5niJiJo
      endp-LJZdtF9PLPkinChrmgTm8F endp-GXvVZRRWU74XLYiTDjYHgN endp-H5rtQnGvav69kCJfgPWTuJ
      endp-iv7S4DTZvjUg8jrxfcTaSB]'
    text: '[UPDATED] testcase-BbRnAdfGESVx3zFzKX9NfD body'
    short_fields: false
    link_names: false
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '[UPDATED] testcase-BbRnAdfGESVx3zFzKX9NfD to Endpoints [endp-6khsAZ8VVWoXzCx5niJiJo
      endp-LJZdtF9PLPkinChrmgTm8F endp-GXvVZRRWU74XLYiTDjYHgN endp-H5rtQnGvav69kCJfgPWTuJ
      endp-iv7S4DTZvjUg8jrxfcTaSB]'
    text: '[UPDATED] testcase-BbRnAdfGESVx3zFzKX9NfD body'
    short_fields: false
    link_names: false
- name: opni.memory.DJUgDdGyHPLPxbP6YvfoHF
  email_configs:
  - send_resolved: false
    to: endp-LJZdtF9PLPkinChrmgTm8F-to@gmail.com
    from: endp-LJZdtF9PLPkinChrmgTm8F-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-DJUgDdGyHPLPxbP6YvfoHF body'
    html: '[UPDATED] testcase-DJUgDdGyHPLPxbP6YvfoHF body'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '[UPDATED] testcase-DJUgDdGyHPLPxbP6YvfoHF to Endpoints [endp-iv7S4DTZvjUg8jrxfcTaSB
      endp-7LfZWCsAgrYCT4bPZATihA endp-oezwYn49Q7U3MBZF8LqgUi endp-LJZdtF9PLPkinChrmgTm8F]'
    text: '[UPDATED] testcase-DJUgDdGyHPLPxbP6YvfoHF body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
  - send_resolved: false
    url: https://webhook.com/endp-oezwYn49Q7U3MBZF8LqgUi
    max_alerts: 0
- name: opni.memory.k2pUZAGL4MAeEe98UjKDf6
  email_configs:
  - send_resolved: false
    to: endp-LJZdtF9PLPkinChrmgTm8F-to@gmail.com
    from: endp-LJZdtF9PLPkinChrmgTm8F-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-k2pUZAGL4MAeEe98UjKDf6 body'
    html: '[UPDATED] testcase-k2pUZAGL4MAeEe98UjKDf6 body'
  - send_resolved: false
    to: endp-jE8GmTFEUgAbUYoFkEzN8Z-to@gmail.com
    from: endp-jE8GmTFEUgAbUYoFkEzN8Z-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-k2pUZAGL4MAeEe98UjKDf6 body'
    html: '[UPDATED] testcase-k2pUZAGL4MAeEe98UjKDf6 body'
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '[UPDATED] testcase-k2pUZAGL4MAeEe98UjKDf6 body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-k2pUZAGL4MAeEe98UjKDf6 to Endpoints [endp-7LfZWCsAgrYCT4bPZATihA
        endp-iv7S4DTZvjUg8jrxfcTaSB endp-H5rtQnGvav69kCJfgPWTuJ endp-6khsAZ8VVWoXzCx5niJiJo
        endp-jE8GmTFEUgAbUYoFkEzN8Z endp-LJZdtF9PLPkinChrmgTm8F endp-GXvVZRRWU74XLYiTDjYHgN]'
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '[UPDATED] testcase-k2pUZAGL4MAeEe98UjKDf6 body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-k2pUZAGL4MAeEe98UjKDf6 to Endpoints [endp-7LfZWCsAgrYCT4bPZATihA
        endp-iv7S4DTZvjUg8jrxfcTaSB endp-H5rtQnGvav69kCJfgPWTuJ endp-6khsAZ8VVWoXzCx5niJiJo
        endp-jE8GmTFEUgAbUYoFkEzN8Z endp-LJZdtF9PLPkinChrmgTm8F endp-GXvVZRRWU74XLYiTDjYHgN]'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '[UPDATED] testcase-k2pUZAGL4MAeEe98UjKDf6 to Endpoints [endp-7LfZWCsAgrYCT4bPZATihA
      endp-iv7S4DTZvjUg8jrxfcTaSB endp-H5rtQnGvav69kCJfgPWTuJ endp-6khsAZ8VVWoXzCx5niJiJo
      endp-jE8GmTFEUgAbUYoFkEzN8Z endp-LJZdtF9PLPkinChrmgTm8F endp-GXvVZRRWU74XLYiTDjYHgN]'
    text: '[UPDATED] testcase-k2pUZAGL4MAeEe98UjKDf6 body'
    short_fields: false
    link_names: false
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '[UPDATED] testcase-k2pUZAGL4MAeEe98UjKDf6 to Endpoints [endp-7LfZWCsAgrYCT4bPZATihA
      endp-iv7S4DTZvjUg8jrxfcTaSB endp-H5rtQnGvav69kCJfgPWTuJ endp-6khsAZ8VVWoXzCx5niJiJo
      endp-jE8GmTFEUgAbUYoFkEzN8Z endp-LJZdtF9PLPkinChrmgTm8F endp-GXvVZRRWU74XLYiTDjYHgN]'
    text: '[UPDATED] testcase-k2pUZAGL4MAeEe98UjKDf6 body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
- name: opni.prometheusQuery.MG6LkNmDEobUq85a9ZJoFm
  email_configs:
  - send_resolved: false
    to: endp-jE8GmTFEUgAbUYoFkEzN8Z-to@gmail.com
    from: endp-jE8GmTFEUgAbUYoFkEzN8Z-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-MG6LkNmDEobUq85a9ZJoFm body'
    html: '[UPDATED] testcase-MG6LkNmDEobUq85a9ZJoFm body'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '[UPDATED] testcase-MG6LkNmDEobUq85a9ZJoFm to Endpoints [endp-jE8GmTFEUgAbUYoFkEzN8Z
      endp-GXvVZRRWU74XLYiTDjYHgN]'
    text: '[UPDATED] testcase-MG6LkNmDEobUq85a9ZJoFm body'
    short_fields: false
    link_names: false
- name: opni.prometheusQuery.WbCdYxccA5fev4bUkXLV7h
  email_configs:
  - send_resolved: false
    to: endp-LJZdtF9PLPkinChrmgTm8F-to@gmail.com
    from: endp-LJZdtF9PLPkinChrmgTm8F-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-WbCdYxccA5fev4bUkXLV7h body'
    html: '[UPDATED] testcase-WbCdYxccA5fev4bUkXLV7h body'
  - send_resolved: false
    to: endp-jE8GmTFEUgAbUYoFkEzN8Z-to@gmail.com
    from: endp-jE8GmTFEUgAbUYoFkEzN8Z-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-WbCdYxccA5fev4bUkXLV7h body'
    html: '[UPDATED] testcase-WbCdYxccA5fev4bUkXLV7h body'
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '[UPDATED] testcase-WbCdYxccA5fev4bUkXLV7h body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-WbCdYxccA5fev4bUkXLV7h to Endpoints [endp-7LfZWCsAgrYCT4bPZATihA
        endp-iv7S4DTZvjUg8jrxfcTaSB endp-LJZdtF9PLPkinChrmgTm8F endp-jE8GmTFEUgAbUYoFkEzN8Z
        endp-GXvVZRRWU74XLYiTDjYHgN endp-6khsAZ8VVWoXzCx5niJiJo]'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '[UPDATED] testcase-WbCdYxccA5fev4bUkXLV7h to Endpoints [endp-7LfZWCsAgrYCT4bPZATihA
      endp-iv7S4DTZvjUg8jrxfcTaSB endp-LJZdtF9PLPkinChrmgTm8F endp-jE8GmTFEUgAbUYoFkEzN8Z
      endp-GXvVZRRWU74XLYiTDjYHgN endp-6khsAZ8VVWoXzCx5niJiJo]'
    text: '[UPDATED] testcase-WbCdYxccA5fev4bUkXLV7h body'
    short_fields: false
    link_names: false
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '[UPDATED] testcase-WbCdYxccA5fev4bUkXLV7h to Endpoints [endp-7LfZWCsAgrYCT4bPZATihA
      endp-iv7S4DTZvjUg8jrxfcTaSB endp-LJZdtF9PLPkinChrmgTm8F endp-jE8GmTFEUgAbUYoFkEzN8Z
      endp-GXvVZRRWU74XLYiTDjYHgN endp-6khsAZ8VVWoXzCx5niJiJo]'
    text: '[UPDATED] testcase-WbCdYxccA5fev4bUkXLV7h body'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
- name: opni.prometheusQuery.coKWKPD5mpjyZtsVFgcMsc
  email_configs:
  - send_resolved: false
    to: endp-LJZdtF9PLPkinChrmgTm8F-to@gmail.com
    from: endp-LJZdtF9PLPkinChrmgTm8F-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-coKWKPD5mpjyZtsVFgcMsc body'
    html: '[UPDATED] testcase-coKWKPD5mpjyZtsVFgcMsc body'
  - send_resolved: false
    to: endp-jE8GmTFEUgAbUYoFkEzN8Z-to@gmail.com
    from: endp-jE8GmTFEUgAbUYoFkEzN8Z-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-coKWKPD5mpjyZtsVFgcMsc body'
    html: '[UPDATED] testcase-coKWKPD5mpjyZtsVFgcMsc body'
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '[UPDATED] testcase-coKWKPD5mpjyZtsVFgcMsc body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-coKWKPD5mpjyZtsVFgcMsc to Endpoints [endp-GXvVZRRWU74XLYiTDjYHgN
        endp-jE8GmTFEUgAbUYoFkEzN8Z endp-LJZdtF9PLPkinChrmgTm8F endp-6khsAZ8VVWoXzCx5niJiJo
        endp-H5rtQnGvav69kCJfgPWTuJ]'
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '[UPDATED] testcase-coKWKPD5mpjyZtsVFgcMsc body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-coKWKPD5mpjyZtsVFgcMsc to Endpoints [endp-GXvVZRRWU74XLYiTDjYHgN
        endp-jE8GmTFEUgAbUYoFkEzN8Z endp-LJZdtF9PLPkinChrmgTm8F endp-6khsAZ8VVWoXzCx5niJiJo
        endp-H5rtQnGvav69kCJfgPWTuJ]'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '[UPDATED] testcase-coKWKPD5mpjyZtsVFgcMsc to Endpoints [endp-GXvVZRRWU74XLYiTDjYHgN
      endp-jE8GmTFEUgAbUYoFkEzN8Z endp-LJZdtF9PLPkinChrmgTm8F endp-6khsAZ8VVWoXzCx5niJiJo
      endp-H5rtQnGvav69kCJfgPWTuJ]'
    text: '[UPDATED] testcase-coKWKPD5mpjyZtsVFgcMsc body'
    short_fields: false
    link_names: false
- name: opni.prometheusQuery.iLBBaQ5LaGAhLi7vSj9C8F
  email_configs:
  - send_resolved: false
    to: endp-LJZdtF9PLPkinChrmgTm8F-to@gmail.com
    from: endp-LJZdtF9PLPkinChrmgTm8F-from@gmail.com
    smarthost: localhost:55567
    auth_username: test
    auth_password: password
    auth_identity: none
    headers:
      Subject: '[UPDATED] testcase-iLBBaQ5LaGAhLi7vSj9C8F body'
    html: '[UPDATED] testcase-iLBBaQ5LaGAhLi7vSj9C8F body'
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '[UPDATED] testcase-iLBBaQ5LaGAhLi7vSj9C8F body'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '[UPDATED] testcase-iLBBaQ5LaGAhLi7vSj9C8F to Endpoints [endp-7LfZWCsAgrYCT4bPZATihA
        endp-oezwYn49Q7U3MBZF8LqgUi endp-LJZdtF9PLPkinChrmgTm8F endp-H5rtQnGvav69kCJfgPWTuJ]'
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
  - send_resolved: false
    url: https://webhook.com/endp-oezwYn49Q7U3MBZF8LqgUi
    max_alerts: 0
- name: opni.OpniSeverity.Critical
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '{{ .OpniBodyLabel }}'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '{{ .OpniTitleLabel }}'
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '{{ .OpniBodyLabel }}'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '{{ .OpniTitleLabel }}'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '{{ .OpniTitleLabel }}'
    text: '{{ .OpniBodyLabel }}'
    short_fields: false
    link_names: false
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '{{ .OpniTitleLabel }}'
    text: '{{ .OpniBodyLabel }}'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
  - send_resolved: false
    url: https://webhook.com/endp-oezwYn49Q7U3MBZF8LqgUi
    max_alerts: 0
- name: opni.OpniSeverity.Error
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '{{ .OpniBodyLabel }}'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '{{ .OpniTitleLabel }}'
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '{{ .OpniBodyLabel }}'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '{{ .OpniTitleLabel }}'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '{{ .OpniTitleLabel }}'
    text: '{{ .OpniBodyLabel }}'
    short_fields: false
    link_names: false
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '{{ .OpniTitleLabel }}'
    text: '{{ .OpniBodyLabel }}'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
  - send_resolved: false
    url: https://webhook.com/endp-oezwYn49Q7U3MBZF8LqgUi
    max_alerts: 0
- name: opni.OpniSeverity.Info
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '{{ .OpniBodyLabel }}'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '{{ .OpniTitleLabel }}'
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '{{ .OpniBodyLabel }}'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '{{ .OpniTitleLabel }}'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '{{ .OpniTitleLabel }}'
    text: '{{ .OpniBodyLabel }}'
    short_fields: false
    link_names: false
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '{{ .OpniTitleLabel }}'
    text: '{{ .OpniBodyLabel }}'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
  - send_resolved: false
    url: https://webhook.com/endp-oezwYn49Q7U3MBZF8LqgUi
    max_alerts: 0
- name: opni.OpniSeverity.Warning
  pagerduty_configs:
  - send_resolved: false
    service_key: https://pagerduty.com/endp-6khsAZ8VVWoXzCx5niJiJo
    description: '{{ .OpniBodyLabel }}'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '{{ .OpniTitleLabel }}'
  - send_resolved: false
    service_key: https://pagerduty.com/endp-H5rtQnGvav69kCJfgPWTuJ
    description: '{{ .OpniBodyLabel }}'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
      title: '{{ .OpniTitleLabel }}'
  slack_configs:
  - send_resolved: false
    api_url: https://slack.com/endp-GXvVZRRWU74XLYiTDjYHgN
    channel: '#test-endp-GXvVZRRWU74XLYiTDjYHgN'
    title: '{{ .OpniTitleLabel }}'
    text: '{{ .OpniBodyLabel }}'
    short_fields: false
    link_names: false
  - send_resolved: false
    api_url: https://slack.com/endp-iv7S4DTZvjUg8jrxfcTaSB
    channel: '#test-endp-iv7S4DTZvjUg8jrxfcTaSB'
    title: '{{ .OpniTitleLabel }}'
    text: '{{ .OpniBodyLabel }}'
    short_fields: false
    link_names: false
  webhook_configs:
  - send_resolved: false
    url: https://webhook.com/endp-7LfZWCsAgrYCT4bPZATihA
    max_alerts: 0
  - send_resolved: false
    url: https://webhook.com/endp-oezwYn49Q7U3MBZF8LqgUi
    max_alerts: 0
- name: opni.default.hook
  webhook_configs:
  - send_resolved: false
    url: http://localhost:3000
    max_alerts: 0
templates: []

Scale and performance

  • Group by as many labels as we can at the entrypoint, since CPU & memory footprint are determined by the cardinality of incoming labels
  • Reduce cardinality of labels by using labels matchers of the form <namespace>!="" and <namespace>=id instead of namespace=<key>, id=<id> to further reduce cardinality
  • In the metrics subtree, we require to add group_by["..."] since we do not construct the optimizations on the embedded tree
  • Opni-Alerting API scalability is very faintly contingent on building the smallest possible configuration files -- however, size of configuration file doesn't affect the scalability of the runtime performance of AlertManager

Diagram

Use Prometheus's AlertManager visualization tool for interactive diagramming around Opni Alerting's routing model

Restrictions & Limitations

We make opinonated configs that are not necessarily easy to read.

Performance Issues

N/A

Testplan

Unit tests.

Unit tests

  • all the operations on the routing model
  • storage client set tests check for storage functionality

See pkg/alerting/drivers/routing suite for comprehensive tests on the interface.

Integration tests

  • cover using router interface dispatches to receivers correctly
  • cover reconciling alerting configurations (endpoints, conditions) to a router

Security:

  • Security is handled by Alerting Endpoints server redacting information when requested.

Testing:

See unit tests.

Architecture
How Tos
Releases
Clone this wiki locally