Skip to content

Commit

Permalink
more small improvements
Browse files Browse the repository at this point in the history
  • Loading branch information
aantn committed Jan 6, 2025
1 parent 6b4a51c commit 9af1618
Show file tree
Hide file tree
Showing 5 changed files with 49 additions and 23 deletions.
2 changes: 1 addition & 1 deletion docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ Better Prometheus Alerts (and more) for Kubernetes
* :ref:`AI Investigation <AI Analysis>` - Kickstart your alert investigations with AI (optional)
* :ref:`Alert Enrichment <Automatically Investigate a Prometheus Alert>` - see pods log and other data alongside your alerts
* :ref:`Self-Healing <Remediate Prometheus Alerts>` - define auto-remediation rules for faster fixes
* :ref:`Advanced Routing <Notification Basics>` by team, namespace, k8s metadata and more
* :ref:`Advanced Routing <Defining Sinks>` by team, namespace, k8s metadata and more
* :ref:`K8s Problem-Detection <Triggers Reference>` - alert on OOMKills or failing Jobs without PromQL
* :ref:`Change Tracking <Track Kubernetes Changes>` - correlate alerts and Kubernetes rollouts
* :ref:`Auto-Resolve <Jira>` - send alerts, resolve them when updated (e.g. in Jira)
Expand Down
45 changes: 31 additions & 14 deletions docs/notification-routing/configuring-sinks.rst
Original file line number Diff line number Diff line change
@@ -1,31 +1,47 @@
.. _sinks-overview:

Notification Basics
Defining Sinks
==========================

Robusta can send notifications to various destinations, called sinks. For a list of all sinks, refer to :ref:`Sinks Reference`.
Robusta can send notifications to various destinations, called sinks.

Defining Sinks
^^^^^^^^^^^^^^^^^^
For a list of all sinks, refer to :ref:`Sinks Reference`.

A Simple Sink Configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sinks are defined in Robusta's Helm chart, using the ``sinksConfig`` value.

For example, lets add a :ref:`Microsoft Teams <MS Teams sink>`:
For example, lets add a :ref:`Microsoft Teams Sink <MS Teams sink>`:

.. code-block:: yaml
sinksConfig:
- ms_teams_sink: # sink type
name: my_teams_sink # name that uniquely identifies this sink in Robusta
webhook_url: <placeholder> # the webhook URL for MSTeams - each sink has different parameters like this
More Sink Options
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Here are some more options you can set on sinks:

.. code-block:: yaml
sinksConfig:
- ms_teams_sink: # sink type
name: my_teams_sink # arbitrary name
webhook_url: <placeholder> # a sink-specific parameter
stop: false # optional (see `Routing Alerts to only one Sink`)
scope: {} # optional routing rules
default: true # optional (see below)
name: my_teams_sink # name that uniquely identifies this sink in Robusta
stop: false # optional - covered in the next tutorial
scope: {} # optional - covered in the next tutorial
default: true # optional - covered in the next tutorial
# sink-specific parameters
webhook_url: <placeholder>
In addition to the above attributes, some sinks have additional attributes that can be found in their documentation.
The ``stop``, ``scope``, and ``default`` fields are used for :ref:`routing (scopes) <sink-scope-matching>`.

Defining Multiple Sinks
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You can define multiple sinks and by default, notifications will be sent to all of them.
You can define multiple sinks. By default, notifications will be sent to all of them.

In the following example, we define a :ref:`Slack sink <Slack>` and a :ref:`MS Teams sink <MS Teams>` without any routing rules, so both sinks receive all notifications:

Expand All @@ -42,7 +58,8 @@ In the following example, we define a :ref:`Slack sink <Slack>` and a :ref:`MS T
To selectively send notifications to different sinks, refer to :ref:`routing (scopes) <sink-scope-matching>`.

See Also
Learn More
^^^^^^^^^^^^

🔔 :ref:`All Sinks <Sinks Reference>`
* 🔔 :ref:`All Sinks <Sinks Reference>`
* ↳ :ref:`Routing (scopes) <sink-scope-matching>`
Original file line number Diff line number Diff line change
Expand Up @@ -33,4 +33,4 @@ In this example we'll route alerts to different support teams based on the time
- start: 12:00 # 12 PM
end: 23:59 # 11:59 PM
For more information on time-based routing, :ref:`view <Route by Time>`
For more information on time-based routing, view :ref:`Route by Time`.
22 changes: 15 additions & 7 deletions docs/notification-routing/routing-by-namespace.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Prerequisites
Setting Up Routing
----------------------

Assume you have an existing sink, in this case Slack:
Assume you have an existing sink:

.. code-block:: yaml
Expand All @@ -21,7 +21,9 @@ Assume you have an existing sink, in this case Slack:
slack_channel: app-notifications
api_key: secret-key
By default, the sink will receive notifications for all namespaces. Let's duplicate the sink and change only the Slack channel:
By default, the sink will receive notifications for all namespaces.

Let's create a 2nd copy of the sink and change the Slack channel:

.. code-block:: yaml
Expand All @@ -35,7 +37,9 @@ By default, the sink will receive notifications for all namespaces. Let's duplic
slack_channel: system-notifications
api_key: secret-key
We now have two sinks, both receiving all notifications. Restrict the notifications for each sink by adding :ref:`scopes <sink-matchers>`:
Now we have two sinks, both receiving all notifications.

The final step is to restrict notifications for each sink by adding :ref:`scopes <sink-scope-matching>`:

.. code-block:: yaml
Expand All @@ -58,9 +62,13 @@ We now have two sinks, both receiving all notifications. Restrict the notificati
Alerts will be now routed according to Kubernetes namespace.

You can apply this method with as many sinks as you like. If the number of sinks is large, consider setting the channel dynamically. See instructions for :ref:`Slack <Dynamic Alert Routing>` or :ref:`MS Teams <Dynamically Route MS Teams Alerts>`.
Routing for many namespaces
-----------------------------------
Using the above method, you can create one sink for each namespace.

If you have a large number of namespaces, there is an alternative method you can consider: you can define a single sink and set the channel dynamically according to alert metadata. See instructions for :ref:`Slack <Dynamic Alert Routing>` or :ref:`MS Teams <Dynamically Route MS Teams Alerts>`.

Troubleshooting Issues
------------------------
Fallback for alerts without a namespace
--------------------------------------------------

For this guide to work, alerts must be tagged with ``namespace`` metadata. It is recommended that you setup a :ref:`Fallback Sink <Stop Further Notifications>` to catch alerts that don't have a namespace.
The above example assumes that all alerts have ``namespace`` metadata. It is recommended that you setup a :ref:`Fallback Sink <Stop Further Notifications>` to catch alerts that don't have a namespace.
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,4 @@ In this example, we'll route alerts to different Slack channels depending on an
include:
- namespace: [backend]
For more details, refer to :ref:`scopes <sink-scope-matching>`.

0 comments on commit 9af1618

Please sign in to comment.