Skip to content

docs: add cluster-wide inhibition example to alerting tutorial #3015

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yugstar wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: add cluster-wide inhibition example to alerting tutorial #3015

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 25 additions & 1 deletion docs/tutorials/alerting_based_on_metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,4 +81,28 @@ Open [http://localhost:9090/rules](http://localhost:9090/rules) in your browser

<iframe width="560" height="315" src="https://www.youtube.com/embed/xaMXVrle98M" frameborder="0" allowfullscreen></iframe>

Similarly Alertmanager can be configured with other receivers to notify when an alert is firing.
Similarly Alertmanager can be configured with other receivers to notify when an alert is firing.

## Inhibiting alerts from an entire cluster

When a whole cluster (or instance) becomes unreachable, you usually don't want a separate notification for every alert that fires as a consequence. Alertmanager's [inhibition](/docs/alerting/latest/alertmanager/#inhibition) feature lets a single "cluster is down" alert mute all the dependent alerts coming from that same cluster, so you receive one meaningful notification instead of a flood.

@jan--f jan--f Jul 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
When a whole cluster (or instance) becomes unreachable, you usually don't want a separate notification for every alert that fires as a consequence. Alertmanager's [inhibition](/docs/alerting/latest/alertmanager/#inhibition) feature lets a single "cluster is down" alert mute all the dependent alerts coming from that same cluster, so you receive one meaningful notification instead of a flood.
When an entire cluster becomes unreachable, you usually don't want separate notifications for every alert that fires as a consequence. Alertmanager's [inhibition](/docs/alerting/latest/alertmanager/#inhibition) feature lets a single, firing "cluster is down" alert mute all alerts coming from that same cluster, so you receive one meaningful notification instead of a flood.


Inhibition is configured with `inhibit_rules` in `alertmanager.yml`. The following rule mutes every alert that shares the same `cluster` label value as a firing `ClusterUnreachable` alert:

> alertmanager.yml

```yaml
inhibit_rules:
- source_matchers:
- 'alertname = "ClusterUnreachable"'
target_matchers:
- 'alertname != "ClusterUnreachable"'
equal:
- 'cluster'
```
Comment on lines +92 to +102

@jan--f jan--f Jul 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's show users the full alertmanager.yml here, similarly to how this is done previously in this file. You can re-use the previous listing and just add the inihibit rule.


- `source_matchers` selects the alert that suppresses others when it is firing (here, `ClusterUnreachable`).
- `target_matchers` selects the alerts to mute. `ClusterUnreachable` is excluded so the source alert itself is still delivered.

@jan--f jan--f Jul 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Below it says the triggering alert is always delivered. Maybe its worth quoting the docs here:

To prevent an alert from inhibiting itself, an alert that matches both the target and the source side of a rule cannot be inhibited by alerts for which the same is true (including itself). However, we recommend to choose target and source matchers in a way that alerts never match both sides. It is much easier to reason about and does not trigger this special case.

- `equal` lists the labels whose values must match between the source and target alerts for the inhibition to apply. Alerts are muted only when they share the **same** `cluster` value, so an outage in one cluster never hides alerts from another.

For this to work, both the `ClusterUnreachable` alert and the alerts you want to mute must carry a `cluster` label, for example set on your alerting rules or added through `external_labels`. An alert is also never inhibited by itself, so `ClusterUnreachable` is always delivered.