Skip to main content

Using Alertmanager with Prometheus

Hands-On Lab

 

Photo of Elle Krout

Elle Krout

Content Team Lead in Content

Length

00:30:00

Difficulty

Intermediate

Prometheus does not just limit us to recording metrics. One of Prometheus's core functionalities is the ability to define and route alerts to any alert management endpoint we define -- or, in the case of this Hands-On Lab, Prometheus's own sideproject, Alertmanager. Once we have our desired alerting thresholds defined, we need to set up our routes and receivers for the Alertmanager, ensuring our notifications are going to the right end user at the correct frequency and with the right information.

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

Add an Alert

  1. Add a rules file configuration to the Prometheus config:

    $ sudo $EDITOR /etc/prometheus/prometheus.yml

    rule_files:

    • "rules.yml"

    Save and exit.

  2. Create and open the rules.yml file:

    $ sudo $EDITOR /etc/prometheus/rules.yml

  3. Before creating the alert itself, create a recording of the desired metric:

    groups:

    • name: uptime rules:
      • record: job:uptime:average:ft expr: avg without (instance) (up{job="forethought"})
  4. Create the alert to see the application can gone down based on this recording:

    groups:

    • name: uptime rules:
      • record: job:uptime:average:ft expr: avg without (instance) (up{job="forethought"})
      • alert: ForethoughtApplicationDown expr: job:uptime:average:ft < .75 for: 30s labels: severity: page team: devops

    Save and exit.

  5. Restart Prometheus:

    $ sudo systemctl restart prometheus $ sudo systemctl status prometheus

Configure Alertmanager

  1. Open the Alertmanager configuration file:

    $ sudo $EDITOR /etc/alertmanager/alertmanager.yml

  2. Define the global settings:

    global: resolve_timeout: 5m smtp_smarthost: 'localhost:25' smtp_from: 'prometheus'

  3. Set up the backup route:

    route: receiver: 'email_backup' group_by: ['alertname'] group_wait: 10s group_interval: 10s repeat_interval: 1m

  4. Set up the route for critical alerts:

    route: receiver: 'email_backup' group_by: ['alertname'] group_wait: 10s group_interval: 10s repeat_interval: 1m routes:

    • match: severity: 'critical' group_by: ['team'] receiver: 'email_pager'
  5. Set up the route for team alerts:

    route: receiver: 'email_backup' group_by: ['alertname'] group_wait: 10s group_interval: 10s repeat_interval: 1m routes:

    • match: severity: 'critical' group_by: ['team'] receiver: 'email_pager' routes:
      • match: team: devops receiver: 'email_devops'
  6. Create the receivers:

    receivers:

    • name: 'email_backup' email_configs:
      • to: 'alerts@forethoughtapp.io'
    • name: 'email_pager' email_configs:
      • to: 'oncall@forethoughtapp.io'
    • name: 'email_devops' email_configs:
      • to: 'devops@forethoughtapp.io'

    Save and exit.

  7. Restart Alertmanager:

    $ sudo systemctl restart alertmanager

  8. Test your alert!