Skip to main content

Using Prometheus Recording Rules

Hands-On Lab

 

Photo of Will Boyd

Will Boyd

DevOps Team Lead in Content

Length

00:45:00

Difficulty

Intermediate

Prometheus is a great way to collect a variety of metrics for your servers and applications. However, sometimes you need to perform calculations on the raw data Prometheus collects in order to gain actionable information. While queries can help you do this, recording rules provide an additional layer of optimization by allowing you to periodically pre-calculate query results and save them. In this lab, you will have the opportunity to implement some basic recording rules. This will provide you with some hands-on familiarity with the process of creating and managing recording rules.

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

Using Prometheus Recording Rules

Introduction

Prometheus is a great way to collect a variety of metrics for your servers and applications. However, sometimes you need to perform calculations on the raw data Prometheus collects in order to gain actionable information. While queries can help you do this, recording rules provide an additional layer of optimization by allowing you to periodically pre-calculate query results and save them. In this lab, you will have the opportunity to implement some basic recording rules. This will provide you with some hands-on familiarity with the process of creating and managing recording rules.

Solution

Log in to the Prometheus server using the credentials provided:

ssh cloud_user@<PROMETHEUS_SERVER_PUBLIC_IP>

Implement a Recording Rule to Pre-Calculate CPU Usage

  1. Create a directory to store rules files:

    sudo mkdir -p /etc/prometheus/rules
  2. Open the Prometheus config:

    sudo vi /etc/prometheus/prometheus.yml
  3. Edit it to add a rules location:

    ...
    
    rule_files:
      - "/etc/prometheus/rules/*.yml"
    
    ...
  4. Save and exit the file by pressing Escape followed by :wq.

  5. Create a rules file for limedrop-gateway server metrics:

    sudo vi /etc/prometheus/rules/limedrop-gateway.yml
  6. Implement a recording rule to pre-calculate CPU usage for the server:

    groups:
    - name: limedrop_gateway
      rules:
      - record: limedrop_gateway:cpu_usage
        expr: sum(rate(node_cpu_seconds_total{instance='limedrop-gateway:9100',mode!='idle'}[5m])) * 100 / 2
  7. Save and exit the file by pressing Escape followed by :wq.

  8. Restart Prometheus to load the new configuration:

    sudo systemctl restart prometheus
  9. Access the expression browser: http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090/graph.

  10. Run a query to verify you can see the pre-calculated data:

    limedrop_gateway:cpu_usage[5m]

Add a Recording Rule to Pre-Calculate Memory Usage

  1. On the Prometheus server, open the rules file:

    sudo vi /etc/prometheus/rules/limedrop-gateway.yml
  2. Edit it to add a recording rule to pre-calculate memory usage for the server:

    groups:
    - name: limedrop_gateway
      rules:
      - record: limedrop_gateway:cpu_usage
        expr: sum(rate(node_cpu_seconds_total{instance='limedrop-gateway:9100',mode!='idle'}[5m])) * 100 / 2
      - record: limedrop_gateway:memory_usage
        expr: 100 - (node_memory_MemAvailable_bytes{instance='limedrop-gateway:9100'} / node_memory_MemTotal_bytes{instance='limedrop-gateway:9100'}) * 100
  3. Save and exit the file by pressing Escape followed by :wq.

  4. Restart Prometheus to load the new configuration:

    sudo systemctl restart prometheus
  5. Access the expression browser: http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090/graph.

  6. Run a query to verify you can see the pre-calculated data:

    limedrop_gateway:memory_usage[5m]
  7. You can also check out the rules interface (http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090/rules) to see the status of your rules.

Conclusion

Congratulations on successfully completing this hands-on lab!