Skip to main content

Creating a Recording Rule

Hands-On Lab

 

Photo of Travis Thomsen

Travis Thomsen

Course Development Director in Content

Length

01:00:00

Difficulty

Intermediate

One of our Prometheus expressions in Grafana is running a bit slow. After a careful diagnosis, we’ve decided that the expression would be more effective if it was turned into a recording rule. We will need to configure our Prometheus environment to be able to use recording rules.

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

Creating a Recording Rule

The Scenario

One of our Prometheus expressions in Grafana is running a bit slow. After a careful diagnosis, we’ve decided that the expression would be more effective if it was turned into a recording rule. We will need to configure our Prometheus environment to be able to use recording rules.

Logging In and Setting up the Environment

Use the IP address and credentials provided on the hands-on lab overview page, and log in with SSH to the server. Once we're in, become root right off and navigate into /root/prometheus. In that directory, there is a bootstrap.sh script that we need to execute. Once we've done that, we can use kubectl to show running pods. There should be two, and they both should be in a running state. Here are all of those commands:

sudo su -
cd /root/prometheus
./bootstrap.sh
kubectl get pods -n monitoring

Create the Recording Rule

  1. Add the job:prcoess_cpu_seconds:rate5m recording rule to prometheus-rules-config-map.yml. The file should look like this when we're done editing it:
apiVersion: v1
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: prometheus-rules-conf
  namespace: monitoring
data:
  process_rules.yml: |
    groups:
    - name: process_rules
      rules:
      - record: job:process_cpu_seconds:rate5m
        expr: (rate(process_cpu_seconds_total[5m]))
  1. Apply the changes with this command:
kubectl apply -f prometheus-rules-config-map.yml

Update the Prometheus Configuration

  1. Edit prometheus-config-map.yml

    vi prometheus-config-map.yml
  2. Add the rule files to prometheus-config-map.yml:

    rule_files:
    - /var/prometheus/rules/*_rules.yml
  3. It should look like this when we're done:

    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: prometheus-server-conf
    labels:
    name: prometheus-server-conf
    namespace: monitoring
    data:
    prometheus.yml: |-
    global:
      scrape_interval: 5s
      evaluation_interval: 5s
    
    rule_files:
    - /var/prometheus/rules/*_rules.yml
    
    scrape_configs:
      - job_name: 'kubernetes-apiservers'
    ...
  4. Apply the changes with this command:

    kubectl apply -f prometheus-config-map.yml

Add prometheus-rules-conf to the Prometheus Deployment

Edit prometheus-deployment.yml and add the prometheus-rules-volume volume to it. We've got to add a name and a mountPath down in the volumeMounts section. Then we've got to add the name again, and where the name of the confgMap, down in the volumes section. When we're finished, it should look like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus-deployment
  namespace: monitoring
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:v2.2.1
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
            - "--web.enable-lifecycle"
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
            - name: prometheus-rules-volume
              mountPath: /var/prometheus/rules
      volumes:
        - name: prometheus-config-volume
          configMap:
            defaultMode: 420
            name: prometheus-server-conf

        - name: prometheus-rules-volume
          configMap:
            name: prometheus-rules-conf

        - name: prometheus-storage-volume
          emptyDir: {}
...

Apply the changes with this command:

kubectl apply -f prometheus-deployment.yml

View It in a Browser

Back on the hands-on lab overview page, grab our server's public IP. In a new browser tab, visit that IP on port 8080:

http://<IP>:8080

In the web interface we land on, click the Status dropdown, and select Rules.

Conclusion

If we see our process recording rule there, we've succeeded. Congratulations!