Using Prometheus with Kubernetes

Hands-On Lab

 

Photo of John Marx

John Marx

Training Architect

Length

01:00:00

Difficulty

Beginner

This lab guides the student through basic Prometheus Queries (PromQL) and demonstrates how the Kubernetes architecture may be interrogated. The lab also introduces a simple means of stressing a cluster and demonstrates how those techniques affect the metrics being stored in the time series database.

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

Using Prometheus with Kubernetes

Introduction

This lab guides the student through basic Prometheus Queries (PromQL) and demonstrates how the Kubernetes architecture may be interrogated. The lab also introduces a simple means of stressing a cluster and demonstrates how those techniques affect the metrics being stored in the time series database.

Solution

Check that your lab environment is ready

  1. Log in to the master node:

    ssh cloud_user@[Public IP here]
  2. List the home directory contents:

    ls -l 
  3. Verify that stress-test.yaml is present.

Access Prometheus from your browser

  1. Access Prometheus from your browser:

    http://[Master Node Public IP address]:9090

    Status > Targets will show you more information on the targets in the cluster.

  2. You may also access the cAdvisor dashboard at the following address:

    http://[Worker Node Public IP Address]:8080

Use the provided PromQL queries to interrogate your cluster

The following are the suggested PromQL queries you may perform.

  1. To measure CPU utilization:

    node_cpu_seconds_total
    irate(node_cpu_seconds_total{job="node"}[5m])
    avg(irate(node_cpu_seconds_total{job="node"}[5m])) by (instance)
    avg(irate(node_cpu_seconds_total{job="node",mode="idle"}[5m])) by (instance) * 100
    100 - avg(irate(node_cpu_seconds_total{job="node",mode="idle"}[5m])) by (instance) * 100
  2. To measure memory, use:

    (node_memory_MemTotal_bytes - (node_memory_MemFree_bytes + node_memory_Cached_bytes + node_memory_Buffers_bytes)) / node_memory_MemTotal_bytes * 100

Deploy the stress test and vary its replicas to examine changes

In the terminal emulator session that is established to the Master Node:

  1. Deploy the Stress-Test Deployment:

    kubectl create -f stress-test.yaml
  2. Interrogate the number of replicas deployed:

    kubectl get deployments
  3. Interrogate the pods running:

    kubectl get pods

Refresh your Prometheus graphs and scale the deployment up and down to vary metrics

  1. Use refresh on your browser to see the time series metrics change over time.

  2. Use the following command to increase and decrease the number of replicas running in the stress-test deployment.

    kubectl scale deployment.v1.apps/stress-test --replicas=[from 1 to 50 here]

Conclusion

Congratulations, you've completed this hands-on lab!