Skip to main content

Setting Up Prometheus and Adding Endpoints

Hands-On Lab

 

Photo of Elle Krout

Elle Krout

Content Team Lead in Content

Length

01:00:00

Difficulty

Beginner

In this hands-on lab, we get our monitoring solution up and running by installing Prometheus, Alertmanager, and Grafana, as well as ensuring metrics from all three services are being fed into Prometheus. This gives us a full monitoring platform so we can start performing both infrastructure and application monitoring and everything in between. As we install Prometheus and Alertmanager, we'll also configure our environment so we can easily and effectively change our configurations without any added work.

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

Setting Up Prometheus and Adding Endpoints

Introduction

In this hands-on lab, we'll get our monitoring solution up and running by installing Prometheus, Alertmanager, and Grafana, as well as ensuring metrics from all three services are being fed into Prometheus. As we install Prometheus and Alertmanager, we'll also configure our environment so we can easily and effectively change our configurations without any added work.

To get started, log in to the monitoring server via SSH using the credentials provided on the lab page.

Prometheus Setup

First, we'll install Prometheus on the monitoring server and configure it so that it contains a systemd service file.

  1. Create a user for Prometheus:

    sudo useradd --no-create-home --shell /bin/false prometheus
  2. Create the needed directories:

    sudo mkdir /etc/prometheus
    sudo mkdir /var/lib/prometheus
  3. Set the ownership of the /var/lib/prometheus directory:

    sudo chown prometheus:prometheus /var/lib/prometheus
  4. Download Prometheus:

    cd /tmp/
    wget https://github.com/prometheus/prometheus/
    releases/download/v2.7.1/prometheus-2.7.1.linux-amd64.tar.gz
  5. Extract the files:

    tar -xvf prometheus-2.7.1.linux-amd64.tar.gz
  6. Move the configuration file and set the owner to the prometheus user:

    cd prometheus-2.7.1.linux-amd64/
    sudo mv console* /etc/prometheus
    sudo mv prometheus.yml /etc/prometheus
    sudo chown -R prometheus:prometheus /etc/prometheus
  7. Move the binaries and set the owner:

    sudo mv prometheus /usr/local/bin/
    sudo mv promtool /usr/local/bin/
    sudo chown prometheus:prometheus /usr/local/bin/prometheus
    sudo chown prometheus:prometheus /usr/local/bin/promtool
  8. Change back to your main directory, and then create the service file using an editor of your choice:

    sudo $EDITOR /etc/systemd/system/prometheus.service

    Add:

    [Unit] Description=Prometheus Wants=network-online.target After=network-online.target

    [Service] User=prometheus Group=prometheus Type=simple ExecStart=/usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries

    [Install] WantedBy=multi-user.target

    Save and exit.

  9. Start Prometheus and make sure it automatically starts on boot:

    sudo systemctl start prometheus
    sudo systemctl enable prometheus
  10. Check Prometheus's status to ensure everything is woring correctly:

    sudo systemctl status prometheus

Alertmanager Setup

Still in the monitoring server, we'll now set up Alertmanager.

  1. Create the alertmanager system user:

    cd
    sudo useradd --no-create-home --shell /bin/false alertmanager
  2. Create needed directories:

    sudo mkdir /etc/alertmanager
  3. Download Alertmanager:

    cd /tmp/
    wget https://github.com/prometheus/alertmanager
    /releases/download/v0.16.1/alertmanager-0.16.1.linux-amd64.tar.gz
  4. Extract the files:

    tar -xvf alertmanager-0.16.1.linux-amd64.tar.gz
    cd alertmanager-0.16.1.linux-amd64/
  5. Move the binaries:

    sudo mv alertmanager /usr/local/bin/
    sudo mv amtool /usr/local/bin/
  6. Set the ownership of the binaries:

    sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager
    sudo chown alertmanager:alertmanager /usr/local/bin/amtool
  7. Move the configuration file into the /etc/alertmanager directory:

    sudo mv alertmanager.yml /etc/alertmanager/
  8. Set the ownership of the /etc/alertmanager directory:

    sudo chown -R alertmanager:alertmanager /etc/alertmanager/
  9. Create the alertmanager.service file for systemd:

    sudo vim /etc/systemd/system/alertmanager.service

    Add:

    [Unit] Description=Alertmanager Wants=network-online.target After=network-online.target

    [Service] User=alertmanager Group=alertmanager Type=simple WorkingDirectory=/etc/alertmanager/ ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml

    [Install] WantedBy=multi-user.target

    Save and exit.

  10. Stop Prometheus, change back to your main directory, and then update the Prometheus configuration file to use Alertmanager:

    sudo systemctl stop prometheus
    sudo $EDITOR /etc/prometheus/prometheus.yml

    Add:

    alerting:
      alertmanagers:
      - static_configs:
        - targets:
          - localhost:9093

    Save and exit.

  11. Reload systemd, and then start the prometheus and alertmanager services:

    sudo systemctl daemon-reload
    sudo systemctl start prometheus
    sudo systemctl start alertmanager
  12. Make sure alertmanager starts on boot:

    sudo systemctl enable alertmanager

Grafana Setup

Now, we're going to configure Grafana.

Open a new terminal window, and log in to the grafana server via SSH using the credentials provided on the lab page.

  1. Install the prerequisite package:

    sudo apt-get install libfontconfig
  2. Download and install Grafana using the .deb package provided on the Grafana download page:

    cd /tmp/
    wget https://dl.grafana.com/oss/release/grafana_5.4.3_amd64.deb
    sudo dpkg -i grafana_5.4.3_amd64.deb
  3. Start Grafana:

    sudo systemctl start grafana-server
  4. Ensure Grafana starts at boot:

    sudo systemctl enable grafana-server
  5. Using the Grafana IP address listed on the lab page, access Grafana's web UI by navigating to <GRAFANA_IP_ADDRESS>:3000.

  6. Log in with the username admin and the password admin. Reset the password when prompted.

Add a Data Source

  1. Click Add data source on the home page.

  2. Select Prometheus.

  3. Using the monitoring server public IP address listed on the lab page, set the URL to http://<MONITORING_IP_ADDRESS>:9090.

  4. Click Save & Test.

Add Endpoints

Back in the monitoring server, we're going to add endpoints to Prometheus.

  1. Open the Prometheus configuration file:

    sudo $EDITOR /etc/prometheus/prometheus.yml
  2. Add the Alertmanager endpoint:

      - job_name: 'alertmanager'
        static_configs:
        - targets: ['localhost:9093']
  3. Add the Grafana endpoint:

      - job_name: 'grafana'
        static_configs:
        - targets: ['<GRAFANA_IP_ADDRESS>:3000']

    Save and exit.

  4. Restart Prometheus:

    sudo systemctl restart prometheus
  5. Head back to the Prometheus web UI (by entering http://<MONITORING_IP_ADDRESS>:9090 into a browser) to check that the endpoints exist on the Targets page (via the Status menu).

Conclusion

Congratulations on completing this lab!