ELK Stack 5.0 Installation and configuration. Part 3 - Metricbeat

Metricbeat is a lightweight shipper that you can install on your servers to periodically collect metrics from the operating system and from services running on the server. In this guide, Metricbeat will take metrics and statistics from both elkmaster1 and elkslave1 and ship them directly to Elasticsearch with recommended templates.

Metricbeat is not a replacement for monitoring systems, but instead, it helps you to see hardware and software trends and predict load, resource usage, and other stats which could be used for Business Intelligence.

For example, Metricbeat could help monitor servers by collecting metrics from these systems and services:


Web servers

  • Apache
  • Nginx

Databases

  • MongoDB
  • MySQL
  • PostgreSQL
  • Redis


Other services

  • HAProxy
  • System
  • Zookeeper
Metricbeat can insert collected metrics not only into Elasticsearch as in our guide, or Logstash, but also Redis and Apache Kafka. In this guide we will ship data directly to Elasticsearch, but keep in mind that there are other options.

To get started with your own Metricbeat setup, install and configure these related products:


  • Elasticsearch for storage and indexing the data. (Part 1 of guide)

  • Kibana for the UI. (Part 2 of guide)


Metricbeat 5.0 Installation on elkmaster1

In this step we will install Metricbeat on our CentOS 7 elkmaster1 host. We will use Elasticsearch official repositories for our installation. 


As a first step, you should check that the official Elasticsearch repository is enabled (it should be enabled if you followed our previous guides). If not, please follow the following steps. Otherwise you can skip forward to the Metricbeat installation step.

First we will add the Elasticsearch public GPG signing key:


[root@elkmaster1 ~]# sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

Then, create the repository configuration file:


[root@elkmaster1 ~]# vi /etc/yum.repos.d/elasticsearch.repo

With the following contents:


[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md


Now we can check that Metricbeat is available from the official repo, by checking the package information:


[root@elkmaster1 ~]# yum info metricbeat
Loaded plugins: fastestmirror
Determining fastest mirrors
* base: ftp.colocall.net
* extras: ftp.colocall.net
* updates: ftp.colocall.net
Available Packages
Name : metricbeat
Arch : i686
Version : 5.0.0
Release : 1
Size : 8.0 M
Repo : elasticsearch-5.x
Summary : Sends metrics to Elasticsearch.
URL : https://github.com/elastic/beats/metricbeat
License : ASL 2.0
Description : Sends metrics to Elasticsearch.

Name : metricbeat
Arch : x86_64
Version : 5.0.0
Release : 1
Size : 8.6 M
Repo : elasticsearch-5.x
Summary : Sends metrics to Elasticsearch.
URL : https://github.com/elastic/beats/metricbeat
License : ASL 2.0
Description : Sends metrics to Elasticsearch.

[root@elkmaster1 ~]#


Important note:


We can assume that the official repository provides Metricbeat version 5.0. If you use the 5.x version of Elasticsearch, you should use Kibana 5.x, Filebeat 5.x, Metricbeat 5.x, etc.

After this basic check, we can install Metricbeat


[root@elkmaster1 ~]# yum -y install metricbeat
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: ftp.colocall.net
* extras: ftp.colocall.net
* updates: ftp.colocall.net
Resolving Dependencies
--> Running transaction check
---> Package metricbeat.x86_64 0:5.0.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=====================================================================================================================
Package Arch Version Repository Size
=====================================================================================================================
Installing:
metricbeat x86_64 5.0.0-1 elasticsearch-5.x 8.6 M

Transaction Summary
=====================================================================================================================
Install 1 Package

Total download size: 8.6 M
Installed size: 29 M
Downloading packages:
metricbeat-5.0.0-x86_64.rpm | 8.6 MB 00:00:05
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : metricbeat-5.0.0-1.x86_64 1/1
Verifying : metricbeat-5.0.0-1.x86_64 1/1

Installed:
metricbeat.x86_64 0:5.0.0-1

Complete!
[root@elkmaster1 ~]#


Now, we will configure Metricbeat autostart to make sure it starts on server reboot


[root@elkmaster1 ~]# systemctl enable metricbeat
Created symlink from /etc/systemd/system/multi-user.target.wants/metricbeat.service to /usr/lib/systemd/system/metricbeat.service.
[root@elkmaster1 ~]#


We are almost ready to start the service, but we do have a few small configuration changes to make.

The configuration file is in /etc/metricbeat/metricbeat.yml

Let’s not use complex modules. Instead, we can just include the module ‘system’, which will allow us to monitor system and server stats:


  • CPU stats
  • System Load stats
  • Per CPU core stats
  • IO stats
  • Per filesystem stats
  • File system summary stats
  • Memory stats
  • Network stats

  • Per process stats

Uncomment these configuration lines:
 #- core 
#- diskio
#- fsstat


Then, let’s direct all output to Elasticsearch


name: elkmaster1metricbeat

Important note:

Because we installed Metricbeat on the same server as the Elasticsearch server, we don’t need to change output settings. In case you installed Metricbeat on elkslave1, you should set the elkmaster1 IP address in /etc/metricbeat/metricbeat.yml


output.elasticsearch:
# Array of hosts to connect to.
Hosts: ["10.0.2.4:9200"]


Save the configuration and restart Metricbeat:


[root@elkmaster1 ~]# systemctl restart metricbeat

Check that Metricbeat is up and running:


[root@elkmaster1 ~]# systemctl status metricbeat
● metricbeat.service - metricbeat
Loaded: loaded (/usr/lib/systemd/system/metricbeat.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2016-11-10 10:49:15 EST; 22s ago
Docs: https://www.elastic.co/guide/en/beats/metricbeat/current/index.html
Main PID: 3122 (metricbeat)
CGroup: /system.slice/metricbeat.service
└─3122 /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml -path.home /usr/share/met...

Nov 10 10:49:15 elkmaster1 systemd[1]: Started metricbeat.
Nov 10 10:49:15 elkmaster1 systemd[1]: Starting metricbeat...
[root@elkmaster1 ~]#


You can see that there are no errors listed after the string ‘Nov 10 10:49:15 elkmaster1 systemd[1]: Starting metricbeat...’

It is also useful to check the Metricbeat logfile:


[root@elkmaster1 ~]# head -n15 /var/log/metricbeat/metricbeat
2016-11-10T10:49:15-05:00 INFO Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2016-11-10T10:49:15-05:00 INFO Setup Beat: metricbeat; Version: 5.0.0
2016-11-10T10:49:15-05:00 INFO Loading template enabled. Reading template file: /etc/metricbeat/metricbeat.template.json
2016-11-10T10:49:15-05:00 INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /etc/metricbeat/metricbeat.template-es2x.json
2016-11-10T10:49:15-05:00 INFO Elasticsearch url: http://localhost:9200
2016-11-10T10:49:15-05:00 INFO Activated elasticsearch as output plugin.
2016-11-10T10:49:15-05:00 INFO Publisher name: elkmaster1metricbeat
2016-11-10T10:49:15-05:00 INFO Flush Interval set to: 1s
2016-11-10T10:49:15-05:00 INFO Max Bulk Size set to: 50
2016-11-10T10:49:15-05:00 INFO Register [ModuleFactory:[system], MetricSetFactory:[apache/status, haproxy/info, haproxy/stat, mongodb/status, mysql/status, nginx/stubstatus, postgresql/activity, postgresql/bgwriter, postgresql/database, redis/info, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/process, zookeeper/mntr]]
2016-11-10T10:49:15-05:00 INFO Metrics logging every 30s
2016-11-10T10:49:15-05:00 INFO metricbeat start running.
2016-11-10T10:49:15-05:00 INFO Connected to Elasticsearch version 5.0.0
2016-11-10T10:49:15-05:00 INFO Trying to load template for client: http://localhost:9200
2016-11-10T10:49:15-05:00 INFO Elasticsearch template with name 'metricbeat' loaded
[root@elkmaster1 ~]#


Pay attention to log lines highlighted in bold. In Elasticsearch 5.0, index templates are used to define settings and mappings that determine how incoming data from fields should be analyzed by Elasticsearch.

The recommended index template file for Metricbeat 5.0 is installed by the Metricbeat package. As we use default configurations for template loading in the metricbeat.yml config file, Metricbeat loads the template automatically after successfully connecting to Elasticsearch. If the template already exists, it’s not overwritten unless you configure Metricbeat to do so.

Here’s an example of a Metricbeat template configuration:


output.elasticsearch:
# Array of hosts to connect to.
hosts: ["10.0.2.4:9200"]

template.name: "metricbeat"
template.path: "metricbeat.template.json"
template.overwrite: false


metricbeat.template.json – is default and recommended template file.

Importing Dashboards to Elasticsearch

Dashboards are a predefined set of Kibana templates, which help systematize data display in correct order for target values. The Metricbeat package comes with a script which allows you to download sample dashboards, visualizations, and searches from Elasticsearch.


[root@elkmaster1 ~]# cd /usr/share/metricbeat/scripts/
[root@elkmaster1 scripts]# pwd
/usr/share/metricbeat/scripts
[root@elkmaster1 scripts]# ll
total 11848
-rwxr-xr-x. 1 root root 12112988 Oct 26 00:35 import_dashboards
-rwxr-xr-x. 1 root root 14397 Oct 26 00:35 migrate_beat_config_1_x_to_5_0.py
[root@elkmaster1 scripts]#
[root@elkmaster1 scripts]# ./import_dashboards


By default, the script assumes that you are running Elasticsearch on localhost:9200. Use the -es option to specify a different location. For example:


[root@elkmaster1 scripts]# ./import_dashboards -es http://10.0.2.4:9200

To verify that your server’s statistics are present in Elasticsearch on elkmaster1 we can run this command:


[root@elkmaster1 ~]# curl -XGET 'http://localhost:9200/metricbeat-*/_search?pretty' | head -n15
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 13985 100 13985 0 0 1485k 0 --:--:-- --:--:-- --:--:-- 1517k
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 5594,
"max_score" : 1.0,
"hits" : [
{
"_index" : "metricbeat-2016.11.10",
"_type" : "metricsets",
[root@elkmaster1 ~]#


We used ‘localhost:9200’ because the Elasticsearch instance is running on this host (elkmaster1). In case you need to check a remote host, you need to change localhost to the server IP in your curl command:


# curl -XGET 'http://10.0.2.4:9200/metricbeat-*/_search?pretty' | head

Install Metricbeat on elkslave1

Now it is time to install Metricbeat on the elkslave1 server.

Please follow the same steps for the official repository configuration as we did previously with the elkmaster1 host.

Install the Metricbeat package:


[root@elkslave1 ~]# yum -y install metricbeat

Enable Metricbeat on server start:


[root@elkslave1 ~]# systemctl enable metricbeat
Created symlink from /etc/systemd/system/multi-user.target.wants/metricbeat.service to /usr/lib/systemd/system/metricbeat.service.
[root@elkslave1 ~]#


Let’s make a small configuration change before restarting Metricbeat.

The configuration file is located at /etc/metricbeat/metricbeat.yml


[root@elkslave1 ~]# vi /etc/metricbeat/metricbeat.yml

We again won’t use any complex modules, so just include the system module, which will allow us to monitor system and server stats:


  • CPU stats
  • System Load stats
  • Per CPU core stats
  • IO stats
  • Per filesystem stats
  • File system summary stats
  • Memory stats
  • Network stats

  • Per process stats


Uncomment these configuration lines in that same configuration file:


 #- core 
#- diskio
#- fsstat


Now we need to direct all output to Elasticsearch:


name: elkslave1metricbeat

Important:


You will need to point Metricbeat from elkslave1 to Elasticsearch on elkmaster1:


#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["10.0.2.4:9200"]


Save the configuration and restart Metricbeat:


[root@elkslave1 ~]# systemctl restart metricbeat

Check that Metricbeat is up and running:


[root@elkslave1 ~]# systemctl status metricbeat
● metricbeat.service - metricbeat
Loaded: loaded (/usr/lib/systemd/system/metricbeat.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2016-11-10 11:26:05 EST; 7s ago
Docs: https://www.elastic.co/guide/en/beats/metricbeat/current/index.html
Main PID: 3043 (metricbeat)
CGroup: /system.slice/metricbeat.service
└─3043 /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml -path.home /usr/share/metricbeat -path.config /etc/metricbeat -path.data /var/lib/metricbeat -path.logs /var/log/metric...

Nov 10 11:26:05 elkslave1 systemd[1]: Started metricbeat.
Nov 10 11:26:05 elkslave1 systemd[1]: Starting metricbeat...
[root@elkslave1 ~]#


We can check the log file to be sure that metrics from metricbeat are transferring to Elasticsearch without errors:


[root@elkslave1 ~]# tail -n20 /var/log/metricbeat/metricbeat
2016-11-10T11:26:05-05:00 INFO Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2016-11-10T11:26:05-05:00 INFO Setup Beat: metricbeat; Version: 5.0.0
2016-11-10T11:26:05-05:00 INFO Loading template enabled. Reading template file: /etc/metricbeat/metricbeat.template.json
2016-11-10T11:26:05-05:00 INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /etc/metricbeat/metricbeat.template-es2x.json
2016-11-10T11:26:05-05:00 INFO Elasticsearch url: http://10.0.2.4:9200
2016-11-10T11:26:05-05:00 INFO Activated elasticsearch as output plugin.
2016-11-10T11:26:05-05:00 INFO Publisher name: elkslave1
2016-11-10T11:26:05-05:00 INFO Flush Interval set to: 1s
2016-11-10T11:26:05-05:00 INFO Max Bulk Size set to: 50
2016-11-10T11:26:05-05:00 INFO Register [ModuleFactory:[system], MetricSetFactory:[apache/status, haproxy/info, haproxy/stat, mongodb/status, mysql/status, nginx/stubstatus, postgresql/activity, postgresql/bgwriter, postgresql/database, redis/info, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/process, zookeeper/mntr]]
2016-11-10T11:26:05-05:00 INFO Metrics logging every 30s
2016-11-10T11:26:05-05:00 INFO metricbeat start running.
2016-11-10T11:26:05-05:00 INFO Connected to Elasticsearch version 5.0.0
2016-11-10T11:26:05-05:00 INFO Trying to load template for client: http://10.0.2.4:9200
2016-11-10T11:26:05-05:00 INFO Template already exists and will not be overwritten.
[root@elkslave1 ~]#


Opening dashboards in Kibana

After importing the dashboards, launch the Kibana web interface by pointing your Host OS browser to port 5601. In our guide, as we previously configured port forwarding from our Host OS to our Guest servers, so we will use this URI: http://127.0.0.1:5601.

If Kibana shows a “No default index pattern” warning, you must select or create an index pattern to continue. To resolve the issue, select the predefined metricbeat-* index pattern and set it as the default.


user_5256_58488c2d198cf.png_800.jpg


Metrics from elkmaster1 host

user_5256_58488c7ac9f4b.png_800.jpg


Metricbeat visualizations 1

user_5256_58488ce1379d7.png_800.jpg


Metricbeat visualizations 2

user_5256_58488d196ffb9.png_800.jpg

user_5256_58488d60a6177.png_800.jpg


Process stats visualization

user_5256_58488dd666601.png_800.jpg


System overview by host

user_5256_58488e1ef1d84.png_800.jpg

Metricbeat general system overview


user_5256_58488ea366739.png_800.jpg

CPU Usage over time

user_5256_58488f5ee1357.png_800.jpg


Memory usage

user_5256_58488faa64b35.png_800.jpg


Top processes by memory usage

user_5256_584890133f005.png_800.jpg


Traffic stats (input and output traffic)

user_5256_58489089dba75.png_800.jpg

Disc space usage

user_5256_584890d21130f.png_800.jpg

Filesystem per host usage

user_5256_584891250ddbf.png_800.jpg

That concludes it for this part of a multipart guide on using the ELK stack. We saw how to install Metricbeat, and how to start collecting information and sending it to Elasticsearch and Kibana for visualization. Thank you for reading!


Dmitry Korzhevin,

Crytek Lead System Administrator,

Head of Crytek CERT (Computer Emergency Response Team)

https://www.linkedin.com/in/dkorzhevin





  • post-author-pic
    Johnny J
    12-07-2016

     @dkorzhevin: Nice guide on ELK!

  • post-author-pic
    Shahan K
    12-08-2016

    GreatJob!

  • post-author-pic
    John B
    12-08-2016

    This is cool!

  • post-author-pic
    Rahul J
    01-23-2018

    Awesome.

  • post-author-pic
    Lev A
    01-24-2018

    Awesome! Are you planning to do ELK course?!:)

  • post-author-pic
    exnewbie
    02-19-2018

    metricbeat looks interesting. have you tried to compare it with telegraph/graphite ingrated with influxdb? 

  • post-author-pic
    Venkatesen K
    06-18-2018

    Have you tried Fluentd? I am looking for options to import JSON files into Elasticsearch. I was suggested fluentd. If you have experience with it, please let me know.

  • post-author-pic
    Shri G
    07-05-2018

    Awesome doc, I am gonna try building this in my lab.

Looking For Team Training?

Learn More