Skip to main content


Introduction

In second part of ELK Stack 5.0 Installation and configuration we will configure Kibana – analytics and search dashboard for Elasticsearch and Filebeat – lightweight log data shipper for Elasticsearch (initially based on the Logstash-Forwarder source code).


Looking for part 1 on installing Elasticsearch? Click here.


Getting Started

We will install and configure Kibana and after, proceed to Filebeat installation and configuration on both elkmaster1 and elkslave1 hosts.

Sources / Resources

https://www.elastic.co/

Kibana – Installation

You will need to import the PGP Key, if you did not do this previously with Elasticsearch.

[root@elkmaster1 ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

After that, we need to create a repository configuration file, if you did not do this in the Elasticsearch installation steps

[root@elkmaster1 ~]# vi /etc/yum.repos.d/elasticsearch.repo

Insert the following

[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Now, we are ready to install Kibana

[root@elkmaster1 ~]# yum -y install kibana
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: ftp.colocall.net
* extras: ftp.colocall.net
* updates: ftp.colocall.net
Resolving Dependencies
--> Running transaction check
---> Package kibana.x86_64 0:5.0.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================================================================
Package Arch Version Repository Size
===================================================================================================================================================================================================================
Installing:
kibana x86_64 5.0.0-1 elasticsearch-5.x 39 M

Transaction Summary
===================================================================================================================================================================================================================
Install 1 Package

Total download size: 39 M
Installed size: 140 M
Downloading packages:
kibana-5.0.0-x86_64.rpm | 39 MB 00:00:37
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : kibana-5.0.0-1.x86_64 1/1
Verifying : kibana-5.0.0-1.x86_64 1/1

Installed:
kibana.x86_64 0:5.0.0-1

Complete!
[root@elkmaster1 ~]#

Now we will need to change default configurations for Kibana:

[root@elkmaster1 ~]# vi /etc/kibana/kibana.yml

We will set and change the following settings

server.port: 5601
server.host: "10.0.2.4"
server.name: "elkmaster1"
elasticsearch.url: "http://10.0.2.4:9200"

And ensure that Kibana will be started after server restart

[root@elkmaster1 ~]# systemctl daemon-reload
[root@elkmaster1 ~]# systemctl enable kibana.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service.
[root@elkmaster1 ~]# systemctl start kibana.service
[root@elkmaster1 ~]#

Before you will be able to access Kibana from Host OS, you need to create another port forwarding rule in VirtualBox network settings. Remember that you should use your own IPs for ‘Guest IP’ fields (they may be different than mine depending on how you set them up in our previous guide).

user_5256_5823b2d7c3e59.png

We are adding port forwarding from localhost 127.0.0.1 port 5601 to our elkmaster1 server with IP 10.0.2.4 port 5601 (kibana)

Now check the status of your Kibana service

[root@elkmaster1 ~]# systemctl status kibana
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-10-31 14:12:21 EDT; 7min ago
Main PID: 2821 (node)
CGroup: /system.slice/kibana.service
└─2821 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Oct 31 14:15:02 elkmaster1 kibana[2821]: {"type":"response","@timestamp":"2016-10-31T18:15:02Z","tags":[],"pid":2821,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","he...user-agent":"M
Oct 31 14:15:02 elkmaster1 kibana[2821]: {"
type":"response","
@timestamp":"2016-10-31T18:15:02Z","tags":[],"pid":2821,"method":"get","statusCode":304,"req":{"url":"/plugins/kibana/assets/discover...OW64) AppleWeb
Oct 31 14:15:02 elkmaster1 kibana[2821]: {"type":"response","@timestamp":"2016-10-31T18:15:02Z","tags":[],"pid":2821,"method":"get","statusCode":304,"req":{"url":"/plugins/kibana/assets/visualiz...WOW64) AppleWe
Oct 31 14:15:02 elkmaster1 kibana[2821]: {"
type":"response","
@timestamp":"2016-10-31T18:15:02Z","tags":[],"pid":2821,"method":"get","statusCode":304,"req":{"url":"/plugins/kibana/assets/dashboar...WOW64) AppleWe
Oct 31 14:15:02 elkmaster1 kibana[2821]: {"type":"response","@timestamp":"2016-10-31T18:15:02Z","tags":[],"pid":2821,"method":"get","statusCode":304,"req":{"url":"/plugins/timelion/icon.svg","me...leWebKit/537.3
Oct 31 14:15:02 elkmaster1 kibana[2821]: {"
type":"response","@timestamp":"2016-10-31T18:15:02Z","tags":[],"pid":2821,"method":"get","statusCode":304,"req":{"url":"/plugins/kibana/assets/settings...OW64) AppleWeb
Oct 31 14:15:02 elkmaster1 kibana[2821]: {"type":"response","@timestamp":"2016-10-31T18:15:02Z","tags":[],"pid":2821,"method":"get","statusCode":304,"req":{"url":"/plugins/kibana/assets/wrench.s...64) AppleWebKi
Oct 31 14:15:02 elkmaster1 kibana[2821]: {"
type":"response","
@timestamp":"2016-10-31T18:15:02Z","tags":[],"pid":2821,"method":"get","statusCode":304,"req":{"url":"/plugins/kibana/assets/play-cir...; WOW64) Apple
Oct 31 14:15:02 elkmaster1 kibana[2821]: {"type":"response","@timestamp":"2016-10-31T18:15:02Z","tags":[],"pid":2821,"method":"get","statusCode":304,"req":{"url":"/bundles/src/ui/public/images/k....3; WOW64) App
Oct 31 14:15:02 elkmaster1 kibana[2821]: {"
type":"response","
@timestamp":"2016-10-31T18:15:02Z","tags":[],"pid":2821,"method":"get","statusCode":304,"req":{"url":"/bundles/node_modules/font-awes..."Mozilla/5.0 (
Hint: Some lines were ellipsized, use -l to show in full.
[root@elkmaster1 ~]#

Now we are able to connect to Kibana within our browser from our host OS to the elkmaster1 guest OS

http://127.0.0.1:5601

user_5256_5823b34be6b4f.png_800.jpg

We should also check the status to be sure everything works as expected

http://127.0.0.1/5601/status

user_5256_5823b38b53f50.png_800.jpg

That concludes it for this part of installing Kibana. Next, it is time to install Filebeat.

Filebeat – Installation

We will need to import Elasticsearch PGP key (in case you did not already do this with elasticsearch and kibana installations)

[root@elkmaster1 ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

Create a file with repository information

[root@elkmaster1 ~]# vi /etc/yum.repos.d/elasticsearch.repo

With the following

[elastic-5.x]
name=Elastic repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

After this we will be able to install filebeat

[root@elkmaster1 ~]# yum -y install filebeat
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: ftp.colocall.net
* extras: ftp.colocall.net
* updates: ftp.colocall.net
Resolving Dependencies
--> Running transaction check
---> Package filebeat.x86_64 0:5.0.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================================================================
Package Arch Version Repository Size
===================================================================================================================================================================================================================
Installing:
filebeat x86_64 5.0.0-1 elasticsearch-5.x 8.2 M

Transaction Summary
===================================================================================================================================================================================================================
Install 1 Package

Total download size: 8.2 M
Installed size: 27 M
Downloading packages:
filebeat-5.0.0-x86_64.rpm | 8.2 MB 00:00:05
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : filebeat-5.0.0-1.x86_64 1/1
Verifying : filebeat-5.0.0-1.x86_64 1/1

Installed:
filebeat.x86_64 0:5.0.0-1

Complete!
[root@elkmaster1 ~]#

Now you need to be sure that filebeat will be started after elkmaster1 restarts

[root@elkmaster1 ~]# systemctl enable filebeat
Created symlink from /etc/systemd/system/multi-user.target.wants/filebeat.service to /usr/lib/systemd/system/filebeat.service.
[root@elkmaster1 ~]# systemctl start filebeat
[root@elkmaster1 ~]#

You can also check the status of filebeat with the last log messages

[root@elkmaster1 ~]# systemctl status filebeat
● filebeat.service - filebeat
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-10-31 14:33:53 EDT; 45s ago
Docs: https://www.elastic.co/guide/en/beats/filebeat/current/index.html
Main PID: 2936 (filebeat)
CGroup: /system.slice/filebeat.service
└─2936 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat

Oct 31 14:33:53 elkmaster1 systemd[1]: Started filebeat.
Oct 31 14:33:53 elkmaster1 systemd[1]: Starting filebeat...
[root@elkmaster1 ~]#

We can now go to our Kibana management dashboard and look at Index Patterns

http://127.0.0.1:5601/app/kibana#/management/

And add index pattern *

user_5256_5823b47d653e3.png_800.jpg

We will see that our filebeat installment is already transferring default data to elasticsearch on elkmaster1.

The default data is configured in /etc/filebeat/filebeat.yml

- input_type: log

# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/*.log

user_5256_5823b4d4acc0a.png_800.jpg

Now, after we add a default index pattern, we can go to the Discover menu – and select beat.hostname from Available Fields


user_5256_5823b520d3330.png_800.jpg

And you will see our first results from Elasticsearch.

Filebeat configuration on ELK Slave 1

We will need to import the Elasticsearch PGP key (in case you did not already do this with our previous guides)

[root@elkslave1 ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

Create a file with repository information

[root@elkslave1 ~]# vi /etc/yum.repos.d/elasticsearch.repo

With the following contents

[elastic-5.x]
name=Elastic repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

After this we will be able to install filebeat on our elkslave1 server

[root@elkslave1 ~]# yum -y install filebeat
Loaded plugins: fastestmirror
elastic-5.x | 1.3 kB 00:00:00
elastic-5.x/primary | 4.9 kB 00:00:00
Loading mirror speeds from cached hostfile
* base: ftp.colocall.net
* extras: ftp.colocall.net
* updates: ftp.colocall.net
elastic-5.x 10/10
Resolving Dependencies
--> Running transaction check
---> Package filebeat.x86_64 0:5.0.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================================================================
Package Arch Version Repository Size
===================================================================================================================================================================================================================
Installing:
filebeat x86_64 5.0.0-1 elastic-5.x 8.2 M

Transaction Summary
===================================================================================================================================================================================================================
Install 1 Package

Total download size: 8.2 M
Installed size: 27 M
Downloading packages:
filebeat-5.0.0-x86_64.rpm | 8.2 MB 00:00:17
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : filebeat-5.0.0-1.x86_64 1/1
Verifying : filebeat-5.0.0-1.x86_64 1/1

Installed:
filebeat.x86_64 0:5.0.0-1

Complete!
[root@elkslave1 ~]#

Before any other steps, we will need to point filebeat to our elasticsearch on elkmaster1 server.host

[root@elkslave1 ~]# vi /etc/filebeat/filebeat.yml

And change the IP in the hosts config

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["10.0.2.4:9200"]

After this, we will need to add filebeat to automatically start on reboot

[root@elkslave1 ~]# systemctl enable filebeat
Created symlink from /etc/systemd/system/multi-user.target.wants/filebeat.service to /usr/lib/systemd/system/filebeat.service.
[root@elkslave1 ~]# systemctl start filebeat
[root@elkslave1 ~]#

We will also check the status of our filebeat.service

[root@elkslave1 ~]# systemctl status filebeat
● filebeat.service - filebeat
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-10-31 14:54:20 EDT; 44s ago
Docs: https://www.elastic.co/guide/en/beats/filebeat/current/index.html
Main PID: 2845 (filebeat)
CGroup: /system.slice/filebeat.service
└─2845 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat

Oct 31 14:54:20 elkslave1 systemd[1]: Started filebeat.
Oct 31 14:54:20 elkslave1 systemd[1]: Starting filebeat...
[root@elkslave1 ~]#

Now we can go to Kibana on elkmaster1, select the Discover menu and select beat.hostname from Available Fields. There we will see logs, transferred from elkslave1 with Filebeat to elkmaster1 Elasticsearch.

user_5256_5823b6363bea1.png_800.jpg

Congratulations!

You have now installed, and configured, the ELK Stack 5.0!

Regardless of whether you wanted to set this up for work requirements, or whether this was just a project to learn, I hope this guide helped you achieve your goal!

Go to Part 1 – Installing Elasticsearch.

Dmitry Korzhevin,

Crytek Lead System Administrator,

Head of Crytek CERT (Computer Emergency Response Team)

https://www.linkedin.com/in/dkorzhevin

Comments are disabled for this guide.