Skip to main content

System Log Aggregation and Visualization with Elastic Stack

Hands-On Lab

 

Photo of Myles Young

Myles Young

BigData Training Architect II in Content

Length

00:45:00

Difficulty

Intermediate

For security professionals and system administrators, knowing what is going on with your systems is an important aspect of maintaining their integrity and uptime. With Elastic Stack, we can quickly create a log aggregation pipeline for the system authentication and syslog log files on a Linux/Unix system to collect usage data about our systems and store them in Elasticsearch to be centrally searched and visualized with Kibana.

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

System Log Aggregation and Visualization with Elastic Stack

Introduction

For security professionals and system administrators, knowing what is going on with your systems is an important aspect of maintaining their integrity and uptime. With Elastic Stack, we can quickly create a log aggregation pipeline for the system authentication and syslog log files on a Linux/Unix system to collect usage data about our systems and store them in Elasticsearch to be centrally searched and visualized with Kibana. In this lab, we are going to aggregate some system logs for easy access via dashboards using Elasticsearch and Kibana.

Objectives

  • Install Elasticsearch
  • Install Kibana
  • Install and configure Filebeat
  • Connect to Kibana

Lab Setup

Log in to the cloud server using the public IP address provided on the lab instructions page.

ssh clouduser@<PUBLIC_IP>

You'll see a message that asks, "Are you sure you want to continue connecting (yes/no)?" Type yes. Then, enter your password at the prompt.

Next, elevate permissions to root.

sudo su -

Install Elasticsearch

Before we can install Elasticsearch, we first need to install Java.

yum install java-1.8.0-openjdk -y

This will take a few minutes to install.

Next, import the GPG key for Elastic's RPMs.

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Now let's download the Elasticsearch RPM.

curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.rpm

Next, install the RPM.

rpm --install elasticsearch-6.2.3.rpm

Reload the system.

systemctl daemon-reload

Finally, enable and start the Elasticsearch service.

systemctl enable elasticsearch
systemctl start elasticsearch

We have now successfully installed and enabled the Elasticsearch service.

Install Kibana

Download the Kibana RPM.

curl -O https://artifacts.elastic.co/downloads/kibana/kibana-6.2.3-x86_64.rpm

Then, install Kibana.

rpm --install kibana-6.2.3-x86_64.rpm

Finally, enable and start Kibana.

systemctl enable kibana
systemctl start kibana

Install and Configure Filebeat

Download Filebeat.

curl -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.3-x86_64.rpm

Then, install Filebeat with the default configuration.

rpm --install filebeat-6.2.3-x86_64.rpm

Now we need to edit the system module to convert timestamp timezones to UTC. Open the /etc/filebeat/modules.d/system.yml.disabledfile.

vim /etc/filebeat/modules.d/system.yml.disabled

Next, change the convert_timezone: setting from false to true. Change this setting for both the syslog and auth sections.

Now enable the system Filebeat module.

filebeat modules enable system

Next, install the ingest-geoip filter plugin for the Elasticsearch ingest node.

/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip

At the "Continue with installation? [y/N]" prompt, enter y to confirm.

Then, restart Elasticsearch so that it can use the new ingest-geoip plugin.

systemctl restart elasticsearch

Once Elasticsearch starts up, push the module assets to Elasticsearch and Kibana.

filebeat setup

Finally, enable and start Filebeat.

systemctl enable filebeat
systemctl start filebeat

Connect to Kibana

Open a new terminal and log in to your cloud server's public IP with port forwarding.

ssh cloud_user@<PUBLIC_IP> -L 5601:localhost:5601

Then, enter your password at the prompt.

Next, navigate to localhost:5601 in your web browser.

Click Dashboard in the left sidebar, and type "system" in the search bar at the top of the page. You should see four preconfigured system dashboards in your search results. Click through these dashboards to explore your system log data.

Conclusion

Congratulations, you've successfully completed this lab!