Skip to main content

Deploy and Configure an Elasticsearch Cluster

Hands-On Lab

 

Photo of Myles Young

Myles Young

BigData Training Architect II in Content

Length

02:00:00

Difficulty

Advanced

Before we can get hands-on with indexing, searching, and aggregating our data with Elasticsearch, we first need to know how to prepare a system and how to deploy and configure Elasticsearch. In this hands-on lab, you will deploy a 3-node Elasticsearch cluster with a specific set of configuration requirements. Specifically, you will: Install Java Create an elastic user with specific open file limits Set memory map limits via sysctl Deploy Elasticsearch from an archive Specify Elasticsearch cluster and node names Create custom attributes for Elasticsearch nodes Assign Elasticsearch node roles Configure the Elasticsearch Java virtual machine (JVM) heap Bind Elasticsearch to specific network addresses Configure Elasticsearch node discovery Set cluster limitations to avoid split brain Start Elasticsearch as a daemon

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

Deploy and Configure an Elasticsearch Cluster

Introduction

Before we can get hands-on with indexing, searching, and aggregating our data with Elasticsearch, we first need to know how to prepare a system and how to deploy and configure Elasticsearch. In this hands-on lab, you will deploy a 3-node Elasticsearch cluster with a specific set of configuration requirements. Specifically, you will:

  • Install Java
  • Create an elastic user with specific open file limits
  • Set memory map limits via sysctl
  • Deploy Elasticsearch from an archive
  • Specify Elasticsearch cluster and node names
  • Create custom attributes for Elasticsearch nodes
  • Assign Elasticsearch node roles
  • Configure the Elasticsearch Java virtual machine (JVM) heap
  • Bind Elasticsearch to specific network addresses
  • Configure Elasticsearch node discovery
  • Set cluster limitations to avoid split brain
  • Start Elasticsearch as a daemon

Solution

  1. Begin by logging in to the lab servers using the credentials provided on the hands-on lab page:

    ssh cloud_user@PUBLIC_IP_ADDRESS

Prepare the system, create the elastic user, and deploy Elasticsearch

On Node 1:

Install Java

  1. Install the Java 8 developer kit.

    sudo yum install java-1.8.0-openjdk -y

Create the elastic User

  1. Create the elastic user.

    sudo useradd elastic

Configure Open File Limits

  1. Open the limits.conf file as root.

    sudo vim /etc/security/limits.conf
  2. Add the following line near the bottom:

    elastic - nofile 65536
  3. Save and close the file.

Configure Memory Map Limits

  1. Open the sysctl.conf file as root.

    sudo vim /etc/sysctl.conf
  2. Add the following line at the bottom:

    vm.max_map_count=262144
  3. Save and close the file.

  4. Load the new sysctl values.

    sudo sysctl -p

Deploy Elasticsearch

  1. Become the elastic user.

    sudo su - elastic
  2. Download the binaries for Elasticsearch 6.2.4 in the elastic user's home directory.

    curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz
  3. Unpack the archive.

    tar -xzvf elasticsearch-6.2.4.tar.gz
  4. Remove the archive.

    rm elasticsearch-6.2.4.tar.gz
  5. Rename the unpacked directory.

    mv elasticsearch-6.2.4 elasticsearch

Repeat

  1. Repeat these steps for Node 2 and Node 3.

Configure Node 1

Open the Config File

  1. Log in to Node 1 and become the elastic user.

    sudo su - elastic
  2. Open the elasticsearch.yml file.

    vim /home/elastic/elasticsearch/config/elasticsearch.yml

Name the Cluster

  1. Change the following line:

    #cluster.name: my-application

    to

    cluster.name: linux_academy

Name the Node

  1. Change the following line:

    #node.name: node-1

    to

    node.name: master

Configure Node Attributes

  1. Change the following line:

    #node.attr.rack: r1

    to

    node.attr.zone: 1

Configure Node Roles

  1. Add the following lines:

    node.master: true
    node.data: false
    node.ingest: false

Bind to Local and Site-Local Addresses

  1. Change the following:

    #network.host: 192.168.0.1

    to

    network.host: [_local_, _site_]

Configure Host Discovery

  1. Change the following:

    #discovery.zen.ping.unicast.hosts: ["host1", "host2"]

    to

    discovery.zen.ping.unicast.hosts: ["private_ip_of_node_2", "private_ip_of_node_3"]
  2. Remember to replace the tokens private_ip_of_node_2 and private_ip_of_node_3 with your actual IP addresses.

Avoid Split Brain

  1. Change the following:

    #discovery.zen.minimum_master_nodes:

    to

    discovery.zen.minimum_master_nodes: 2

Configure Node 2

Open the Config File

  1. Log in to Node 2 and become the elastic user.

    sudo su - elastic
  2. Open the elasticsearch.yml file.

    vim /home/elastic/elasticsearch/config/elasticsearch.yml

Name the Cluster

  1. Change the following line:

    #cluster.name: my-application

    to

    cluster.name: linux_academy

Name the Node

  1. Change the following line:

    #node.name: node-1

    to

    node.name: data1

Configure Node Attributes

  1. Change the following line:

    #node.attr.rack: r1

    to

    node.attr.zone: 2
  2. Add the following line:

    node.attr.temp: hot

Configure Node Roles

  1. Add the following lines:

    node.master: true
    node.data: true
    node.ingest: true

Bind to Local and Site-Local Addresses

  1. Change the following:

    #network.host: 192.168.0.1

    to

    network.host: [_local_, _site_]

Configure Host Discovery

  1. Change the following:

    #discovery.zen.ping.unicast.hosts: ["host1", "host2"]

    to

    discovery.zen.ping.unicast.hosts: ["private_ip_of_node_1", "private_ip_of_node_3"]
  2. Remember to replace the tokens private_ip_of_node_1 and private_ip_of_node_3 with your actual IP addresses.

Avoid Split Brain

  1. Change the following:

    #discovery.zen.minimum_master_nodes:

    to

    discovery.zen.minimum_master_nodes: 2

Configure Node 3

Open the Config File

  1. Log in to Node 3 and become the elastic user.

    sudo su - elastic
  2. Open the elasticsearch.yml file.

    vim /home/elastic/elasticsearch/config/elasticsearch.yml

Name the Cluster

  1. Change the following line:

    #cluster.name: my-application

    to

    cluster.name: linux_academy

Name the Node

  1. Change the following line:

    #node.name: node-1

    to

    node.name: data2

Configure Node Attributes

  1. Change the following line:

    #node.attr.rack: r1

    to

    node.attr.zone: 3
  2. Add the following line:

    node.attr.temp: warm

Configure Node Roles

  1. Add the following lines:

    node.master: true
    node.data: true
    node.ingest: true

Bind to Local and Site-Local Addresses

  1. Change the following:

    #network.host: 192.168.0.1

    to

    network.host: [_local_, _site_]

Configure Host Discovery

  1. Change the following:

    #discovery.zen.ping.unicast.hosts: ["host1", "host2"]

    to

    discovery.zen.ping.unicast.hosts: ["private_ip_of_node_1", "private_ip_of_node_2"]
  2. Remember to replace the tokens private_ip_of_node_1 and private_ip_of_node_2 with your actual IP addresses.

Avoid Split Brain

  1. Change the following:

    #discovery.zen.minimum_master_nodes:

    to

    discovery.zen.minimum_master_nodes: 2

Configure the master node's heap size

Open the Options File

  1. Log in to Node 1 and become the elastic user.

    sudo su - elastic
  2. Open the jvm.options file.

    vim /home/elastic/elasticsearch/config/jvm.options

Configure Heap Size

  1. Change the following lines:

    -Xms1g
    -Xmx1g

    to

    -Xms512m
    -Xmx512m

Configure the heaps for both data nodes

Open the Options File

  1. Log in to Node 2 and become the elastic user.

    sudo su - elastic
  2. Open the jvm.options file.

    vim /home/elastic/elasticsearch/config/jvm.options

Configure the Heap Size

  1. Change the following lines:

    -Xms1g
    -Xmx1g

    to

    -Xms768m
    -Xmx768m

Repeat

  1. Repeat these steps on Node 3.

Start Elasticsearch as a daemon on each node

Execute the Binary

  1. Log in on Node 1 and become the elastic user.

    sudo su - elastic
  2. Switch to the elasticsearch directory.

    cd /home/elastic/elasticsearch
  3. Start Elasticsearch as a daemon.

    ./bin/elasticsearch -d -p pid

Verify Cluster State and Configuration

  1. Check the startup process.

    less /home/elastic/elasticsearch/logs/linux_academy.log
  2. Check the node configuration.

    curl localhost:9200/_cat/nodes?v

Repeat

  1. Repeat these steps for Node 2 and Node 3.

Conclusion

Congratulations — you've completed this hands-on lab!