Skip to main content

Upgrading Elasticsearch

Hands-On Lab

 

Photo of Myles Young

Myles Young

BigData Training Architect II in Content

Length

01:00:00

Difficulty

Intermediate

Elastic Stack products have a relatively high velocity of releases. In order to maintain a cluster and stay up to date with the latest bug fixes and feature enhancements, sysadmins need to know how to update an Elasticsearch cluster. In this activity, we will update a single Elasticsearch node, but we are going to pretend that it is apart of a large production cluster with dozens of nodes. Therefore, the way we upgrade your Elasticsearch node will need to adhere to the live-rolling update procedure that ensures zero downtime of the cluster, and 100% data availability throughout the upgrade process.

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

Upgrading Elasticsearch

Introduction

Elastic Stack products have a relatively high velocity of releases. In order to maintain a cluster and stay up to date with the latest bug fixes and feature enhancements, sysadmins need to know how to update an Elasticsearch cluster. In this activity, we will update a single Elasticsearch node, but we are going to pretend that it is apart of a large production cluster with dozens of nodes. Therefore, the way we upgrade your Elasticsearch node will need to adhere to the live-rolling update procedure that ensures zero downtime of the cluster, and 100% data availability throughout the upgrade process.

The Scenario

We manage a 25-node Elasticsearch cluster running version 6.2.x, and it needs to be updated to version 6.3. The cluster's Service Level Agreement (SLA) is 99.999% uptime. Therefore, we will need to perform a live version update of our cluster with zero downtime. Since the upgrade procedure is the same for each node, we will perform the same upgrade steps on each node, one at a time. We'll follow these steps:

  • Ensure we have a green cluster state
  • Reduce shard allocation to new_primaries
  • Stop the Elasticsearch instance
  • Install version 6.3.0
  • Start the Elasticsearch instance
  • Increase shard allocation to all

Logging In

Use the credentials on the hands-on lab overview page to log into the provided server. We're going to be doing some admin type tasks, so run sudo su - right off to get root privileges.

Check the Cluster State and Reduce Shard Allocation to "new_primaries"

First, we want to check the cluster state. If the cluster is not green, then we should wait until shard allocation is finished before proceeding. Use the CAT API to check the cluster health like so:

curl localhost:9200/_cat/health?v

We should see something like this:

epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1529337171 11:52:51  elasticsearch green           1         1      0   0    0    0        0             0                  -                100.0%

Now we can reduce shard allocation to new_primaries with this:

curl -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d '
{
  "persistent": {
    "cluster.routing.allocation.enable": "new_primaries"
  }
}'

Our output should look like this:

{
  "acknowledged" : true,
  "persistent" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "enable" : "new_primaries"
        }
      }
    }
  },
  "transient" : { }
}

Now let's run this:

curl localhost:9200/_cluster/settings?pretty

We should get some output that looks like this:

{
   "persistent" : {
     "cluster" : {
       "routing" : {
         "allocation" : {
           "enable" : "new_primaries"
         }
       }
     }
   },
   "transient" : { }
}

Stop Elasticsearch

Stop the elasticsearch service with:

systemctl stop elasticsearch

Check that the service is shut down with:

systemctl status elasticsearch

Upgrade Elasticsearch to 6.3.0 via YUM

Upgrade Elasticsearch to version 6.3 with:

yum update elasticsearch-6.3.0 -y

Perform a daemon-reload to pick up the changes to elasticsearch.service with:

systemctl daemon-reload

Start Elasticsearch

Start Elasticsearch with:

systemctl start elasticsearch

Check that the service has started with:

systemctl status elasticsearch

Increase Shard Allocation to all

Increase shard allocation to all with:

curl -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d '
{
  "persistent": {
    "cluster.routing.allocation.enable": "all"
  }
}'

We should see output like this:

{
  "acknowledged" : true,
  "persistent" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "enable" : "all"
        }
      }
    }
  },
  "transient" : { }
}

Conclusion

We're finished. It will take a while for these changes to take effect on the rest of the replicas and primaries if we've got a really big node. Congratulations!