Define Elasticsearch Index Templates with Dynamic Mapping

Hands-On Lab

 

Photo of Myles Young

Myles Young

BigData Training Architect II in Content

Length

01:30:00

Difficulty

Advanced

One of the most user-friendly features of Elasticsearch is dynamic mapping. Dynamic mapping is Elasticsearch's mechanism for detecting fields and mapping them to an appropriate data type. Using index templates, we can lay out the structure of a series of indexes to adhere to specific requirements or override and control dynamic mapping behavior. With dynamic mapping, we can skip the process of explicitly defining every possible field and its data type and instead tell Elasticsearch how to detect the desired data types for us as they are discovered during indexing. As a result, dynamic mapping allows us to get up and running with our data very quickly. In this hands-on lab, you will complete the following tasks: Create an index template Define explicit field mappings * Define dynamic field mappings

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

Define Elasticsearch Index Templates with Dynamic Mapping

Introduction

One of the most user-friendly features of Elasticsearch is dynamic mapping. Dynamic mapping is Elasticsearch's mechanism for detecting fields and mapping them to an appropriate data type. Using index templates, we can lay out the structure of a series of indexes to adhere to specific requirements or override and control dynamic mapping behavior. With dynamic mapping, we can skip the process of explicitly defining every possible field and its data type and instead tell Elasticsearch how to detect the desired data types for us as they are discovered during indexing. As a result, dynamic mapping allows us to get up and running with our data very quickly. In this hands-on lab, you will complete the following tasks:

  • Create an index template
  • Define explicit field mappings
  • Define dynamic field mappings

OPTIONAL: Establish a remote tunnel to Kibana on the master node

NOTE: This task only needs to be performed if you're using the Kibana UI to interact with Elasticsearch.

Set Up a Remote Tunnel

  1. Open a new terminal window and use SSH to log in to the master node as cloud_user with port forwarding.

    ssh cloud_user@your_public_ip -L 5601:localhost:5601

Open the Kibana Console Tool

  1. In your local web browser, go to http://localhost:5601.
  2. In Kibana, navigate to Dev Tools in the side navigation bar.
  3. Select the Console tool (it should be the default tool that loads).

Create the customers index template

Option 1: Kibana Console Tool

Create the Index Template

  1. To create the customers index template, use the Console to send the following request to Elasticsearch:

    PUT _template/customers
    {
      "aliases": {
        "customers": {}
      },
      "index_patterns": ["customers-*"],
      "mappings": {
        "doc": {
          "dynamic_templates": [
            {
              "long_to_integer": {
                "match_mapping_type": "long",
                "mapping": {
                  "type": "integer"
                }
              }
            }
          ],
          "properties": {
            "year_to_date": {
              "type": "double"
            }
          }
        }
      }, 
      "settings": {
        "number_of_shards": 2,
        "number_of_replicas": 1
      }
    }

Option 2: Command line curl

Create the Index Template

  1. To create the customers index template, execute the following from the command line of one of the nodes:

    curl -XPUT "http://localhost:9200/_template/customers" -H 'Content-Type: application/json' -d'
    {
      "aliases": {
        "customers": {}
      },
      "index_patterns": ["customers-*"],
      "mappings": {
        "doc": {
          "dynamic_templates": [
            {
              "long_to_integer": {
                "match_mapping_type": "long",
                "mapping": {
                  "type": "integer"
                }
              }
            }
          ],
          "properties": {
            "year_to_date": {
              "type": "double"
            }
          }
        }
      }, 
      "settings": {
        "number_of_shards": 2,
        "number_of_replicas": 1
      }
    }'

Create the partners index template

Option 1: Kibana Console Tool

Create the Index Template

  1. To create the partners index template, use the Console to send the following request to Elasticsearch:

    PUT _template/partners
    {
      "aliases": {
        "partners": {}
      },
      "index_patterns": ["partners-*"],
      "mappings": {
        "doc": {
          "dynamic_templates": [
            {
              "string_to_keyword": {
                "match_mapping_type": "string",
                "mapping": {
                  "type": "keyword"
                }
              }
            }
          ],
          "properties": {
            "address": {
              "type": "text"
            }
          }
        }
      }, 
      "settings": {
        "number_of_shards": 2,
        "number_of_replicas": 1
      }
    }

Option 2: Command line curl

Create the Index Template

  1. To create the partners index template, execute the following from the command line of one of the nodes:

    curl -XPUT "http://localhost:9200/_template/partners" -H 'Content-Type: application/json' -d'
    {
      "aliases": {
        "partners": {}
      },
      "index_patterns": ["partners-*"],
      "mappings": {
        "doc": {
          "dynamic_templates": [
            {
              "string_to_keyword": {
                "match_mapping_type": "string",
                "mapping": {
                  "type": "keyword"
                }
              }
            }
          ],
          "properties": {
            "address": {
              "type": "text"
            }
          }
        }
      }, 
      "settings": {
        "number_of_shards": 2,
        "number_of_replicas": 1
      }
    }'

Create the leads index template

Option 1: Kibana Console Tool

Create the Index Template

  1. To create the leads index template, use the Console to send the following request to Elasticsearch:

    PUT _template/leads
    {
      "aliases": {
        "leads": {}
      },
      "index_patterns": ["leads-*"],
      "mappings": {
        "doc": {
          "dynamic_templates": [
            {
              "string_to_keyword": {
                "match_mapping_type": "string",
                "match": "lead_*",
                "unmatch": "*_text",
                "mapping": {
                  "type": "keyword"
                }
              }
            }
          ],
          "properties": {
            "address": {
              "type": "text"
            },
            "estimate": {
              "type": "double"
            }
          }
        }
      }, 
      "settings": {
        "number_of_shards": 2,
        "number_of_replicas": 1
      }
    }

Option 2: Command line curl

Create the Index Template

  1. To create the leads index template, execute the following from the command line of one of the nodes:

    curl -XPUT "http://localhost:9200/_template/leads" -H 'Content-Type: application/json' -d'
    {
      "aliases": {
        "leads": {}
      },
      "index_patterns": ["leads-*"],
      "mappings": {
        "doc": {
          "dynamic_templates": [
            {
              "string_to_keyword": {
                "match_mapping_type": "string",
                "match": "lead_*",
                "unmatch": "*_text",
                "mapping": {
                  "type": "keyword"
                }
              }
            }
          ],
          "properties": {
            "address": {
              "type": "text"
            },
            "estimate": {
              "type": "double"
            }
          }
        }
      }, 
      "settings": {
        "number_of_shards": 2,
        "number_of_replicas": 1
      }
    }'

Conclusion

Congratulations — you've completed this hands-on lab!