How to configure logstash to create an elasticsearch index?

3
votes

elasticsearch version: elasticsearch-2.2.0.rpm logstash version: logstash-2.2.2-1.noarch.rpm

I start elasticsearch, then logstash with /etc/logstash/conf.d/logstash.conf having a basic stdin/stdout, but no elasticsearch index is created. If I add the following to my logstash output configuration I get an index that indicates a yellow status:

action => "create" index => "main_index"

The reason it's yellow, and not usable, is because the number of shards is 5 and replicas is 3. If I run:

curl -XPUT 'http://localhost:9200/index2/' -d ' index : number_of_shards : 1 number_of_replicas : 0 '

"index2" is green and usable. How do I tell either logstash and/or elasticsearch that I want my index to have 1 shard with 0 replicas without issuing a curl command?

Thanks.

3
indexingelasticsearchlogstash
Would index templates work for you? stackoverflow.com/questions/24553718/… – Filip
Also maybe this read would probably provide some better context to your predicament... github.com/logstash-plugins/logstash-output-elasticsearch/… – Filip

3 Answers

4
votes

You have three solutions:

  1. You override the default index template that Logstash uses and you provide your own with the proper settings, i.e. with "number_of_replicas": 0
  2. You create an index template in ES with the proper index settings
  3. In elasticsearch.yml, you change the setting called index.number_of_replicasand set it to 0 (and then restart your ES)
2
votes

In order to complete Val´s answer, here is an update for ES version 5.x:

Solution 3 would not work as index level configurations are disabled from config files: "Since elasticsearch 5.x index level settings can NOT be set on the nodes configuration like the elasticsearch.yaml"

Solution 1 does work, and below is an example:

  • Download and edit the base template for ES 5.x from here.

  • Change template name to match your index name pattern, and add the index settings you wanted to update in the first place:

    {
  "template" : "syslog*",
  "version" : 50001,
  "settings" : {
   "index.refresh_interval" : "5s",
   "index.number_of_replicas" : 0,
   "index.number_of_shards" : 1
  },
...
}

  • Update the logstash configuration so the output uses the created template:

    output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "syslog%{+YYYY.MM.dd}"
    template => "path_to_your_template.json"
    template_name => "syslog*"
    template_overwrite => true
  }
}

  • Restart the service

0
votes

For instance, You could create a small index—just one primary shard—and no replica shards with the following request:

   PUT /my_index
       {
        "settings": {
            "number_of_shards" :   1,
            "number_of_replicas" : 0
       }
       }