How to set logstash.conf?

Question

I'm using elasticsearch 5.1.1 with logstash for using mysql.

logstash.conf

input {
    jdbc {
            jdbc_driver_library => "lib/mysql-connector-java-5.1.33.jar"
            jdbc_driver_class => "com.mysql.jdbc.Driver"
            jdbc_connection_string => "jdbc:mysql://localhost:3306/test"
            jdbc_user => "test"
            jdbc_password => "test"
            statement => "SELECT * FROM test"
            schedule => "* * * * *"
                type => "test"
       }
    jdbc {
            jdbc_driver_library => "lib/mysql-connector-java-5.1.33.jar"
            jdbc_driver_class => "com.mysql.jdbc.Driver"
            jdbc_connection_string => "jdbc:mysql://localhost:3306/test"
            jdbc_user => "test"
            jdbc_password => "test"
            statement => "SELECT * FROM test2"
            schedule => "* * * * *"
                type => "test2"
        }
}

output {
        elasticsearch {
                hosts => ["localhost:9200"]
                 index => "test"
                document_type => "%{type}"
                document_id => "%{test_id}"
        }
}

$ bin/logstash -f logstash.conf

It worked.

But I want to add this options.

{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "korean": {
            "type": "custom",
            "tokenizer": "seunjeon_default_tokenizer"
          }
        },
        "tokenizer": {
          "seunjeon_default_tokenizer": {
            "type": "seunjeon_tokenizer",
            "index_eojeol": false,
            "user_words": [
              "abc"
            ]
          }
        }
      }
    }
  }
}

How to add this option to conf file?

And If I add more tables which have different column, how to set document_id dynamically?

I would recommended you to create the index and the mapping first and then point log stash to that configuration. Logstash can either reuse your current index settings or try to create one itself. — Yeikel

fgal fgal · Accepted Answer · 2017-01-03T03:06:38

Regarding the first question:

How to add this option to conf file?

I believe the index settings can be set in two different ways

At index creation via index templates
After index creation in real time via index settings updates

In your case, I suppose the first alternative is better. For that you need to create an index template containing the settings you want to apply to your index or indices when they are created. The value of the "template" key has to match the name of the indices you will be creating ("test" in your example).

I don't think you can add index settings to the logstash configuration file directly. You can upload the template using the REST API (PUT _template/template_name endpoint) or let logstash load your template.

If you want to have logstash create the template, you can refer to a template file in the logstash configuration in the elasticsearch block using the template option. The template file referenced in the logstash config file has to be stored in the same host where logstash is running. You can use the template_overwrite option to control whether or not an existing index template will be overwritten.

The logstash configuration will look like:

output {
    elasticsearch {
            hosts => ["localhost:9200"]
            index => "test"
            document_type => "%{type}"
            document_id => "%{test_id}"
            template => "path/to/the/template/file"
    }

}

The template file would contain:

{
  "template": "test",
  "settings" : {
   "index":{
     "analysis":{
       "analyzer":{
         "korean":{
           "type":"custom",
           "tokenizer":"seunjeon_default_tokenizer"
         }
       },
       "tokenizer": {
         "seunjeon_default_tokenizer": {
           "type": "seunjeon_tokenizer",
           "index_eojeol": false,
           "user_words": ["abc"]
         }
       }
     }
   }
  }
}

You may also want to include mappings in your index template

Hope this helps!

(note that I didn't try this specific scenario)

How to set logstash.conf?

1 Answers