One elasticsearch/logstash instance with multiple applications with a unique index

Question

We use swisscoms application cloud and are currently evaluating the new Elasticsearch service. We set it up including logstash and kibana.

We now added a user provided service to each of our apps that should use the common elasticsearch/logstash/kibana instance. When we first logged in into kibana we saw there was an index called logstash-, where all the logs of all applications go.

Now what we want is to have a index for each of the apps that writes to our elk instance. Lets say we have e apps (app1, app2, app3). We d like to have three indices (app1-..., app2-... and app3-...). Any ideas on how we can achieve that?

Is that a configuration that has to be done using ENV variables on Cloud foundry or is it something we have to configure within our Java and NodeJS apps

(app1-... , ...)?

Thanks in advance for your help.

Sufiyan Ghori Sufiyan Ghori · Accepted Answer · 2018-06-03T01:09:27

You can use Elasticsearch output plugin for logstash which is the recommended method of storing logs in Elasticsearch. This plugin has a configuration option called index which is used to define the name of the index to write events to. The default index name is logstash-%{+YYYY.MM.dd}

Use it along with if conditional to assign a name of the index for each app based on type, like this,

output {
  if [type] == "apache" {
    elasticsearch {
      index => "apache-website-index"
    }
  } elseif [type] == "nginx" {
      elasticsearch {
        index => "nginx-website-index"
    }
  }
}

Please have a look at this answer as well

Please comment if you have any question.

One elasticsearch/logstash instance with multiple applications with a unique index

1 Answers