Logstash not creating correct index for Filebeat and Packetbeat

Question

I have set up my Elastic stack like this. I am trying to ship logs and top data through Filebeat and Topbeat with custom index name.

Although, Logstash is not creating any index for the data I am passing with custom index name.

Logstash config:

input{
    beats{
      port => 27080
      congestion_threshold => 1500
    }
    jmx {
      path => "file://Machine01/Users/username/projects/Logstash/logstash/bin/jmx"
      polling_frequency => 15
      type => "jmx"
      nb_thread => 4
 }
}
filter {
    if [type] == "Type1"{
        grok{
          break_on_match => false
          patterns_dir => ["C:\Users\users\projects\Logstash\logstash\bin\patterns"]
          match => { "message" => "%{YEAR:Year}%{MONTHNUM:Month}%{MONTHDAY:Day} %{HOUR:Hour}%{MINUTE:Minute}%{SECOND:Second} %{LogLevel:LogVerbosity} %{MODULE:MODULENAME}%{SPACE}%{MESSAGEID:MESSAGEID} %{SUBMODULE:SUBMODULE} %{MESSAGE:MESSAGE}"}
          add_field => [ "received_at", "%{@timestamp}" ]
          add_field => [ "received_from", "%{host}" ]
          add_tag => ["Groked"]
        }



 if "_grokparsefailure" in [tags] {
              drop { }
    }

   if [type] == "jmx" {
   if ("OperatingSystem.ProcessCpuLoad" in [metric_path] or "OperatingSystem.SystemCpuLoad" in [metric_path]) {
     ruby {
     code => "event['cpuLoad'] = event['metric_value_number'] * 100"
     add_tag => [ "cpuLoad" ]
     } 
   }
 }
  }
}

output {  
    if [type] == "jmx" {
        elasticsearch {  
            hosts => ["http://localhost:9200"]  
            index => "jmx-%{+YYYY.MM.dd}"   
        }
    } else {
        elasticsearch {  
            hosts => ["http://localhost:9200"] 
            manage_template => true
            index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
            document_type => "%{[@metadata][type]}"
        }

         if [type] == "dbtable" {
        elasticsearch {  
            hosts => ["http://localhost:9200"]  
            index => "dbtable-%{+YYYY.MM.dd}"  

        }
    } 
    }
}

Filebeat config:

filebeat:
  prospectors:
    - paths:
        - test.log
      input_type: log
      tail_files: false
      scan_frequency: 3s
      backoff: 20s
      backoff_factor: 1
      document_type: custom
      registry: 
      fields:
        type: custom
  spool_size: 10000
  idle_timeout: 2s
output:
  logstash:
    index: custom
    hosts: ["valid hostname"]
logging:
  to_files: true
  files:
    path: ./
    name: filebeat.log
    rotateeverybytes: 10485760
    level: debug

I am expecting when I set index: custom, it should create an index in Elasticsearch as "custom-YYYY.MM.dd". But it's just creating the index in Elasticsearch as "%{[@metadata][beat]}-%{+YYYY.MM.dd}".

If I comment #index: custom it is creating the index in Elasticsearch as filebeat-YYYY.MM.dd.

Where I am going wrong, why is it not working for a custom index pattern?

A J A J · Accepted Answer · 2016-08-09T18:21:58

Setting the Filebeat output.logstash.index configuration parameter causes it to override the [@metadata][beat] value with the custom index name. Normally the [@metadata][beat] value is the name of the Beat (e.g. filebeat or packetbeat).

Testing your Filebeat config against Logstash shows that the [@metadata][beat] value is indeed set to custom so your Filebeat config is working fine.

There might be an issue with the conditional logic used in your output config. I simplified your output config to make it a bit more concise.

output {
  # Remove this after you finish debugging.
  stdout { codec => rubydebug { metadata => true } }

  if [@metadata][beat] {
    # Use this output only for Beats.
    elasticsearch {
      hosts => ["http://localhost:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }
  } else if [type] == "jmx" or [type] == "dbtable" {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "%{[type]}-%{+YYYY.MM.dd}"
    }
  }
}

When you use a custom index with any of the Beats, you must be sure to install and customize the index template (don't use Logstash's manage_template => true with Beats). Filebeat provides its index template in the filebeat.template.json file distributed in the download. You need to change template line so that it applies to the "custom-*" index instead of "filebeat-*". Then install the template to Elasticsearch using curl -XPUT http://localhost:9200/_template/custom [email protected].

Logstash not creating correct index for Filebeat and Packetbeat

1 Answers