Logstash skipping to many log lines

Question

I setup logstash on my windows box. I run 7 instances of logstash on it. Each one has a folder with log files for input. I ran them at the same time and directed it to AWS es cluster running 7 instances of r3.xlarge and 3 master nodes (r3.xlarge). All input files combined take around 9GB. After all the logstash instances stopped running, I only had 6 million events in elasticsearch, there should be around 30 million. I went back to my one of my logstash cmd windows where I ran it and looked at the last event. It did not correspond to the last log line in the file where it took it from, it was like ~50th line towards the bottom. Then the second to last log event in the same window corresponded not to the log line right before the one I looked up first, I found it about 30 log lines above it in the log file. So it is apparent my logstash is skipping log lines.

Now I checked my elastic search and it shows all zeros, so nothing got dropped? (I looked at bulk.rejected in particular)

_cat/thread_pool?v

Is this data cumulative or does it get refreshed?

Which brings me to my second question. If logstash itself dropped the log lines for some reason, where and how can i troubleshoot it, I know that none of my logstash instances crashed. All I know is it happily dropped 70% of all my logs and I have no error log or clue to go by as to what happened.

Edit:

My logstash configuration:

(It is like it is ingoring all my logs for Friday, Sat and Sun, and just processing for Monday (3/21))

input {
  file {
    type => "apache_logs"
    path => "D:/logs/apache_logs/all/ssl_access.*"    
    start_position => "beginning"
    sincedb_path => "NUL"
  }

}

filter {

    grok {
        match => ["message","%{IPORHOST:client_ip} (?<username>[-]) (?<password>[-]) \[(?<timestamp>\d{2}[/][a-zA-Z]{3}[/]\d{4}:\d{2}:\d{2}:\d{2}\s-\d{0,4})\] \"%{GREEDYDATA:request}\" %{NOTSPACE:obssocookie} %{NOTSPACE:ps_sso_uid_in} %{NOTSPACE:ps_sso_uid_out} (?<status>[0-9]{3}) (?<bytes>[0-9]{1,}|-) %{NOTSPACE:protocol} %{NOTSPACE:ciphers} \"%{GREEDYDATA:referrer}\" \"%{GREEDYDATA:user_agent}\""]
        match => [ "path", "(?<app_node>webpr[0-9]{2}[a-z]{0,1})" ]
        add_field => { "server_node" => "%{app_node}" }
        break_on_match => false
    }

    mutate {
        gsub => ["obssocookie","^.*=",""]
    }

    mutate {
        gsub => ["ps_sso_uid_in","^.*=",""]
    }

    mutate {
        gsub => ["ps_sso_uid_out","^.*=",""]
    }

    date {
        match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
        remove_field => "timestamp"
    }

     geoip {
        source => "client_ip"
    }

    if [geoip] {
        mutate {
            add_field => {        
              "ip_type" => "public"     
            }
        }
    } else {
        mutate{
            add_field => {        
              "ip_type" => "private"        
            }
        }
    }

}

output {  
  stdout{ codec => rubydebug}
  amazon_es {
    hosts => ["apache-logs-xxxxxxxxxxxxxxxxxxxxxxxxxx.us-west-2.es.amazonaws.com"]
    region => "us-west-2"    
    aws_access_key_id => 'xxxxxxxxxxxxxxxxxx'
    aws_secret_access_key => 'xxxxxxxxxxxxxxxxxxxxx'
    index => "logstash-apache-friday"
  }
}

How can I know how many events logstash dropped specifically, not how many elastic search rejected, because I already checked through the API and bulk.rejected=0

Did you check the logstash and elasticsearch logs? If you have mapping problems, documents can be dropped. — Alain Collins
Where are logstash logs on windows? How do I know I have mapping problems in logstash when I am processing it so fast with millions of events? So this is really frustrating, even after feeding all logs sequentially instead of splitting up into several instances, I have the same number of searchable documents as yesterday with 7 instances of logstash ... and only a third of them, except it took 16 hours to load all the logs this time - alot slower since I did not do it in parallel. I am really at a loss guys, would really appreciate some help :) — alexfvolk

alexfvolk alexfvolk · Accepted Answer · 2016-03-22T22:36:56

Found my culprit. I have to include this in my files input, looks like it skips any files older than 24 hours by default

ignore_older => 0

Kind of surprising, I would expect to add settings when I want to narrow my input, otherwise logstash should process any files, older than 24 hours or not. Really not something that was obvious..

Logstash skipping to many log lines

1 Answers