Logstash, grok filter not working for fixed length fields

Question

I am a newbie to logstash, I have an input file with fixed length fields and a config file for log stash configured with the regexp as shown below:

Contents of my log stash configuration file first-pipeline.conf

# The # character at the beginning of a line indicates a comment. Use
# comments to describe your configuration.
input {
    file {
        path => "/Users/priya/sample.log"
        start_position => beginning 
    }
}

filter {
    grok {
        match => ["message", "(?<RECORD_CODE>.{1})(?<SEQUENCE_NUMBER>.{6})(?<REG_NUMBER>.{12})(?<DATA_TYPE>.{3})"]
    }


}
output {

    stdout {}
}

Content of my sample.log file:

50000026311000920150044236080000000026
5000003631100092015005423608000000002
5000004631100092015006615054962

The output i get from log stash is:

priyas-MacBook-Pro:bin priya$ ./logstash -f first-pipeline.conf
Default settings used: Filter workers: 2
Logstash startup completed

Could someone please help me debug the issue and get it to working?

Thanks and regards, Priya

Can someone please help me - i have no idea what is wrong, the same regexp pattern works fine in grok debugger - but it doesn't in the config file. there are no logs - i am not idea how to proceed — Priya Gachinamath

markus markus · Accepted Answer · 2015-11-17T13:31:49

I assume the problem in your case is not the grok expression itself but the way the file input is reading your test file.

The file input remebers where it last read from a logfile and continues reading from that position on subsequent runs (it stores this index in a special file called since_db). start_position => "beginning" only works for the first time you start logstash, on subsequent runs it will start reading from it last ended meaning you won't see any new lines in your console unless you a.) add new lines to your files or b.) manually delete the since_db file (sincedb_path => null is not working under windows, at least when I last tried).

So imho you should first make sure that your grok is working. To do is simply add the stdin input to your input section like this:

input {
    stdin {
    }

    file {
       path => "/Users/priya/sample.log"
       start_position => beginning 
    }
}

Now you can manually create logstash events by simply typing in your console and pressing enter. These events will be parsed as regular logstash events and you will see the resulting json in your console as well (that's done by the stdout output fitler).

After you made sure your grok is working you can check wether or not logstash is picking up the file contents as you would expect it to. Restart logstash and add a new line of data to your /Users/priya/sample.log file (don't forget the newcline/CR at the end of the new line otherwise it wount be picked up). If logstash picks up the new line it should appear in your console output (because you added the stdout output filter).

Logstash, grok filter not working for fixed length fields

1 Answers