How can I check if my log's filepath contains a certain string in logstash config using regex?

Question

I'm using filebeat to send logs to logstash, based on their filename - these logs are sent to specific indexes in elasticsearch. Filebeat works well, logstash receives the log files, but I can't seem to get the regex in my logstash config to check if the filenames contain a certain string working.

Here's how my logstash config looks

input {
    beats {
      port => 5044
    }
}
filter {
    csv {
        separator => ","
        columns => ["Order ID","Status","Remarks"]
    }
}
output {
    if [source] =~ "/path/to/my/logs/log-file-1*.csv" {
        stdout {}
    }
    else if [source] =~ "/path/to/my/logs/log-file-2*.csv" {
        stdout {}
    }
    else if [source] =~ "/path/to/my/logs/log-file-3*.csv" {
        stdout {}
    }
    else if [source] =~ "/path/to/my/logs/log-file-4*.csv" {
        stdout {}
    }
}

I've removed the part that sends to elasticsearch. I expect the output of data being printed to the stdout, but there isn't any.

Am I referencing [source] the correct way?

Answer 1

Did you try to add the $ at the end of the regular expressions?. Most likely, it's about a regex issue.

For example check this:

if [source] =~ "/path/to/my/logs/log-file-1*.csv$"

Another solution is:

if [source] =~ "/path/to/my/logs/log-file-1(.*).csv$"

Answer 2

Thanks for your help guys, but I managed to solve the problem by using the [fields] setting/option inside my filebeat.yml and then accessed [fields] inside my logstash config as part of a condition.

Here's how my filebeat.yml looks:

- type: log

  enabled: true

  paths:
    - /path/to/my/logs/log-file-1*

  fields: {target_index: log-index-1}

- type: log

  enabled: true

  paths:
    - /path/to/my/logs/log-file-2*

  fields: {target_index: log-index-2}

...and then my logstash config file:

output {
    if [fields][target_index] == "log-index-1" {
        elasticsearch {
            # post to specific index
        }
        stdout {}
    }
    else if [fields][target_index] == "log-index-2" {
        elasticsearch {
            # post to specific index
        }
        stdout {}
    }

If there's any more suggestions for me to optimize this please let me know.

Answer 3

You need to write your regex between /, not " and escape the / in your pattern like this: \/. In your cases, your regex should look like this:

if [source] =~ /\/path\/to\/my\/logs\/log-file-1*.csv/ {
    stdout {}
}
else if [source] =~ /\/path\/to\/my\/logs\/log-file-2*.csv/ {
    stdout {}
}
else if [source] =~ /\/path\/to\/my\/logs\/log-file-3*.csv/ {
    stdout {}
}
else if [source] =~ /\/path\/to\/my\/logs\/log-file-4*.csv/ {
    stdout {}
}