Logstash Filter: aggregate - auto save on timeout

Question

I have a Lambda function in AWS which reports logs to an ELK instance. Each invocation of the lambda function generates a unique invocation_id that is sent with every log event, so the events from a single invocation can be identified in ELK. At the end of the operation, I send a "Done" event.

A Lambda function can fail, or timeout, and then the "Done" event is not sent.

I want to use the logstash aggregate filter to identify the failed invocations. Meaning - each invocation_id will be a task_id in the aggregation map, and the "Done" event will be the end_of_task.

And I need to tell it "on timeout (there was no done event received after X time) save the aggregated event with status=failed".

Is that possible with this filter? If so, what is the syntax? It's not clear from the docs..

apparently there's an open feature request for this: github.com/logstash-plugins/logstash-filter-aggregate/issues/14 — Malki
Have you seen the elapsed{} filter, which will mark when the "Done" event hasn't been received. — Alain Collins
@AlainCollins That looks like it could work. I'll try it out and post an update. Thanks :) — Malki

Tomek Habiger Tomek Habiger · Accepted Answer · 2016-09-13T13:59:39

Logstash aggregate filter supports timeout event generation since version 2.3.0. Here is how to achieve what you want using this feature:

if [action] == "BEGIN" {
  aggregate {
    task_id => "%{id}"
    code => "map['bytes'] = 0"
    map_action => "create"
  }
} elseif [action] == "DONE" {
  aggregate {
    task_id => "%{id}"
    code => "event['bytes'] += map['bytes']"
    timeout_code => "event.tag('failed')"
    map_action => "update"
    end_of_task => true
    timeout => 10
    push_map_as_event_on_timeout => true
} else {
    aggregate {
    task_id => "%{id}"
    code => "map['bytes'] += event['bytes']"
    map_action => "update"
    add_tag => [ "drop" ]
}

Logstash Filter: aggregate - auto save on timeout

1 Answers