Conditional creating fields depends on filtering results in logstash influxdb output

Question

I'm using the logstash for collecting sar metrics from the server and store its in influxdb. Metrics from different sources (CPU, Memory, Network) should be inserted to the different series in influxdb. Of course amount and names of fields in those series depends type of metric source.

This is my config file: https://github.com/evgygor/test/blob/master/logstash.conf

For each [type] of metrics I should configure separate influxdb output. In this example, I configured two types of metrics, but I'm planning to use it for SAR metrics, JMX metrics, csv from Jmeter metrics, that mean - I need configure the appropriate output for each of them (tens).

Questions:

How can I elaborate desired configuration? I there any option to use conditions inside plugin. Example:

if [type]=="system.cpu" {

          data_points => {
                        "time" => "%{time}"
                        "user"    => "%{user}"
          }
  }
 else {
          data_points => {
                         "time" => "%{time}"
                         "kbtotalmemory" => "%{kbtotalmemory}"
                         "kbmemfree" => "%{kbmemfree}"
                         "kbmemused" => "%{kbmemused}"
          }
}

Is there any flag to define to influxdb plugin to use by default fields names/data types from input? Is there any flag/ability to define default datatype? Is there any ability to set field name "time" reserved with datatype integer? Thank a lot.

Eugene Gorelik Eugene Gorelik · Accepted Answer · 2015-02-23T11:06:38

I cooked some nice solution. This fork permits to create fields on the fly, accrding to fields names and datatypes that arrives to that output plugin.

I added 2 configuration paramters:

This settings revokes the needs to use data_points and coerce_values configuration # to create appropriate insert to influxedb. Should be used with fields_to_skip configuration # This setting sets data points (column) names as field name from arrived to plugin event, # value for data points config :use_event_fields_for_data_points, :validate => :boolean, :default => true

The array with keys to delete from future processing. # By the default event that arrived to the output plugin contains keys "@version", "@timestamp" # and can contains another fields like, for example, "command" that added by input plugin EXEC. # Of course we doesn't needs those fields to be processed and inserted to influxdb when configuration # use_event_fields_for_data_points is true. # We doesn't deletes the keys from event, we creates new Hash from event and after that, we deletes unwanted # keys.

config :fields_to_skip, :validate => :array, :default => []

This is my example config file: I'm retrieving different number of fields with differnt names from CPU, memory, disks, but I doesn't need defferent configuration per data type as in master branch. I'm creating relevant fields names and datatypes on filter stage and just skips the unwanted fields in outputv plugin.

https://github.com/evgygor/logstash-output-influxdb

Conditional creating fields depends on filtering results in logstash influxdb output

1 Answers