How to reference environment variables in logstash configuration file?

Question

Is it possible to reference environment variables in logstash configuration?

In my case, i want to make my elasticsearch address configurable that i have set in the environment.

Answer 1

With logstash 2.3, you can set environment variable references into Logstash plugins configuration using ${var} or $var. https://www.elastic.co/guide/en/logstash/current/environment-variables.html

Before logstash 2.3, you can use the "environment" filter plugin which is community maintained.

Documentation at : https://www.elastic.co/guide/en/logstash/current/plugins-filters-environment.html#plugins-filters-environment-add_field_from_env

How to install this plugin:

$LOGSTASH_HOME/bin/plugin install logstash-filter-environment

Source code at : https://github.com/logstash-plugins/logstash-filter-environment

The main part is:

# encoding: utf-8
require "logstash/filters/base"
require "logstash/namespace"


# Set fields from environment variables
class LogStash::Filters::Environment < LogStash::Filters::Base
  config_name "environment"

  # Specify a hash of fields to the environment variable
  # A hash of matches of `field => environment` variable
  config :add_field_from_env, :validate => :hash, :default => {}

  public
  def register
    # Nothing
  end # def register

  public
  def filter(event)
    return unless filter?(event)
    @add_field_from_env.each do |field, env|
      event[field] = ENV[env]
    end
    filter_matched(event)
  end # def filter
end # class LogStash::Filters::Environment

Answer 2

I can hardly believe, that these are the only solutions left: Hacking logstash or using some kind of templating system to re-write the config.

Actually, I do not want to touch or tweak the config for different deployment-scenarios: All I want is to pass in some parameters to connect logstash to the outside world (e.g. where elasticsearch is located, usernames/credentials to connect to other systems). I googled for an hour now an all I could find were these awkwardly complicated solutions for this very simple and common problem.

I sincerely hope, that someone comes up with a better idea like

%{ENV[ELASTICSEARCH_HOST]}}

Answer 3

That's not directly supported, no.

However, if you're running a version later than 1.4.0, it would be pretty trivial to edit elasticsearch.rb to add this feature. Around line 183:

client_settings["network.host"] = @bind_host if @bind_host

You could tweak it to read an environment variable:

if ENV["ESHOST"].nil? then
    client_settings["network.host"] = ENV["ESHOST"]
else
    client_settings["network.host"] = @bind_host if @bind_host
end

If you prefer, you can run Logstash with the -e command-line option to pass config via STDIN. You could cat in some file with special tokens that you've replaced with your environment variable(s).

Answer 4

The logstash configuration as of this writing is just a configuration file, but it's not a programing language. Thus, it has a few reasonable "limitations", for example, it cannot reference environment variables, cannot pass parameters it, hard to reuse other configuration file. Those limitations would make the logstash configuration file hard to maintain when the configuration file grows or you want to adjust its behavior on the fly.

My approach is to use template engine to generate the logstash configuration file. I used Jinja2 in Python.

For example, the elastic search output could be templated as

output {
  elasticsearch {
    index => {{ es_index_name }}
    host => {{ es_hostname }}
  }
}

Then I wrote a simple python code using Jinja2 to generate the logstash configuration file, and the value of es_index_name and es_hostname could be passed via the Python script argument. See here for Jiaja2 tutorial: http://kagerato.net/articles/software/libraries/jinja-quickstart.html

In this way, a big logstash configuration could be splitted into reusable pieces and its behavior can be adjusted on the fly.

Answer 5

As explained in logstash-issues

Connections are established at plugin registration time (during initialization, as they almost certainly should be), but field interpolation (like %{escluster}) is an event-processing-time operation. As such, host isn't really eligible for this behavior.

So unless input or output plugin will natively supports %{foo} syntax, doing any environment variable evaluation at the stage of event filtering is too late for the input and output plugin to take advantage of it.

Answer 6

.conf file support environment variables. you just have to export the environment variable:

export EXAMPLE_VAR=123

and use it in the configuration file this way:

${EXAMPLE_VAR}