Logstash can't create an index in Elasticsearch

1
votes

I have logstash and elasticsearch on one box. Everything seems to be working (no obvious errors), but no information is getting to elasticsearch. I have noticed that logstash seems to be unable to create an index in ES:

http://192.168.250.49:9200/_status?pretty=true

{
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "failed" : 0
  },
  "indices" : { }
}

http://192.168.250.49:9200/_aliases?pretty

{}

logstash console (bin/logstash -f configfile --debug):

{:timestamp=>"2015-10-20T15:25:35.776000-0400", :message=>"Reading config file", :file=>"logstash/agent.rb", :level=>:debug, :line=>"309", :method=>"local_config"}
{:timestamp=>"2015-10-20T15:25:35.894000-0400", :message=>"Compiled pipeline code:\n        @inputs = []\n        @filters = []\n        @outputs = []\n        @periodic_flushers = []\n        @shutdown_flushers = []\n\n          @input_file_1 = plugin(\"input\", \"file\", LogStash::Util.hash_merge_many({ \"type\" => (\"syslog\") }, { \"path\" => [(\"/var/log/python_apps/*.log\"), (\"/var/log/python_apps/*.log.*\")] }))\n\n          @inputs << @input_file_1\n\n          @output_elasticsearch_2 = plugin(\"output\", \"elasticsearch\", LogStash::Util.hash_merge_many({ \"host\" => (\"192.168.250.49\") }, { \"protocol\" => (\"http\") }))\n\n          @outputs << @output_elasticsearch_2\n\n          @output_stdout_3 = plugin(\"output\", \"stdout\", LogStash::Util.hash_merge_many({ \"codec\" => (\"rubydebug\") }))\n\n          @outputs << @output_stdout_3\n\n  def filter_func(event)\n    events = [event]\n    @logger.debug? && @logger.debug(\"filter received\", :event => event.to_hash)\n    events\n  end\n  def output_func(event)\n    @logger.debug? && @logger.debug(\"output received\", :event => event.to_hash)\n    @output_elasticsearch_2.handle(event)\n    @output_stdout_3.handle(event)\n    \n  end", :level=>:debug, :file=>"logstash/pipeline.rb", :line=>"29", :method=>"initialize"}
{:timestamp=>"2015-10-20T15:25:35.907000-0400", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"input", :name=>"file", :path=>"logstash/inputs/file", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}
{:timestamp=>"2015-10-20T15:25:35.925000-0400", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"codec", :name=>"plain", :path=>"logstash/codecs/plain", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}
{:timestamp=>"2015-10-20T15:25:35.949000-0400", :message=>"config LogStash::Codecs::Plain/@charset = \"UTF-8\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:35.963000-0400", :message=>"config LogStash::Inputs::File/@type = \"syslog\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:35.975000-0400", :message=>"config LogStash::Inputs::File/@path = [\"/var/log/python_apps/*.log\", \"/var/log/python_apps/*.log.*\"]", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:35.988000-0400", :message=>"config LogStash::Inputs::File/@debug = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:35.991000-0400", :message=>"config LogStash::Inputs::File/@codec = <LogStash::Codecs::Plain charset=>\"UTF-8\">", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.004000-0400", :message=>"config LogStash::Inputs::File/@add_field = {}", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.006000-0400", :message=>"config LogStash::Inputs::File/@stat_interval = 1", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.007000-0400", :message=>"config LogStash::Inputs::File/@discover_interval = 15", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.020000-0400", :message=>"config LogStash::Inputs::File/@sincedb_write_interval = 15", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.022000-0400", :message=>"config LogStash::Inputs::File/@start_position = \"end\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.023000-0400", :message=>"config LogStash::Inputs::File/@delimiter = \"\\n\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.037000-0400", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"output", :name=>"elasticsearch", :path=>"logstash/outputs/elasticsearch", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}
{:timestamp=>"2015-10-20T15:25:36.422000-0400", :message=>"config LogStash::Codecs::Plain/@charset = \"UTF-8\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.453000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@host = [\"192.168.250.49\"]", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.466000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@protocol = \"http\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.480000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@type = \"\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.492000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@tags = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.515000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@exclude_tags = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.528000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@codec = <LogStash::Codecs::Plain charset=>\"UTF-8\">", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.539000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@workers = 1", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.551000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@index = \"logstash-%{+YYYY.MM.dd}\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.563000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@manage_template = true", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.586000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@template_name = \"logstash\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.588000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@template_overwrite = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.611000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@embedded = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.623000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@embedded_http_port = \"9200-9300\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.635000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@max_inflight_requests = 50", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.644000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@flush_size = 5000", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.651000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@idle_flush_time = 1", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.655000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@action = \"index\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.662000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@path = \"/\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.667000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@ssl = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.672000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@ssl_certificate_verification = true", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.673000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@sniffing = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.675000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@max_retries = 3", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.676000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@retry_max_items = 5000", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.689000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@retry_max_interval = 5", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.691000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@doc_as_upsert = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.692000-0400", :message=>"config LogStash::Outputs::ElasticSearch/@upsert = \"\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.694000-0400", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"output", :name=>"stdout", :path=>"logstash/outputs/stdout", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}
{:timestamp=>"2015-10-20T15:25:36.754000-0400", :message=>"Plugin not defined in namespace, checking for plugin file", :type=>"codec", :name=>"rubydebug", :path=>"logstash/codecs/rubydebug", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}
{:timestamp=>"2015-10-20T15:25:36.782000-0400", :message=>"config LogStash::Codecs::RubyDebug/@metadata = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.797000-0400", :message=>"config LogStash::Outputs::Stdout/@codec = <LogStash::Codecs::RubyDebug metadata=>false>", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.809000-0400", :message=>"config LogStash::Outputs::Stdout/@type = \"\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.831000-0400", :message=>"config LogStash::Outputs::Stdout/@tags = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.843000-0400", :message=>"config LogStash::Outputs::Stdout/@exclude_tags = []", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:36.860000-0400", :message=>"config LogStash::Outputs::Stdout/@workers = 1", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"111", :method=>"config_init"}
{:timestamp=>"2015-10-20T15:25:37.114000-0400", :message=>"Registering file input", :path=>["/var/log/python_apps/*.log", "/var/log/python_apps/*.log.*"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"77", :method=>"register"}
{:timestamp=>"2015-10-20T15:25:37.127000-0400", :message=>"No sincedb_path set, generating one based on the file path", :sincedb_path=>"/root/.sincedb_8f309eb34476af59efaabf28f6aac73a", :path=>["/var/log/python_apps/*.log", "/var/log/python_apps/*.log.*"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"120", :method=>"register"}
{:timestamp=>"2015-10-20T15:25:37.141000-0400", :message=>"Normalizing http path", :path=>"/", :normalized=>"/", :level=>:debug, :file=>"logstash/outputs/elasticsearch.rb", :line=>"353", :method=>"register"}
{:timestamp=>"2015-10-20T15:25:37.148000-0400", :message=>"Create client to elasticsearch server on 192.168.250.49:", :level=>:info, :file=>"logstash/outputs/elasticsearch.rb", :line=>"440", :method=>"register"}
{:timestamp=>"2015-10-20T15:25:37.150000-0400", :message=>"_sincedb_open: reading from /root/.sincedb_8f309eb34476af59efaabf28f6aac73a", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"219", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.193000-0400", :message=>"_sincedb_open: setting [\"10131811\", 0, 64768] to 776284", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.208000-0400", :message=>"_sincedb_open: setting [\"10131826\", 0, 64768] to 1445796", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.210000-0400", :message=>"_sincedb_open: setting [\"10131834\", 0, 64768] to 8271711", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.221000-0400", :message=>"_sincedb_open: setting [\"10131838\", 0, 64768] to 6486678", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.234000-0400", :message=>"_sincedb_open: setting [\"10131855\", 0, 64768] to 3707417", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.246000-0400", :message=>"_sincedb_open: setting [\"10131860\", 0, 64768] to 82806", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.256000-0400", :message=>"_sincedb_open: setting [\"10131861\", 0, 64768] to 4591241", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.269000-0400", :message=>"_sincedb_open: setting [\"10131865\", 0, 64768] to 0", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.334000-0400", :message=>"_sincedb_open: setting [\"10131866\", 0, 64768] to 0", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.340000-0400", :message=>"_sincedb_open: setting [\"10131869\", 0, 64768] to 0", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.401000-0400", :message=>"_sincedb_open: setting [\"9870144\", 0, 64768] to 0", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.432000-0400", :message=>"_sincedb_open: setting [\"10068312\", 0, 64768] to 10485833", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.445000-0400", :message=>"_sincedb_open: setting [\"10068313\", 0, 64768] to 10486125", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.454000-0400", :message=>"_sincedb_open: setting [\"10068314\", 0, 64768] to 10485833", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.466000-0400", :message=>"_sincedb_open: setting [\"10077159\", 0, 64768] to 10486142", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.468000-0400", :message=>"_sincedb_open: setting [\"10131808\", 0, 64768] to 10485990", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.480000-0400", :message=>"_sincedb_open: setting [\"10131809\", 0, 64768] to 10485800", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.512000-0400", :message=>"_sincedb_open: setting [\"10131810\", 0, 64768] to 10485883", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.526000-0400", :message=>"_sincedb_open: setting [\"10131812\", 0, 64768] to 10485802", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.539000-0400", :message=>"_sincedb_open: setting [\"10131813\", 0, 64768] to 10485915", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.555000-0400", :message=>"_sincedb_open: setting [\"10131814\", 0, 64768] to 10486067", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.577000-0400", :message=>"_sincedb_open: setting [\"10131815\", 0, 64768] to 10485843", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.588000-0400", :message=>"_sincedb_open: setting [\"10131816\", 0, 64768] to 10485762", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.602000-0400", :message=>"_sincedb_open: setting [\"10131817\", 0, 64768] to 10485831", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.663000-0400", :message=>"_sincedb_open: setting [\"10131818\", 0, 64768] to 10485945", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.666000-0400", :message=>"_sincedb_open: setting [\"10131819\", 0, 64768] to 10485872", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.673000-0400", :message=>"_sincedb_open: setting [\"10131820\", 0, 64768] to 10485787", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.699000-0400", :message=>"_sincedb_open: setting [\"10131821\", 0, 64768] to 10485819", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.714000-0400", :message=>"_sincedb_open: setting [\"10131822\", 0, 64768] to 10486265", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.730000-0400", :message=>"_sincedb_open: setting [\"10131823\", 0, 64768] to 10485792", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.741000-0400", :message=>"_sincedb_open: setting [\"10131824\", 0, 64768] to 10485982", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.744000-0400", :message=>"_sincedb_open: setting [\"10131825\", 0, 64768] to 10485807", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.782000-0400", :message=>"_sincedb_open: setting [\"10131827\", 0, 64768] to 10486007", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.802000-0400", :message=>"_sincedb_open: setting [\"10131828\", 0, 64768] to 10485770", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.815000-0400", :message=>"_sincedb_open: setting [\"10131829\", 0, 64768] to 10492229", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.860000-0400", :message=>"_sincedb_open: setting [\"10131830\", 0, 64768] to 10485793", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.875000-0400", :message=>"_sincedb_open: setting [\"10131831\", 0, 64768] to 10485828", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.887000-0400", :message=>"_sincedb_open: setting [\"10131832\", 0, 64768] to 10485773", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.899000-0400", :message=>"_sincedb_open: setting [\"10131833\", 0, 64768] to 10485845", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.911000-0400", :message=>"_sincedb_open: setting [\"10131835\", 0, 64768] to 10485965", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.923000-0400", :message=>"_sincedb_open: setting [\"10131836\", 0, 64768] to 10485881", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.935000-0400", :message=>"_sincedb_open: setting [\"10131837\", 0, 64768] to 10485809", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.860000-0400", :message=>"Automatic template management enabled", :manage_template=>"true", :level=>:info, :file=>"logstash/outputs/elasticsearch.rb", :line=>"448", :method=>"register"}
{:timestamp=>"2015-10-20T15:25:37.953000-0400", :message=>"_sincedb_open: setting [\"10131839\", 0, 64768] to 10485806", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.987000-0400", :message=>"_sincedb_open: setting [\"10131840\", 0, 64768] to 10485819", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:37.994000-0400", :message=>"_sincedb_open: setting [\"10131841\", 0, 64768] to 10485820", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:38.019000-0400", :message=>"_sincedb_open: setting [\"10131842\", 0, 64768] to 10486363", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}
{:timestamp=>"2015-10-20T15:25:38.021000-0400", :message=>"_sincedb_open: setting [\"10131843\", 0, 64768] to 10485815", :level=>:debug, :file=>"filewatch/tail.rb", :line=>"223", :method=>"_sincedb_open"}

(If you need more console data, please tell me. @ char limit)

logstash conf:

input {
  file {
    type => "syslog"
    path => ["/var/log/python_apps/*.log", "/var/log/python_apps/*.log.*"]
  }
}

output {
  elasticsearch {
              host => "192.138.250.49"
              protocol => "http"
              cluster => "elastic"
             }
  stdout {
       codec => rubydebug
      }
}

logstash err is empty

logstash log is empty

logstash.stdout: Sending logstash logs to /var/log/logstash/logstash.log.

ES.log (only file in /var/log/elasticsearch with data):

[2015-10-20 15:37:35,031][INFO ][node                     ] [Harald Jaekelsson] version[1.4.4], pid[2573], build[c88f77f/2015-02-19T13:05:36Z]
[2015-10-20 15:37:35,032][INFO ][node                     ] [Harald Jaekelsson] initializing ...
[2015-10-20 15:37:35,034][INFO ][plugins                  ] [Harald Jaekelsson] loaded [], sites []
[2015-10-20 15:37:37,484][INFO ][node                     ] [Harald Jaekelsson] initialized
[2015-10-20 15:37:37,484][INFO ][node                     ] [Harald Jaekelsson] starting ...
[2015-10-20 15:37:37,591][INFO ][transport                ] [Harald Jaekelsson] bound_address {inet[/192.168.250.49:9300]}, publish_address {inet[/192.168.250.49:9300]}
[2015-10-20 15:37:37,607][INFO ][discovery                ] [Harald Jaekelsson] elasticeagle/3_vjAMdtR4C7oiTgr03wng
[2015-10-20 15:37:41,370][INFO ][cluster.service          ] [Harald Jaekelsson] new_master [Harald Jaekelsson][3_vjAMdtR4C7oiTgr03wng][localhost.localdomain][inet[/192.168.250.49:9300]], reason: zen-disco-join (elected_as_master)
[2015-10-20 15:37:41,512][INFO ][gateway                  ] [Harald Jaekelsson] recovered [0] indices into cluster_state
[2015-10-20 15:37:41,528][INFO ][http                     ] [Harald Jaekelsson] bound_address {inet[/192.168.250.49:9200]}, publish_address {inet[/192.168.250.49:9200]}
[2015-10-20 15:37:41,528][INFO ][node                     ] [Harald Jaekelsson] started

UPDATE: Things work if I start logstash manually. People have inferred that because this works, logstash isn't acting under user/group logstash during init. But, etc/init.d/logstash reveals that it's working as logstash.

People have also told me .sincedb could be the cause of some of this. But, I can't find it on my system, even though it's referenced by logstash and it claims such is in the /root folder. 

{:timestamp=>"2015-10-21T08:57:56.694000-0400", :message=>"No sincedb_path set, generating one based on the file path", :sincedb_path=>"/root/.sincedb_8f309eb34476af59efaabf28f6aac73a", :path=>["/var/log/python_apps/*.log", "/var/log/python_apps/*.log.*"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"120", :method=>"register"}
1
elasticsearchlogstash
Is new content being written to your log files? – Alain Collins
I don't know which one you're referring to, but there hasn't been any new info written to either logstash.log or [clustername].log for about 30min – Josh Usre
The log files you're trying to ship with logstash, e.g. ["/var/log/python_apps/*.log", "/var/log/python_apps/*.log.*"] – Alain Collins
Negative. It's a static folder for testing consisting of various .log files from python projects. – Josh Usre
logstash remembers where it left off (see all the sincedb references). With static files, it probably thinks that it has processed them already. On your input, remove the existing sincedb file and try again. – Alain Collins

1 Answers

0
votes

Discussion in the link may help : https://discuss.elastic.co/t/logstash-doesnt-create-index-in-es-no-errors/32617

It, says

"The reason it worked when you ran Logstash by hand was that sincedb state isn't shared between users so the files were treated as brand new and were processed from the begininning. Again, if you want to reprocess log files you have to clear the sincedb entries. If you do that it'll work even when you run Logstash as a service." You can find sincedb files in logstash log, provided that you are running in debug mode.