UnknownException in Logstash HTTP output plugin

Question

Some requests in my logstash failed the http output plugin, and the log showing

[2020-10-16T18:44:54,574][ERROR][logstash.outputs.http ] [HTTP Output Failure] Could not fetch URL {:url=>"https://www.example.com/api.php", :method=>:post, :body=>"{"id":"test"}", :headers=>{"Content-Type"=>"text/plain"}, :message=>"Connection reset", :class=>"Manticore::UnknownException", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:37:in block in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:79:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-http-5.2.4/lib/logstash/outputs/http.rb:239:in send_event'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-http-5.2.4/lib/logstash/outputs/http.rb:175:in send_events'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-http-5.2.4/lib/logstash/outputs/http.rb:124:in multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:118:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:239:in block in start_workers'"], :will_retry=>false}

What would be the reason? And how to fix it, i.e. retry again later?

My logstash output configuration is

output {
    if [message] == "..." {
        http {
            format=>"message"
                http_method=>"post"
                url=>"xxx"
                message=>'...'
        }
    }
}

You can implement retry strategy which might be useful if this is a temporary issue. If you are facing this persistently I would suggest looking at network connectivity for the API endpoint to see if there is a misconfiguration that is causing a connection reset, check if there is a firewall or network whitelist at he API end which is blocking the connection or if the API requires mutual auth and is expecting the logstash http output to present client cert. — karan shah
@karanshah but why connection reset will not trigger a retry? — Ryan
Can you update the question with logstash output configuration? — karan shah
The retry_failed option determines whether the logstash output plugin will retry requests. This is set to true by default, so it should keep retrying infinitely. I am surprised why that is not happening in your case. Can see will_retry=>false in error message, will need to see how output plugin determines when to retry and when not to. — karan shah

karan shah karan shah · Accepted Answer · 2020-11-11T10:20:55

The Logstash Http output plugin uses Manticore HTTP Client. It will do automatic retries if the exception falls in below categories. Check this.

  ::Manticore::Timeout,
  ::Manticore::SocketException,
  ::Manticore::ClientProtocolException,
  ::Manticore::ResolutionFailure,
  ::Manticore::SocketTimeout

For your use-case the http client is getting Manticore::UnknownException which is not part of the default list for retries. And I do not see a way to override it using configuration.

I can think of two options.

Debug why the connection reset is happening at network level and resolve the underlying issue. Ex: firewall or network whitelist at the API end, API requires mutual auth
If 1 is not an option and you are sure that retrying is an option and will solve your problem, fork the github repo, add Manticore::UnknownException to the list of retry-able exceptions and use that version of plugin.

UnknownException in Logstash HTTP output plugin

1 Answers