Logary with Logstash to Elastic Search - simple configuration

2
votes

I've been trying to set up logging for my .NET application using Logary > Logstash > ElasticSearch for nearly the entire day. I've tried every variation of the configuration I can think of and went through docs for hours to no avail.

What is wrong here?

 var logary = LogaryFactory.New("Logary Specs", with => with

                .Target<Logstash.Builder>("ls", conf => conf.Target
                        .Hostname("localhost")
                        .Port(1939)
                        .EventVersion(Logstash.EventVersion.One)
                        .Done())

                .Target<Debugger.Builder>("debugger")
            );

Here's the logstash.conf:

input { 
    stdin {}

    tcp {
         codec => json_lines { charset => "UTF-8" }
         host  => localhost
         port  => 1939
         type  => "app"
     }
}

output {

   stdout {}
   elasticsearch {
      host => localhost
      protocol => "http"
   }
}

I've tried without output.elasticsearch.protocol as with host => "0.0.0.0" like in docs, and every combination in between.

I have ES up on 9200 and Kibana loaded up and I can see logs show up through stdin/stdout but not from my application logs using Logary. It's also verified that the Logary debugger target works.

What's wrong with my pipeline here?

Logstash 1.4.2

ES 1.3.4

2
elasticsearchlogstashkibanalogary
Can you see the logs from stdin in Kibana? – Henrik
Are you using the pre-release version of Logary? It's probably even better at this point as it has much better internal logging and structure. – Henrik
What logs are you getting from Logary? – Henrik
...................from inside Logary? – Henrik

2 Answers

1
votes

You need some components to debug it.

I wrote a sample program for you to use to debug it; this is the simplest possible thing that can work:

open System

open Logary
open Logary.Configuration
open Logary.Targets

[<EntryPoint>]
let main argv = 
  use logary =
    withLogary' "logstash-sample" (
      withTargets [
        Logstash.create (Logstash.LogstashConf.Create "10.0.0.120") "logstash"
        Debugger.create Debugger.empty "debugger"
      ] >>
      withRules [
        Rule.createForTarget "logstash"
        Rule.createForTarget "debugger"
      ] >>
      withInternalTargets Verbose [
        Debugger.create (Debugger.empty) "console"
      ]
    )

  let logger = logary.GetLogger "logstash-sample"
  Logger.debug logger "Hello World"

  Console.ReadKey true |> ignore
  0

Here's its expected Debugger output (not console).

'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\logstash_sample.exe', Symbols loaded.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\Logary.dll', Symbols loaded.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\NodaTime.dll'
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\FSharp.Core\v4.0_4.3.0.0__b03f5f7f11d50a3a\FSharp.Core.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Core\v4.0_4.0.0.0__b77a5c561934e089\System.Core.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Xml\v4.0_4.0.0.0__b77a5c561934e089\System.Xml.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\Newtonsoft.Json.dll'
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\Intelliplan.JsonNet.dll'
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\Intelliplan.JsonNet.NodaTime.dll'
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\FSharp.Actor.dll'
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Configuration\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Configuration.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
D 2014-10-18T10:25:42.4192736+00:00: supervisor status: Running [Logary.Registry.runRegistry]
I 2014-10-18T10:25:42.4192736+00:00: initing logstash target, connecting to 10.0.0.120:1936... [Logary.Targets.Logstash.logstashLoop]
D 2014-10-18T10:25:42.4383230+00:00: scheduler status: Running [Logary.Registry.runRegistry]
D 2014-10-18T10:25:42.4473276+00:00: registry status: Running [Logary.Registry.runRegistry]
D 2014-10-18T10:25:42.4977127+00:00: will poll metrics every 0:00:00:00.5 [Logary.Registry.registry]
D 2014-10-18T10:25:42.5335455+00:00: Hello World [logstash-sample]
I 2014-10-18T10:25:42.5515369+00:00: initing logstash target, connected [Logary.Targets.Logstash.logstashLoop]
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Numerics\v4.0_4.0.0.0__b77a5c561934e089\System.Numerics.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.ComponentModel.DataAnnotations\v4.0_4.0.0.0__31bf3856ad364e35\System.ComponentModel.DataAnnotations.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Runtime.Serialization\v4.0_4.0.0.0__b77a5c561934e089\System.Runtime.Serialization.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Xml.Linq\v4.0_4.0.0.0__b77a5c561934e089\System.Xml.Linq.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
D 2014-10-18T10:25:42.7268749+00:00: polling metrics [Logary.Registry.registry]
D 2014-10-18T10:25:42.8571421+00:00: send-recv stream not open, opening [Logary.Targets.Logstash.logstashLoop]
I 2014-10-18T10:25:43.1096339+00:00: start shutdown [Logary.Configuration.Config.shutdown]
I 2014-10-18T10:25:43.1186538+00:00: flush start [Logary.Registry.registry]
I 2014-10-18T10:25:43.1667422+00:00: flush Ack [Logary.Registry.registry]
I 2014-10-18T10:25:43.1787861+00:00: shutdown metrics polling [Logary.Registry.registry]
I 2014-10-18T10:25:43.1798152+00:00: shutdown schedules [Logary.Registry.registry]
I 2014-10-18T10:25:43.1798152+00:00: shutdown targets [Logary.Registry.registry]
I 2014-10-18T10:25:43.2018104+00:00: shutdown Ack [Logary.Registry.registry]
I 2014-10-18T10:25:43.2018104+00:00: shutting down immediately [Logary.Registry.registry]
I 2014-10-18T10:25:43.2118258+00:00: stop shutdown [Logary.Configuration.Config.shutdown]
The program '[4820] logstash_sample.exe: Managed (v4.0.30319)' has exited with code 0 (0x0).

This code is here:

https://github.com/logary/logary/commit/0dca450677dc1ca691292d379a734c7051743c1c

You can do Install-Package Intelliplan.Logary -Pre and then copy over the Logary.dll-dll file. Ensure you have the same output.

If you can successfully connect to logstash as per the output above, you can verify that it's receiving the events or failing by printing to its log;

tail -f /var/log/logstash/logstash-agent.log
1
votes

There are a couple of things you can try. First double check whether your elasticsearch cluster is actually green/yellow. I wasted a good few hours once because I had specified a higher number of replicas in the elasticsearch index template I configured.

Of course check both the elasticsearch and logstash logs for any errors.

Next, try using

elasticsearch_http { host => "localhost" }

This uses the http protocol rather than the internal protocol and frees you having to have the exact same elastic search version as was bundled with logstash.

Finally, if that doesn't work, I recommend poking around in the logstash directory and modifying the ruby script that implements the plugin. It's nice when it works but the logging is somewhat lacking when it doesn't; which is easily fixed with some log statements.