ELK stack not persisting some documents

Question

I have logstash pushing events to elasticsearch, some work and some don't.

This is an example of a document which does appear in elasticsearch:

{
       "Application" => "API",
       "Environment" => "myenv",
        "LoggerName" => "MyLoggerName",
        "@timestamp" => "2015-02-01T17:18:30.454Z",
          "LogLevel" => "Warn",
      "DeploymentId" => "ebb9e128b8d44994b7bbbf27b6893b03",
    "RoleInstanceId" => "MyRoleInstance",
           "Message" => "The message.",
          "@version" => "1"
}

However the following one (and many more similar) do not:

    {
           "Application" => "API",
           "Environment" => "myenv",
            "LoggerName" => "Common.Services.RequestLogger",
            "@timestamp" => "2015-02-01T17:19:46.265Z",
              "LogLevel" => "Info",
          "DeploymentId" => "0a56017c4ad14cfe818afdbc52dabe76",
        "RoleInstanceId" => "Instance",
                  "Data" => {
                       "Elapsed" => "PT0.0119377S",
                    "RequestDto" => {
                "__type" => "Structure.Definition.LoginRequest, Structure.Definition",
                 "Email" => "[email protected]"
            },
                   "ResponseDto" => {
                               "__type" => "ServiceStack.HttpResult, ServiceStack",
                              "Headers" => {},
                "AllowsPartialResponse" => false,
                              "Options" => {},
                               "Status" => 200,
                           "StatusCode" => "OK",
                             "Response" => {
                            "__type" => "Structure.Definition.LoginResponse, Structure.Definition",
                            "UserId" => "xxxx",
                    "OrganisationId" => "xxxx",
                             "Token" => "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
                            "TimeMs" => 0
                },
                       "ResponseFilter" => {


                  "__type" => "ServiceStack.Host.ContentTypes, Service
Stack",
                "ContentTypeFormats" => {
                           "csv" => "text/csv",
                      "markdown" => "text/markdown",
                         "plain" => "text/plain",
                    "x-protobuf" => "application/x-protobuf"
                }
            },
                    "PaddingLength" => 0,
                     "IsPartialRequest" => false
            },
                 "OperationName" => "LoginRequest",
                          "Verb" => "POST",
                   "AbsoluteUri" => "https://xxxx/auth/login",
                        "RawUrl" => "/auth/login",
                       "IsLocal" => false,
            "IsSecureConnection" => true,
                      "RemoteIp" => "000.000.000.000",
               "UserHostAddress" => "000.000.000.000",
                     "UserAgent" => "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit
    /537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36",
                    "StatusCode" => 200,
             "StatusDescription" => "OK",
                          "Type" => "Request"
        },
              "@version" => "1"
    }

These have been copied from the stdout { codec => rubydebug {} } so I know logstash understands the document, however the second type never appears in elasticsearch.

I have looked to see if this is related to file size, but I have found nothing that would limit this.

Here is the logstash config:

input {
  redis {
    codec => json {}
    data_type => "list"
    host => "xxxx.redis.cache.windows.net"
    key => "logstash"
    password => "xxxxxxx"
  }
  redis {
    codec => json {}
    data_type => "list"
    host => "xxxx.redis.cache.windows.net"
    key => "logstash"
    password => "xxxxxx"
  }
}
output {
  elasticsearch {
    host => "127.0.0.1"
    port => "9201"
    protocol => http
  }
  stdout { codec => rubydebug{} }
}

Is there something I am missing from the elasticsearch config? The logstash node connects to elasticsearch over HTTP via SSH port map.

Can you add the mapping of this index type taken from Elasticsearch. — Vineeth Mohan
Here @VineethMohan is the gist of the mapping type, it looks like it has all the fields that have ever been posted: gist.github.com/jamescarter-le/17dfa86410b3cc0d2687 — James

Christian Dahlqvist Christian Dahlqvist · Accepted Answer · 2015-02-02T21:48:05

In the document that failed you appear to have fields named "__type". Fields starting with an underscore are restricted, and will cause problems. I would therefore suggest you rename the "__type" field to "type" and try again. This and other field name limitations is described here: http://grokbase.com/t/gg/elasticsearch/144s6e3877/illegal-characters-in-elasticsearch-field-names

ELK stack not persisting some documents

1 Answers