Logstash codec and character encoding problem

Question

I send logs from a desktop python application (Python 3.6) to Logstash (7.5.0), when I want to log an error message for example with the text ">>>>>>>> ERROR <<<<<<<", in the logstash log file I see the following entry:

[2020-01-22T13:25:02,330][WARN ][logstash.codecs.line ][main] Received an event that has a different character encoding than you configured. {:text=>"\u0000\u0000\u0000MainThreadq\u001AX\v\u0000\u0000\u0000processNameq\eX\v\u0000\u0000\u0000MainProcessq\u001CX\a\u0000\u0000\u0000processq\u001DM\u001D\xEDu.\u0000\u0000\u0002\u001D}q\u0000(X\u0004\u0000\u0000\u0000nameq\u0001X\b\u0000\u0000\u0000__main__q\u0002X\u0003\u0000\u0000\u0000msgq\u0003X\u0018\u0000\u0000\u0000>>>>>>>> ERROR <<<<<<"UTF-8"}

And in Kibana, when I query the received messages, I see that some (in this case, 6) individual messages have been sent to Logstash per each log message that I sent (in this case, ">>>>>>>> ERROR <<<<<<<") as follows:

{
        "_index" : "logstash-2020.01.23",
        "_type" : "doc",
        "_id" : "lNXhz28BzTlrr0WBIjwA",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost",
          "port" : 50197,
          "message" : """\u0000\u0000\u0000stack_infoq\u0011NX\u0006\u0000\u0000\u0000linenoq\u0012K'X\b\u0000\u0000\u0000funcNameq\u0013X\b\u0000\u0000\u0000<module>q\u0014X\a\u0000\u0000\u0000createdq\u0015GA\u05CA;vϯWX\u0005\u0000\u0000\u0000msecsq\u0016G@n\xA2u\xEC\u0000\u0000\u0000X\u000F\u0000\u0000\u0000relativeCreatedq\u0017G@E\u001DM\xD0\u0000\u0000\u0000X\u0006\u0000\u0000\u0000threadq\u0018L4437804480L""",
          "@version" : "1",
          "@timestamp" : "2020-01-23T00:50:35.362Z"
        }
      },
      {
        "_index" : "logstash-2020.01.23",
        "_type" : "doc",
        "_id" : "k9Xhz28BzTlrr0WBITyc",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost",
          "port" : 50197,
          "message" : """threadNameqX""",
          "@version" : "1",
          "@timestamp" : "2020-01-23T00:50:35.362Z"
        }
      },
      {
        "_index" : "logstash-2020.01.23",
        "_type" : "doc",
        "_id" : "kdXhz28BzTlrr0WBITyc",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost",
          "port" : 50197,
          "message" : """MainThreadqXprocessNameqXMainProcessqXprocessqMC0u.""",
          "@version" : "1",
          "@timestamp" : "2020-01-23T00:50:35.369Z"
        }
      },
      {
        "_index" : "logstash-2020.01.23",
        "_type" : "doc",
        "_id" : "ktXhz28BzTlrr0WBITyc",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost",
          "port" : 50197,
          "message" : "X",
          "@version" : "1",
          "@timestamp" : "2020-01-23T00:50:35.362Z"
        }
      },
      {
        "_index" : "logstash-2020.01.23",
        "_type" : "doc",
        "_id" : "j9Xhz28BzTlrr0WBITyc",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost",
          "port" : 50197,
          "message" : """XfilenameqXtest2.pyqXmoduleq
Xtest2qXexc_infoqNXexc_textqNX""",
          "@version" : "1",
          "@timestamp" : "2020-01-23T00:50:35.345Z"
        }
      },
      {
        "_index" : "logstash-2020.01.23",
        "_type" : "doc",
        "_id" : "kNXhz28BzTlrr0WBITyc",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost",
          "port" : 50197,
          "message" : """}q(XnameqX__main__qXmsgqX>>>>>>>> ERROR <<<<<<<qXargsqNX   levelnameqXERRORqXlevelnoqK2Xpathnameq  X1/Users/e0h014b/PycharmProjects/logstash2/test2.pyq""",
          "@version" : "1",
          "@timestamp" : "2020-01-23T00:50:35.331Z"
        }
      }

The logstash config file which I’m using is as the following:

input {
    tcp {
        port => 5959
        codec => plain {
                charset => "UTF-8"
        }
     }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

What should I do to have a normal format of logging in Logstash? Which codec and character encoding are proper in this application?

Thanks,
Elahe

I tried default codec by not mentioning the codec in the config file and it didn't have any change. I didn't see anything related to default codec in apt-get_install_skill, could you please mention where you mentioned that point? — Elahe Sadat Hosseini

apt-get_install_skill apt-get_install_skill · Accepted Answer · 2020-01-24T15:55:25

If your log-messages just contain simple lines, you should go with the default codec, namely line.

I would always start with the default codec, test, verify the indexed data and then do some fine-tuning/changes to the codec if necessary.

Refer to this documentation about all available codecs.

I hope I could help you.

Logstash codec and character encoding problem

1 Answers