Logstash编解码器和字符编码问题

Question

当我想记录错误消息（例如，带有文本“ >>>>>>>>”）时，我从桌面python应用程序（Python 3.6）发送日志到Logstash（7.5.0）ERROR <<<<<<< ”，在logstash日志文件中，我看到以下条目：

[2020-01-22T13：25：02,330] [WARN] [logstash.codecs.line] [main]接收到一个事件，该事件的字符编码与您配置的不同。 {：text =>“ \ u0000 \ u0000 \ u0000MainThreadq \ u001AX \ v \ u0000 \ u0000 \ u0000processNameq \ eX \ v \ u0000 \ u0000 \ u0000MainProcessq \ u001CX \ a \ u0000 \ u0000 \ u0000processq \ u001DM \ u001D \ xEDu。\ u0000 \ u0000 \ u0002 \ u001D} q \ u0000（X \ u0004 \ u0000 \ u0000 \ u0000nameq \ u0001X \ b \ u0000 \ u0000 \ u0000__main__q \ u0002X \ u0003 \ u0000 \ u0000 \ u0000msgq \ u0003X \ u0018 \ u0000 \ u0000 \ u0000 u0000 >>>>>>>>错误<<<<<

[在Kibana中，当我查询收到的消息时，我发现每发送一条日志消息，某些消息（在这种情况下为6）已发送到Logstash（在这种情况下为“ >>>>>> >>错误<<<<<<

{
        "_index" : "logstash-2020.01.23",
        "_type" : "doc",
        "_id" : "lNXhz28BzTlrr0WBIjwA",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost",
          "port" : 50197,
          "message" : """\u0000\u0000\u0000stack_infoq\u0011NX\u0006\u0000\u0000\u0000linenoq\u0012K'X\b\u0000\u0000\u0000funcNameq\u0013X\b\u0000\u0000\u0000<module>q\u0014X\a\u0000\u0000\u0000createdq\u0015GA\u05CA;vϯWX\u0005\u0000\u0000\u0000msecsq\u0016G@n\xA2u\xEC\u0000\u0000\u0000X\u000F\u0000\u0000\u0000relativeCreatedq\u0017G@E\u001DM\xD0\u0000\u0000\u0000X\u0006\u0000\u0000\u0000threadq\u0018L4437804480L""",
          "@version" : "1",
          "@timestamp" : "2020-01-23T00:50:35.362Z"
        }
      },
      {
        "_index" : "logstash-2020.01.23",
        "_type" : "doc",
        "_id" : "k9Xhz28BzTlrr0WBITyc",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost",
          "port" : 50197,
          "message" : """threadNameqX""",
          "@version" : "1",
          "@timestamp" : "2020-01-23T00:50:35.362Z"
        }
      },
      {
        "_index" : "logstash-2020.01.23",
        "_type" : "doc",
        "_id" : "kdXhz28BzTlrr0WBITyc",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost",
          "port" : 50197,
          "message" : """MainThreadqXprocessNameqXMainProcessqXprocessqMC0u.""",
          "@version" : "1",
          "@timestamp" : "2020-01-23T00:50:35.369Z"
        }
      },
      {
        "_index" : "logstash-2020.01.23",
        "_type" : "doc",
        "_id" : "ktXhz28BzTlrr0WBITyc",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost",
          "port" : 50197,
          "message" : "X",
          "@version" : "1",
          "@timestamp" : "2020-01-23T00:50:35.362Z"
        }
      },
      {
        "_index" : "logstash-2020.01.23",
        "_type" : "doc",
        "_id" : "j9Xhz28BzTlrr0WBITyc",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost",
          "port" : 50197,
          "message" : """XfilenameqXtest2.pyqXmoduleq
Xtest2qXexc_infoqNXexc_textqNX""",
          "@version" : "1",
          "@timestamp" : "2020-01-23T00:50:35.345Z"
        }
      },
      {
        "_index" : "logstash-2020.01.23",
        "_type" : "doc",
        "_id" : "kNXhz28BzTlrr0WBITyc",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost",
          "port" : 50197,
          "message" : """}q(XnameqX__main__qXmsgqX>>>>>>>> ERROR <<<<<<<qXargsqNX   levelnameqXERRORqXlevelnoqK2Xpathnameq  X1/Users/e0h014b/PycharmProjects/logstash2/test2.pyq""",
          "@version" : "1",
          "@timestamp" : "2020-01-23T00:50:35.331Z"
        }
      }

我正在使用的logstash配置文件如下：

input {
    tcp {
        port => 5959
        codec => plain {
                charset => "UTF-8"
        }
     }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

如何使Logstash具有正常的日志记录格式？在此应用程序中哪种编解码器和字符编码合适？

谢谢，Elahe

Answer 1

如果您的日志消息仅包含简单的行，则应使用默认的编解码器，即line。

我将始终从默认编解码器开始，进行测试，验证索引数据，然后根据需要对编解码器进行一些微调/更改。

关于所有可用的编解码器，请参考此documentation。>>

希望我能为您提供帮助。