如何使用grok和logstsh解析“日志2017-12-18 00：00：00,098 DEBUG测试 - 在测试类中”

Question

我有以下格式的日志。 2017-12-18 00：00：00,098 DEBUG测试 - 在测试类中我想用logstash和grok filter存储日志。我得到以下格式的输出。

 {
          "@version" => "1",
              "host" => "ip-172-30-0-112",
              "path" => "/home/ubuntu/logstash-6.0.0/apl.log",
        "@timestamp" => 2017-12-19T11:32:03.692Z,
           "message" => "2017-12-18 00:00:00,098 DEBUG SdkTLSSocketFactory - Starting handshake",
              "type" => "apache_access"
 }

但我希望解析的日志包括timestamp，loglevel，Class name，name log message。

我正在使用%{DATESTAMP:timestamp} %{WORD:level} %{WORD:location} \- %{GREEDYDATA}来解析日志。

Answer 1

在你的配置的input和output部分之间，如果它还不存在，添加一个filter块，其中grok过滤器配置了你在问题中指明的模式。

input{
  ...
}

filter {
  grok {
    match => ["message", "%{DATESTAMP:timestamp} %{WORD:level} %{WORD:location} \- %{GREEDYDATA}"]
  }
}

output{
  ...
}

Answer 2

pipeline.conf

input{
        file{
                path => "/path/to/logfile"
                sincedb_path => "/dev/null"
                start_position => "beginning"
        }
}
filter{
        grok{
                match => {"message" => "%{DATESTAMP:timestamp} %{WORD:logLevel} %{WORD:className} \- %{GREEDYDATA:logMessage}"}
        }
}
output{
        stdout{
                codec => rubydebug
        }
}

样本输出::

{
          "path" => "/path/to/logfile",
    "@timestamp" => 2017-12-19T13:37:26.542Z,
      "logLevel" => "DEBUG",
    "logMessage" => "Starting handshake",
      "@version" => "1",
          "host" => "HOST",
     "className" => "SdkTLSSocketFactory",
       "message" => "2017-12-18 00:00:00,098 DEBUG SdkTLSSocketFactory - Starting handshake",
     "timestamp" => "17-12-18 00:00:00,098"
}