创建自定义的GROK模式

Question

当前，我正在尝试为此日志创建一个grok模式

2020-03-11 05:54:26,174 JMXINSTRUMENTS-Threading [{"timestamp":"1583906066","label":"Threading","ObjectName":"java.lang:type\u003dThreading","attributes":[{"name":"CurrentThreadUserTime","value":18600000000},{"name":"ThreadCount","value":152},{"name":"TotalStartedThreadCount","value":1138},{"name":"CurrentThreadCpuTime","value":20804323112},{"name":"PeakThreadCount","value":164},{"name":"DaemonThreadCount","value":136}]}]

目前，通过使用此模式，我可以正确匹配直到JMXINTRUMENTS-Threading：

%{TIMESTAMP_ISO8601:timestamp} (?<instrument>[^\ ]*) ?%{GREEDYDATA:log_message}

但是我似乎无法匹配此后的所有值。是否有人知道我应该使用哪种模式？

Answer 1

我正在https://grokdebug.herokuapp.com/（这是Logstash的官方调试器）中尝试您的模式，并且确实将“ JMXINTRUMENTS-Threading”之后的所有内容与您的模式匹配到一个称为日志消息的大字段中，方式为：

{
  "timestamp": [
    [
      "2020-03-11 05:54:26,174"
    ]
  ],
  "YEAR": [
    [
      "2020"
    ]
  ],
  "MONTHNUM": [
    [
      "03"
    ]
  ],
  "MONTHDAY": [
    [
      "11"
    ]
  ],
  "HOUR": [
    [
      "05",
      null
    ]
  ],
  "MINUTE": [
    [
      "54",
      null
    ]
  ],
  "SECOND": [
    [
      "26,174"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "instrument": [
    [
      "JMXINSTRUMENTS-Threading"
    ]
  ],
  "log_message": [
    [
      "[{"timestamp":"1583906066","label":"Threading","ObjectName":"java.lang:type\\u003dThreading","attributes":[{"name":"CurrentThreadUserTime","value":18600000000},{"name":"ThreadCount","value":152},{"name":"TotalStartedThreadCount","value":1138},{"name":"CurrentThreadCpuTime","value":20804323112},{"name":"PeakThreadCount","value":164},{"name":"DaemonThreadCount","value":136}]}]"
    ]
  ]
}

如果您希望匹配日志消息中包含的所有字段，则应在logstash管道过滤器部分中，就在grok过滤器的正下方使用json过滤器：

例如：

  grok {
     match => { "message" =>"%{TIMESTAMP_ISO8601:timestamp} (?<instrument>[^\ ]*) ?%{GREEDYDATA:log_message}" }
     tag_on_failure => ["no_match"]
  }
  if "no_match" not in [tags] {
    json {
      source => "log_message"
    }
  }

以这种方式，您的json将被拆分为key：value并进行解析。