未为多行事件添加Filebeat多行标签

Question

我做了什么：

我已在filebeat.yml文件中启用多行模式：

  multiline.pattern: '^[[:space:]]+|^Caused by:'
  multiline.negate: false
  multiline.match: after

我的示例日志文件包含多行异常：

Exception in thread "main" java.lang.IllegalStateException: A book has a null property
       at com.example.myproject.Author.getBookIds(Author.java:38)
       at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Caused by: java.lang.NullPointerException
       at com.example.myproject.Book.getId(Book.java:22)
       at com.example.myproject.Author.getBookIds(Author.java:35)
       ... 1 more
Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

从文件记录日志中，我可以看到这些多行异常已得到正确解析：

2017-05-15T08:21:36-07:00 DBG  Publish: {
  "@timestamp": "2017-05-15T15:21:31.768Z",
  "beat": {
    "hostname": "WIN-UV5OA3SO3LF",
    "name": "WIN-UV5OA3SO3LF",
    "version": "5.2.0"
  },
  "input_type": "log",
  "message": "Exception in thread \"main\" java.lang.IllegalStateException: A book has a null property\n       at com.example.myproject.Author.getBookIds(Author.java:38)\n       at com.example.myproject.Bootstrap.main(Bootstrap.java:14)\nCaused by: java.lang.NullPointerException\n       at com.example.myproject.Book.getId(Book.java:22)\n       at com.example.myproject.Author.getBookIds(Author.java:35)\n       ... 1 more",
  "offset": 409,
  "source": "C:\\Filebeat\\test\\testLog.txt",
  "type": "log"
}

2017-05-15T08:21:41-07:00 DBG  Publish: {
  "@timestamp": "2017-05-15T15:21:31.768Z",
  "beat": {
    "hostname": "WIN-UV5OA3SO3LF",
    "name": "WIN-UV5OA3SO3LF",
    "version": "5.2.0"
  },
  "input_type": "log",
  "message": "Exception in thread \"main\" java.lang.NullPointerException\n        at com.example.myproject.Book.getTitle(Book.java:16)\n        at com.example.myproject.Author.getBookTitles(Author.java:25)\n        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)",
  "offset": 669,
  "source": "C:\\Filebeat\\test\\testLog.txt",
  "type": "log"
}

问题：

• 由filebeat解析的多行事件未被标记为多行。
• 由于这个原因，我在logstash上的grok过滤器无法相应地对其进行处理

    if "multiline" in [tags] {
      grok {
        match => ["message", "%{JAVASTACKTRACEPART}"]
      }
    }

任何指针将不胜感激！

Answer 1

很抱歉，我不回答问题，我建议从另一侧面看问题：为什么不为每行新的日志行的开头引入固定模式（这是惯例）？

这允许将negate模式用于多行功能，这是更容易，更直接的方法。

我们接受每个新行都以日期开头，如下所示：

2019.10.23 01:01:01.384500 [ERROR] Exception in thread "main" java.lang.IllegalStateException: A book has a null property
       at com.example.myproject.Author.getBookIds(Author.java:38)
       at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Caused by: java.lang.NullPointerException
       at com.example.myproject.Book.getId(Book.java:22)
       at com.example.myproject.Author.getBookIds(Author.java:35)
       ... 1 more
2019.10.23 01:02:01.384500 [ERROR] Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

现在，只需要定义开始每行的模式并打开求反模式：

filebeat.inputs:
- type: log
  enabled: true
  # https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html
  # This regexp describes the begin of log row (for example, '2019.10.23 01:01:01.384500 ..').
  multiline.pattern: '^\d{4}.\d{2}.\d{2} \d{2}:\d{2}:\d{2}'
  multiline.negate: true
  multiline.match: after  
  paths:
    - ..

processors:
   - ..
   - dissect:
      # tokenizer syntax: https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html.
      tokenizer: "%{timestamp} [%{level}] %{?message}"
      # https://www.elastic.co/guide/en/beats/filebeat/master/dissect.html
      field: "message"
      target_prefix: ""