古怪的问题

Question

我们正在Elasticsearch中使用管道来模拟grok过滤器。我们遇到以下奇怪的行为。

如果时间戳记字段在消息的开头，grok过滤器将不起作用。

## GROK NOT WORK
    POST /_ingest/pipeline/_simulate
    {
      "pipeline": {
        "processors": [
          {
            "grok": {
              "field": "message",
              "patterns": ["""%{TIMESTAMP_ISO8601:@timestamp} %{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration} %{TIMESTAMP_ISO8601:@timestamp}"""]
            }
          }
        ]
      },
        "docs": [
        {
          "_source": {
            "message": "2019-09-29T00:39:02.91ZZ 55.3.244.1 GET /index.html 15824 0.043  "
          }
        }
      ]
    }

我们遇到以下错误：

{
  "docs" : [
    {
      "error" : {
        "root_cause" : [
          {
            "type" : "illegal_argument_exception",
            "reason" : "Provided Grok expressions do not match field value: [2019-09-29T00:39:02.91ZZ 55.3.244.1 GET /index.html 15824 0.043  ]"
          }
        ],
        "type" : "illegal_argument_exception",
        "reason" : "Provided Grok expressions do not match field value: [2019-09-29T00:39:02.91ZZ 55.3.244.1 GET /index.html 15824 0.043  ]"
      }
    }
  ]
}

以这种格式，消息末尾的时间戳，grok过滤器可以正常工作。

    ## GROK WORKS FINE
    POST /_ingest/pipeline/_simulate
    {
      "pipeline": {
        "processors": [
          {
            "grok": {
              "field": "message",
              "patterns": ["""%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration} %{TIMESTAMP_ISO8601:@timestamp}"""]
            }
          }
        ]
      },
        "docs": [
        {
          "_source": {
            "message": "55.3.244.1 GET /index.html 15824 0.043 2019-09-29T00:39:02.91ZZ"
          }
        }
      ]
    }

提前感谢

Answer 1

它们在时间后为“ ZZ”，与ISO不兼容。

您可以添加。*->“任何字符中的0个或更多”

％{TIMESTAMP_ISO8601：@timestamp}。*

和最后一个时间戳记模式之后的*，使其成为可选

{
  "pipeline": {
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{TIMESTAMP_ISO8601:@timestamp}.* %{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration} %{TIMESTAMP_ISO8601:@timestamp}*"
          ]
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": "2019-09-29T00:39:02.91ZZ 55.3.244.1 GET /index.html 15824 0.043  "
      }
    }
  ]

结果：

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "duration" : "0.043",
          "request" : "/index.html",
          "@timestamp" : "2019-09-29T00:39:02.91Z",
          "method" : "GET",
          "bytes" : "15824",
          "client" : "55.3.244.1",
          "message" : "2019-09-29T00:39:02.91ZZ 55.3.244.1 GET /index.html 15824 0.043  "
        },
        "_ingest" : {
          "timestamp" : "2020-04-18T10:43:39.8725873Z"
        }
      }
    }
  ]
}