我正在使用NLog FileTarget来将消息记录在文件中,该文件将被文件拍子拾取并发送到Kibana。
我的消息通常是多行的。
[我注意到,有些消息显示在Kibana中,但多行消息中只有some行显示。
例如,在logfie中,我有这样的内容
2020-05-04 16:23:16.1523|::1|80000037-0000-fb00-b63f-84710c7967bb
2020-05-04 16:23:16.1523|::1|80000037-0000-fb00-b63f-84710c7967bb||INFO|Microsoft.AspNetCore.Hosting.Diagnostics|Request starting HTTP/2.0 GET https://localhost:44328/Foo
2020-05-04 16:23:16.2287|::1|80000037-0000-fb00-b63f-84710c7967bb|user1|DEBUG|MyApp.Web.Controllers.FooController|Validation OK
2020-05-04 16:23:16.2530|::1|80000037-0000-fb00-b63f-84710c7967bb|user1|DEBUG|MyApp.Web.BusinessLogic.FooLogic|Query results time:3ms
2020-05-04 16:23:16.2687|::1|80000037-0000-fb00-b63f-84710c7967bb|user1|DEBUG|MyApp.Web.BusinessLogic.FooLogic|Count:0 time:1ms
2020-05-04 16:23:16.6127|::1|80000037-0000-fb00-b63f-84710c7967bb|user1|INFO|Microsoft.AspNetCore.Hosting.Diagnostics|Request finished in 459.4438ms 200 text/html; charset=utf-8
但是在基巴纳,我只能看到
2020-05-04 16:23:16.1523|::1|80000037-0000-fb00-b63f-84710c7967bb
2020-05-04 16:23:16.1523|::1|80000037-0000-fb00-b63f-84710c7967bb||INFO|Microsoft.AspNetCore.Hosting.Diagnostics|Request starting HTTP/2.0 GET https://localhost:44328/Foo
2020-05-04 16:23:16.2287|::1|80000037-0000-fb00-b63f-84710c7967bb|user1|DEBUG|MyApp.Web.Controllers.FooController|Validation OK
2020-05-04 16:23:16.2530|::1|80000037-0000-fb00-b63f-84710c7967bb|user1|DEBUG|MyApp.Web.BusinessLogic.FooLogic|Query results time:3ms
请注意,这仅对some条消息而不是全部发生,并且日志消息的大小似乎无关紧要。我已经记录了相当长的消息,而较小的消息却被裁剪了。
我能想到的唯一原因是消息不是一次性写入,而是部分写入,而kibana会拾取部分消息,而忽略其余消息。是这样吗如果是这样,我可以将目标配置为一次写所有消息吗?
我的堆栈是:
和我的文件目标配置如下:
"target": {
"type": "File",
"fileName": "c:\\wwwlogs\\MyApp.Web\\Combined.log",
"archiveFileName": "c:\\wwwlogs\\MyApp.Web\\archives\\Combined.{#}.log",
"archiveEvery": "Day",
"archiveNumbering": "Rolling",
"maxArchiveFiles": "7",
"layout": "${longdate}|${aspnet-request-ip}|${aspnet-TraceIdentifier}|${aspnet-user-identity}|${uppercase:${level}}|${logger}|${message} ${exception:format=tostring}"
}
FileBeat的配置是:
- type: log
enabled: true
paths:
- C:\wwwlogs\MyApp.Web\Combined.log
multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after
令人尴尬,但事实证明这不是问题。整个消息已通过filebeat接收,但仅部分显示在Kibana索引视图上。如果单击消息详细信息,则整个消息都是可见的。
因此,NLog和filebeat没问题,这是Kibana中的视图问题
不是FileBeat的专家,但在阅读了文档之后:
然后看起来您可以执行此操作:
- type: log
enabled: true
paths:
- C:\wwwlogs\MyApp.Web\Combined.log
input_type: log
json.message_key: msg
json.keys_under_root: false
json.add_error_key: true
json.overwrite_keys: false
然后您可以使用NLog JsonLayout执行此操作:
"target": {
"type": "File",
"fileName": "c:\\wwwlogs\\MyApp.Web\\Combined.log",
"archiveFileName": "c:\\wwwlogs\\MyApp.Web\\archives\\Combined.{#}.log",
"archiveEvery": "Day",
"archiveNumbering": "Rolling",
"maxArchiveFiles": "7",
"layout": {
"type": "JsonLayout",
"Attributes": [
{
"name": "time",
"layout": "${date:format=o}"
},
{
"name": "lvl",
"layout": "${level}"
},
{
"name": "logger",
"layout": "${logger}"
},
{
"name": "msg",
"layout": "${message}"
},
{
"name": "req_traceid",
"layout": "${aspnet-TraceIdentifier}"
},
{
"name": "req_user",
"layout": "${aspnet-user-identity}"
},
{
"name": "req_ip",
"layout": "${aspnet-request-ip}"
},
{
"name": "error_type",
"layout": "${exception:format=type}"
},
{
"name": "exception",
"layout": "${exception:format=tostring}"
},
{
"name": "properties",
"encode": false,
"layout": {
"type": "JsonLayout",
"includeallproperties": "true"
}
}]
}
}