我有日志条目,如下面的神交模式。
日志:
2017-04-11 18:31:41,938 | INFO | 195 | Process | Bundle Name | logStr: GUID: dl99X/WeN77E2SmyjH9uS1Fy+EDvFQ5R_939bae | ReferenceID: 20170411183141500676 | InstanceID: 70411183141906430422429270016 | ChannelID: EXAMPLE | System: EXAMPLE | ServiceName: EXAMPLE | InvocationPoint: inbound
2017-04-11 18:31:42,743 | INFO | 193 | API | Bundle Name | Outbound Message | RESPONSE=[GUID=[dl99X/WeN77E2SmyjH9uS1Fy+EDvFQ5R_939bae], InstanceID=[70411183141906430422429270016], logStr=[GUID: dl99X/WeN77E2SmyjH9uS1Fy+EDvFQ5R_939bae | ReferenceID: 20170411183141500676 | InstanceID: 70411183141906430422429270016 | ChannelID: EXAMPLE | System: EXAMPLE | ServiceName: EXAMPLE | InvocationPoint: inbound
神交模式:
grok {
#grok general pattern
match => {
"message" => "%{TIMESTAMP_ISO8601:logdate}%{SPACE}\|%{SPACE}%{LOGLEVEL:level}%{SPACE}\|%{SPACE}%{DATA:thread}%{SPACE}\|%{SPACE}%{DATA:serviceName}%{SPACE}\|%{SPACE}%{DATA:bundle}%{SPACE}\|%{SPACE}%{GREEDYDATA:logdetails}"
}
}
#Grok to get GUID
grok {
match => {
"logdetails" => "(?<=GUID:).%{DATA:guid}(?=\s)"
}
}
#Grok to get ChannelID
grok {
match => {
"logdetails" => "(?<=ChannelID:).%{DATA:channelID}(?=\s)"
}
}
#Grok to get ReferenceID
grok {
match => {
"logdetails" => "(?<=ReferenceID:).%{DATA:referenceID}(?=\s)"
}
}
我有几个独立的神交只得到了GUID,的channelID和ReferenceID。有什么办法给groks合并成一个?
先感谢您!
这是最好的知道类型,你正在处理的日志的,但什么时候有太多类型的担心(但他们仍然以相同的格式)我会做:
您的每一个消息都timestamp | loglevel | thread
的基本格式:
LINE %{BASE}\s?\|\s?%{GREEDYDATA:msg}
# Patterns
BASE %{CUSTTIME:timestamp}\s?\|\s?%{WORD:loglevel}\s?\|\s?%{NONNEGINT:thread}
CUSTTIME %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND},%{MILLI}
MILLI (?:([1-9][0-9]{0,2}|0))
然后,你可以用冒号加图案为你在同一个模式文件查找的领域,因为所有的值都非常相似,键=值,而是:
COMMAVALUE (\s?(.*?(?=\s\w+:|$))\s?)
# Fields
GUID GUID:%{COMMAVALUE:guid}
CHANNELID ChannelID:%{COMMAVALUE:channel_id}
REFERENCEID ReferenceID:%{COMMAVALUE:reference_id}
所以,你可以使用两个相邻的神交过滤器要做到这一点,一个提取msg
有效载荷,另一个说味精提取领域的。
filter {
grok {
patterns_dir => "/etc/logstash/patterns"
match => { "message" => "%{LINE}" }
}
grok {
patterns_dir => "/etc/logstash/patterns"
break_on_match => false
match => [
"msg", "%{GUID}",
"msg", "%{CHANNELID}",
"msg", "%{REFERENCEID}"
]
}
}
output {
stdout { codec => "rubydebug" }
}