我正在寻找一个非常基本的管道模板,该模板允许我正确索引日志消息的所有可用字段。
我开箱即用地使用Spring Boot(2.1.x),将其部署到Cloud Foundry并通过stdout / logdrain记录到Logstash,最后到Elasticsearch。
我已经搜索了互联网,仅找到Cloud Foundry应用程序的一个模板:
input {
http {
port => "5044"
user => "inputuser"
password => "inputpassword"
}
}
filter {
grok {
#patterns_dir => "{{ .Env.HOME }}/grok-patterns"
match => { "message" => "%{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:syslog5424_ts}|-) +(?:%{HOSTNAME:syslog5424_host}|-) +(?:%{NOTSPACE:syslog5424_app}|-) +(?:%{NOTSPACE:syslog5424_proc}|-) +(?:%{WORD:syslog5424_msgid}|-) +(?:%{SYSLOG5424SD:syslog5424_sd}|-|)%{SPACE}%{GREEDYDATA:message}" }
add_tag => [ "CF","CF-%{syslog5424_proc}","_grokked"]
add_field => { "format" => "cf" }
tag_on_failure => [ ]
overwrite => [ "message" ]
}
if [syslog5424_proc] =~ /(A[pP]{2}.+)/ {
mutate { add_tag => ["CF-APP"] }
mutate { remove_tag => ["_grokked"] }
}
if ("CF-APP" in [tags]) or !("CF" in [tags]) {
if [message] =~ /^{.*}/ {
json {
source => "message"
add_tag => [ "json", "_grokked"]
}
}
}
if !("_grokked" in [tags]) {
mutate{
add_tag => [ "_ungrokked" ]
}
}
}
output {
#stdout { codec => rubydebug }
if ("_grokked" in [tags]) {
elasticsearch {
hosts => ["https://ac9537fc444c489bb63ac44064c54519.elasticsearch.lyra-836.appcloud.swisscom.com"]
user => "myuser"
password => "mypassword"
ssl => true
ssl_certificate_verification => true
codec => "plain"
workers => 1
index => "parsed-%{+YYYY.MM.dd}"
manage_template => true
template_name => "logstash"
template_overwrite => true
}
} else {
elasticsearch {
hosts => ["https://ac9537fc848c489bb63ac44064c54519.elasticsearch.lyra-836.appcloud.swisscom.com"]
user => "myuser"
password => "mypassword"
ssl => true
ssl_certificate_verification => true
codec => "plain"
workers => 1
index => "unparsed-%{+YYYY.MM.dd}"
manage_template => true
template_name => "logstash"
template_overwrite => true
}
}
}
这看起来很冗长,仅涵盖Cloud Foundry字段,但忽略所有特定于应用程序的字段,例如日志级别(不遵循键/值表示法,而是在日志消息中固定位置)。
一个示例日志消息是:
2019-10-03T09:20:09.37+0200 [APP/PROC/WEB/0] OUT 2019-10-03 09:20:09.378 INFO 19 --- [ main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 2 endpoint(s) beneath base path '/actuator'
非常感谢您的帮助,非常感谢!
Update:基于第一个注释,我将Spring Boot应用程序配置为将消息记录为json。在Cloud Foundry中,我将带有用户提供的服务的那些日志配置为logdrain发送到Logstash。 Logstash收到如下消息:
<14>1 2019-10-03T17:29:17.547195+00:00 cf-organization.cf-space.cf-app abc9dac6-1234-4b62-9eb4-98d1234d9ace [APP/PROC/WEB/1] - - {"app":"cf-app","ts":"2019-10-03T17:29:17.546+00:00","logger":"org.springframework.boot.web.embedded.netty.NettyWebServer","level":"INFO","class":"org.springframework.boot.web.embedded.netty.NettyWebServer","method":"start","file":"NettyWebServer.java","line":76,"thread":"main","msg":"Netty started on port(s): 8080"}
使用上述过滤器,Logstash将其解析为此json:
{
"syslog5424_ts": "2019-10-03T17:29:17.547195+00:00",
"syslog5424_pri": "14",
"syslog5424_ver": "1",
"message": "{\"app\":\"cf-app\",\"ts\":\"2019-10-03T17:29:17.546+00:00\",\"logger\":\"org.springframework.boot.web.embedded.netty.NettyWebServer\",\"level\":\"INFO\",\"class\":\"org.springframework.boot.web.embedded.netty.NettyWebServer\",\"method\":\"start\",\"file\":\"NettyWebServer.java\",\"line\":76,\"thread\":\"main\",\"msg\":\"Netty started on port(s): 8080\"}",
"syslog5424_app": "abc9dac6-1234-4b62-9eb4-98d1234d9ace",
"syslog5424_proc": "[APP/PROC/WEB/1]",
"syslog5424_host": "cf-organization.cf-space.cf-app"
}
[我将如何调整grok /输出以仅将键message
的值作为json发送给Elasticsearch?
好,所以我通过以下步骤设法做到了,感谢this nice article:
implementation 'net.logstash.logback:logstash-logback-encoder:5.2'
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<property resource="application.properties"/>
<contextName>${spring.application.name}</contextName>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
<contextName>
<fieldName>app</fieldName>
</contextName>
<timestamp>
<fieldName>ts</fieldName>
<timeZone>UTC</timeZone>
</timestamp>
<loggerName>
<fieldName>logger</fieldName>
</loggerName>
<logLevel>
<fieldName>level</fieldName>
</logLevel>
<callerData>
<classFieldName>class</classFieldName>
<methodFieldName>method</methodFieldName>
<lineFieldName>line</lineFieldName>
<fileFieldName>file</fileFieldName>
</callerData>
<threadName>
<fieldName>thread</fieldName>
</threadName>
<mdc/>
<arguments>
<includeNonStructuredArguments>false</includeNonStructuredArguments>
</arguments>
<stackTrace>
<fieldName>stack</fieldName>
</stackTrace>
<message>
<fieldName>msg</fieldName>
</message>
</providers>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="CONSOLE"/>
</root>
</configuration>
spring.application.name=<app-name>
spring.main.banner-mode=OFF
这将生成如下所示的日志:
<14>1 2019-10-03T17:29:17.547195+00:00 cf-organization.cf-space.cf-app abc9dac6-1234-4b62-9eb4-98d1234d9ace [APP/PROC/WEB/1] - - {"app":"cf-app","ts":"2019-10-03T17:29:17.546+00:00","logger":"org.springframework.boot.web.embedded.netty.NettyWebServer","level":"INFO","class":"org.springframework.boot.web.embedded.netty.NettyWebServer","method":"start","file":"NettyWebServer.java","line":76,"thread":"main","msg":"Netty started on port(s): 8080"}
现在,我们需要解析前置文本并将其值添加到记录的消息中。
input {
http {
port => "5044"
user => "exampleUser"
password => "examplePassword"
}
}
filter{
grok {
#patterns_dir => "{{ .Env.HOME }}/grok-patterns"
match => { "message" => "%{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:syslog5424_ts}|-) +(?:%{HOSTNAME:syslog5424_host}|-) +(?:%{NOTSPACE:syslog5424_app}|-) +(?:%{NOTSPACE:syslog5424_proc}|-) +(?:%{WORD:syslog5424_msgid}|-) +(?:%{SYSLOG5424SD:syslog5424_sd}|-|)%{SPACE}%{GREEDYDATA:message}" }
add_tag => [ "CF", "CF-%{syslog5424_proc}", "parsed"]
add_field => { "format" => "cf" }
tag_on_failure => [ ]
overwrite => [ "message" ]
}
mutate {
split => ["syslog5424_host", "."]
add_field => { "cf-org" => "%{[syslog5424_host][0]}" }
add_field => { "cf-space" => "%{[syslog5424_host][1]}" }
add_field => { "cf-app" => "%{[syslog5424_host][2]}" }
}
if [syslog5424_proc] =~ /\[(A[pP]{2}.+)/ {
mutate { add_tag => ["CF-APP"] }
mutate { remove_tag => ["parsed"] }
}
if ("CF-APP" in [tags]) or !("CF" in [tags]) {
if [message] =~ /^{.*}/ {
json {
source => "message"
add_tag => [ "json", "parsed"]
}
}
}
if !("CF-APP" in [tags]) {
mutate {
add_field => { "msg" => "%{[message]}" }
add_tag => [ "CF-PAAS"]
}
}
if !("parsed" in [tags]) {
mutate{
add_tag => [ "unparsed" ]
}
}
}
output {
if ("parsed" in [tags]) {
elasticsearch {
hosts => ["https://7875eb592bb94554ad35421dccc6847f.elasticsearch.lyra-836.appcloud.swisscom.com"]
user => "logstash-system-ExjpCND01GbF7knG"
password => "5v9nUztOkz0WUdKK"
ssl => true
ssl_certificate_verification => true
codec => "plain"
workers => 1
index => "parsed-%{+YYYY.MM.dd}"
manage_template => true
template_name => "logstash"
template_overwrite => true
}
} else {
elasticsearch {
hosts => ["https://7875eb592bb94554ad35421dccc6847f.elasticsearch.lyra-836.appcloud.swisscom.com"]
user => "logstash-system-ExjpCND01GbF7knG"
password => "5v9nUztOkz0WUdKK"
ssl => true
ssl_certificate_verification => true
codec => "plain"
workers => 1
index => "unparsed-%{+YYYY.MM.dd}"
manage_template => true
template_name => "logstash"
template_overwrite => true
}
}
}
感谢@Strelok向我指出正确的方向。