GROK 模式未返回结果

Question

我对 ELK 很陌生，我一直停留在提取字段上。下面是示例数据

Dec 9 06:36:01 s-login-01 CRON[2436102]: pam_unix(cron:session): session closed for user mXXt

Dec 9 06:34:07 s-login-01 sshd[2424671]: Disconnected from user sw 10.xx.1x.xx port 4000

Dec 9 06:34:05 s-login-01 systemd-logind[2405]: Session 20923 logged out. Waiting for processes to exit.

我有上面的示例数据，我想知道如何为此编写 .conf 文件。我尝试使用下面的 .conf 但它没有提取字段。

input {
file {
path => "/../syslog.log"
type => "syslog"
start_position => beginning
sincedb_path => "/dev/null"
}
}
filter {
grok {

    match => { "message" => "%{SYSLOGTIMESTAMP:timestamp}%{SPACE}%{NOTSPACE:HOST}%{SPACE}%{NOTSPACE:PROCESS}\[%{NUMBER:PID}\]\:%{GREEDYDATA:activity}" }
    match => { "message" => "%{SYSLOGTIMESTAMP:timestamp}%{SPACE}%{NOTSPACE:HOST}%{SPACE}%{NOTSPACE:PROCESS}\[%{NUMBER:PID}\]\:%{GREEDYDATA:activity}.?*%{WORD:User}" }
    match => { "message" => "%{SYSLOGTIMESTAMP:timestamp}%{SPACE}%{NOTSPACE:HOST}%{SPACE}%{NOTSPACE:PROCESS}\[%{NUMBER:PID}\]\:%{GREEDYDATA:activity}.?*%{WORD:User}.?*%{IP:IP}.?*%{NUMBER:Port}" }
    match => { "message" => "%{SYSLOGTIMESTAMP:timestamp}%{SPACE}%{NOTSPACE:HOST}%{SPACE}%{NOTSPACE:PROCESS}\[%{NUMBER:PID}\]\:%{GREEDYDATA:activity}.?*%{NUMBER:Session_ID}" }
}
}
output {
Elasticsearch {
hosts => ["localhost:9200"]
index => "sample_"
}
}

Answer 1

这是满足您提供的所有示例数据的单个 grok 模式：

%{SYSLOGTIMESTAMP:TIMESTAMP} %{DATA:HOST} %{DATA:PROCESS}\[%{BASE10NUM:PID}\]: %{GREEDYDATA:MSG}

我使用了 GROK DEBUGGER 来创建 grok 模式。

这是输出：

---

在下面找到单个样本数据的 grok 模式：

12 月 9 日 06:34:07 s-login-01 sshd[2424671]：与用户 sw 10.xx.1x.xx 端口 4000 断开连接

%{SYSLOGTIMESTAMP:TIMESTAMP} %{DATA:HOST} %{DATA:PROCESS}\[%{BASE10NUM:PID}\]: %{DATA:ACTIVITY} %{DATA:msg} user %{DATA:USER} %{IP:IPADDRESS} %{DATA:msg} %{BASE10NUM:PORT}

12 月 9 日 06:36:01 s-login-01 CRON[2436102]：pam_unix（cron：session）：用户 mXXt 的会话已关闭

%{SYSLOGTIMESTAMP:TIMESTAMP} %{DATA:HOST} %{DATA:PROCESS}\[%{BASE10NUM:PID}\]: %{GREEDYDATA:ACTIVITY} user %{WORD:USER}

12 月 9 日 06:34:05 s-login-01 systemd-logind[2405]：会话 20923 已注销。等待进程退出。

%{SYSLOGTIMESTAMP:TIMESTAMP} %{DATA:HOST} %{DATA:PROCESS}\[%{BASE10NUM:PID}\]: Session %{BASE10NUM:SESSION_ID} %{GREEDYDATA:ACTIVITY}

Answer 2

你必须这样设置

input {
        file {

        path => "/root/mult.log"
        start_position => "beginning"
        sincedb_path => "/dev/null"
        codec => multiline{

                 pattern => "^ -%{SPACE}%{TIMESTAMP_ISO8601}"
                 negate => true
                 what => "previous"
        }
}
 }
filter {
    grok {
     match => [
       "message", "(?m)^ -%{SPACE}%{TIMESTAMP_ISO8601:time} \[%{WORD:main}\] %{LOGLEVEL:loglevel}%{SPACE}\(%{JAVACLASS:class}\) %{DATA:mydata}\n(\t)?%{GREEDYDATA:stack}",
       "message", "^ -%{SPACE}%{TIMESTAMP_ISO8601:time} \[%{WORD:main}\] %{LOGLEVEL:loglevel}%{SPACE}\(%{JAVACLASS:class}\) %{GREEDYDATA:mydata}" ]
        break_on_match => false
 }
    date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"]
 }
}

output {
  stdout { codec => rubydebug}
elasticsearch {
    host => "localhost"
  }
}