将http_referer url参数和查询字符串拆分为人类可读数据

Question

我有以下格式的nginx日志

165.225.106.84 - - [20/Dec/2017:12:44:45 +0530] "POST /api/auction/auctionmaster/onauctionmasterfilter HTTP/1.1" 200 3227 "http://auction-dev.iquippo.com/viewauctions?type=upcoming" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36" "115.112.162.2" "{\x22auctionType\x22:\x22upcoming\x22,\x22addAuctionType\x22:true}"

我想像这样拆分我的http_referer值

domain:- http://auction-dev.iquippo.com
param1 :- viewauctions
param2:- if any
query_param1:- upcoming
and so on..

我正在尝试弹性搜索论坛上的这篇文章： - https://discuss.elastic.co/t/extracting-domain-from-url/36219

但这对我不起作用。

Answer 1

注意：可能存在拼写错误，您无法直接复制粘贴，但它是您尝试执行的操作的开始。

首先将引荐存储在变量中，然后使用add_tag添加标记，稍后在标记中执行if标记。

grok {
    match => { "access_log_line" => "%{LINE_WITH_REFERAL}"}
    add_tag => [ "referal" ]
}

if "referal" in [tags] {
    grok {
        match => { "referal" => "%{POST0}" }
        add_tag => [ "referal_step2" ]
    }
}

if "referal" in [tags] {
    grok {
        match => [ "referal_uri" => "%{POST_COMP}" ]
    }
}

示例行：

3227 "http://auction-dev.iquippo.com/viewauctions?type=upcoming"    
2522 "http://auction-dev.iquippo.com/viewauctions?foo?type=upcoming"    
327 "http://auction-dev.iquippo.com/viewauctions?foo?bar?type=upcoming"

整个生产线的第一个GROK模式：

POST0 %{INT} "http://%{IPORHOST}/%{WORD:uri}\?%{GREEDYDATA:data}

GrokPatterns将匹配您的参数：

POST1 type=%{WORD:query_param}"
POST2 %{WORD:param1}?type=%{WORD:query_param}"
POST3 %{WORD:param1}?%{WORD:param2}?type=%{WORD:query_param}"
POST4 %{WORD:param2}?%{WORD:param2}?%{WORD:param3}?type=%{WORD:query_param}"

POST_COMP %{POST1}|%{POST2}|%{POST3}|%{POST4}