awk 以及 sort 和 uniq 组合无法正确计数

问题描述 投票：0回答：1

这是我的

file.log
:

{"log":"2023-08-20 10:06:59 [DNSHandler:ProxyResolver] Request: [86.104.44.22:63761] (udp) / 'a1834.dscg2.akamai.NET.' (A)\n","stream":"stdout","time":"2023-08-20T10:07:02.868627873Z"}
{"log":"2023-08-20 10:06:59 [DNSHandler:ProxyResolver] Request: [86.104.44.22:62682] (udp) / 'th.bing.com.' (A)\n","stream":"stdout","time":"2023-08-20T10:07:02.868688906Z"}
{"log":"2023-08-20 10:06:59 [DNSHandler:ProxyResolver] Request: [86.104.44.22:62248] (udp) / 'th.bing.com.' (HTTPS)\n","stream":"stdout","time":"2023-08-20T10:07:02.868705921Z"}
{"log":"2023-08-20 10:06:59 [DNSHandler:ProxyResolver] Request: [86.104.44.22:62719] (udp) / 'a1834.dscg2.akamai.NET.' (HTTPS)\n","stream":"stdout","time":"2023-08-20T10:07:02.868721225Z"}
{"log":"2023-08-20 10:06:59 [DNSHandler:ProxyResolver] Reply: [86.104.44.22:62682] (udp) / 'th.bing.com.' (A) / RRs: CNAME,CNAME,CNAME,A,A,A,A,A,A,A,A,A\n","stream":"stdout","time":"2023-08-20T10:07:02.868739879Z"}
{"log":"2023-08-20 10:06:59 [DNSHandler:ProxyResolver] Reply: [86.104.44.22:62836] (udp) / 'www.bing.com.' (HTTPS) / RRs: CNAME,CNAME,CNAME\n","stream":"stdout","time":"2023-08-20T10:07:02.868760977Z"}
{"log":"2023-08-20 10:06:59 [DNSHandler:ProxyResolver] Reply: [86.104.44.22:63761] (udp) / 'a1834.dscg2.akamai.NET.' (A) / RRs: A,A\n","stream":"stdout","time":"2023-08-20T10:07:02.868810466Z"}
{"log":"2023-08-20 10:06:59 [DNSHandler:ProxyResolver] Reply: [86.104.44.22:62719] (udp) / 'a1834.dscg2.akamai.NET.' (HTTPS) / RRs: \n","stream":"stdout","time":"2023-08-20T10:07:02.868831775Z"}
{"log":"2023-08-20 10:06:59 [DNSHandler:ProxyResolver] Reply: [86.104.44.22:62248] (udp) / 'th.bing.com.' (HTTPS) / RRs: CNAME,CNAME,CNAME\n","stream":"stdout","time":"2023-08-20T10:07:02.868896564Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Request: [86.104.44.22:63587] (udp) / 'hlb.apr-52dd2-0.edgecastdns.NET.' (A)\n","stream":"stdout","time":"2023-08-20T10:07:02.868912596Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Reply: [86.104.44.22:63587] (udp) / 'hlb.apr-52dd2-0.edgecastdns.NET.' (A) / RRs: CNAME,A\n","stream":"stdout","time":"2023-08-20T10:07:02.868926565Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Request: [86.104.44.22:63487] (udp) / 'www.bing.com.' (HTTPS)\n","stream":"stdout","time":"2023-08-20T10:07:02.868940116Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Reply: [86.104.44.22:63487] (udp) / 'www.bing.com.' (HTTPS) / RRs: CNAME,CNAME\n","stream":"stdout","time":"2023-08-20T10:07:02.868953663Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Request: [84.241.34.239:53210] (udp) / 'yunqos.gamesafe.qq.com.' (A)\n","stream":"stdout","time":"2023-08-20T10:07:02.868967065Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Request: [86.104.44.22:63412] (udp) / 'th.bing.com.' (HTTPS)\n","stream":"stdout","time":"2023-08-20T10:07:02.868980656Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Reply: [86.104.44.22:63412] (udp) / 'th.bing.com.' (HTTPS) / RRs: CNAME,CNAME,CNAME\n","stream":"stdout","time":"2023-08-20T10:07:02.86899756Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Request: [86.104.44.22:62452] (udp) / 'onedscolprdwus08.westus.cloudapp.azure.com.' (A)\n","stream":"stdout","time":"2023-08-20T10:07:02.869020234Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Reply: [86.104.44.22:62452] (udp) / 'onedscolprdwus08.westus.cloudapp.azure.com.' (A) / RRs: A\n","stream":"stdout","time":"2023-08-20T10:07:02.86904008Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Request: [86.104.44.22:64304] (udp) / 'www.bing.com.' (HTTPS)\n","stream":"stdout","time":"2023-08-20T10:07:02.869059471Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Reply: [86.104.44.22:64304] (udp) / 'www.bing.com.' (HTTPS) / RRs: CNAME,CNAME,CNAME\n","stream":"stdout","time":"2023-08-20T10:07:02.86907888Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Request: [84.241.34.239:49270] (udp) / 'update.eset.com.' (A)\n","stream":"stdout","time":"2023-08-20T10:07:02.869098025Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Reply: [84.241.34.239:49270] (udp) / 'update.eset.com.' (A) / RRs: CNAME,A\n","stream":"stdout","time":"2023-08-20T10:07:02.869118703Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Reply: [84.241.34.239:53210] (udp) / 'yunqos.gamesafe.qq.com.' (A) / RRs: A,A,A,A,A,A,A,A,A,A\n","stream":"stdout","time":"2023-08-20T10:07:02.869133298Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Request: [94.182.110.194:51644] (udp) / 'raja-bot.utravs.com.' (A)\n","stream":"stdout","time":"2023-08-20T10:07:02.869147079Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Reply: [94.182.110.194:51644] (udp) / 'raja-bot.utravs.com.' (A) / NXDOMAIN\n","stream":"stdout","time":"2023-08-20T10:07:02.869225953Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Request: [84.241.34.239:56795] (udp) / 'dns.msftncsi.com.' (A)\n","stream":"stdout","time":"2023-08-20T10:07:02.869242518Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Reply: [84.241.34.239:56795] (udp) / 'dns.msftncsi.com.' (A) / RRs: A\n","stream":"stdout","time":"2023-08-20T10:07:02.86925624Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Request: [86.104.44.22:62881] (udp) / 'wns.notify.trafficmanager.NET.' (A)\n","stream":"stdout","time":"2023-08-20T10:07:02.869269544Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Reply: [86.104.44.22:62881] (udp) / 'wns.notify.trafficmanager.NET.' (A) / RRs: A\n","stream":"stdout","time":"2023-08-20T10:07:02.869283006Z"}
{"log":"2023-08-20 10:07:00 [DNSHandler:ProxyResolver] Request: [84.241.34.239:61310] (udp) / 'upd.es-eset.com.' (A)\n","stream":"stdout","time":"2023-08-20T10:07:02.869323226Z"}

我尝试使用以下命令仅对域名和 IP 进行排序：

awk '/DNSHandler:ProxyResolver\] Request|Reply/ {print $5, $8}' file.log | sort -k2 | uniq -c | sort -r

这是当前输出：

      2 [94.182.110.194:51644] 'raja-bot.utravs.com.'
      2 [86.104.44.22:64304] 'www.bing.com.'
      2 [86.104.44.22:63761] 'a1834.dscg2.akamai.NET.'
      2 [86.104.44.22:63587] 'hlb.apr-52dd2-0.edgecastdns.NET.'
      2 [86.104.44.22:63487] 'www.bing.com.'
      2 [86.104.44.22:63412] 'th.bing.com.'
      2 [86.104.44.22:62881] 'wns.notify.trafficmanager.NET.'
      1 [86.104.44.22:62836] 'www.bing.com.'
      2 [86.104.44.22:62719] 'a1834.dscg2.akamai.NET.'
      2 [86.104.44.22:62682] 'th.bing.com.'
      2 [86.104.44.22:62452] 'onedscolprdwus08.westus.cloudapp.azure.com.'
      2 [86.104.44.22:62248] 'th.bing.com.'
      1 [84.241.34.239:61310] 'upd.es-eset.com.'
      2 [84.241.34.239:56795] 'dns.msftncsi.com.'
      2 [84.241.34.239:53210] 'yunqos.gamesafe.qq.com.'
      2 [84.241.34.239:49270] 'update.eset.com.'

如您所见，例如

www.bing.com
在
Request
中重复了2次，在Reply中重复了3次，但看起来它正在计算第二列（IP：端口）；

预期产出：

      6 [86.104.44.22:63412] 'th.bing.com.'
      5 [86.104.44.22:64304] 'www.bing.com.'
      4 [86.104.44.22:63761] 'a1834.dscg2.akamai.NET.'
      2 [94.182.110.194:51644] 'raja-bot.utravs.com.'
      2 [86.104.44.22:63587] 'hlb.apr-52dd2-0.edgecastdns.NET.'
      2 [86.104.44.22:62881] 'wns.notify.trafficmanager.NET.'
      2 [86.104.44.22:62452] 'onedscolprdwus08.westus.cloudapp.azure.com.'
      2 [84.241.34.239:56795] 'dns.msftncsi.com.'
      2 [84.241.34.239:53210] 'yunqos.gamesafe.qq.com.'
      2 [84.241.34.239:49270] 'update.eset.com.'
      1 [84.241.34.239:61310] 'upd.es-eset.com.'

看来由于 

IP:Port
（特别是端口）的输出不同，所以它正在计算不同的行。

我打算仅根据第二列（即域名）对它们进行排序。

bash awk uniq
1个回答
1
投票

这就是我解决问题的方法：

awk '/DNSHandler:ProxyResolver\] Request|Reply/ {print $5, $8}' file.log | sort -k2 | uniq -c -f1 | sort -k1r

输出：

      6 [86.104.44.22:62248] 'th.bing.com.'
      5 [86.104.44.22:62836] 'www.bing.com.'
      4 [86.104.44.22:62719] 'a1834.dscg2.akamai.NET.'
      2 [94.182.110.194:51644] 'raja-bot.utravs.com.'
      2 [86.104.44.22:63587] 'hlb.apr-52dd2-0.edgecastdns.NET.'
      2 [86.104.44.22:62881] 'wns.notify.trafficmanager.NET.'
      2 [86.104.44.22:62452] 'onedscolprdwus08.westus.cloudapp.azure.com.'
      2 [84.241.34.239:56795] 'dns.msftncsi.com.'
      2 [84.241.34.239:53210] 'yunqos.gamesafe.qq.com.'
      2 [84.241.34.239:49270] 'update.eset.com.'
      1 [84.241.34.239:61310] 'upd.es-eset.com.'
最新问题
© www.soinside.com 2019 - 2024. All rights reserved.